Can we measure the value of good security?
SIEM
Most people expect good security to be invisible, yet struggle to assign value to it when it is.
by Oliver Rochford, Senior Director
As an industry, we struggle to value or measure security, and I have some anecdotes to support my assertion.
I have heard from several MSSPs that they feel that they have to forward false positives to some clients to forestall a perception that the provider isn’t doing anything. Go figure – apparently creating needless work for all parties involved is required to demonstrate value.
Another example is the trend for making detecting a breach a success criteria for a live proof of concept for a detection (or deception) technology. Oftentimes it is not known if a compromise has even occurred, nor is this ensured through red teaming or adversarial simulation. The solution is dropped into a live network with the expectation that it will find bad stuff – whether it’s there or not. Solution selection is predicated on a hypothetical, and more importantly, something outside of the control of the tested and the tester.
False positives is a further topic that demonstrates this well, with complaints that machine learning or behavioral analytics still have false positives (FPs). There is an expectation of zero FPs, yet how reliably something can be detected rests on many factors – most of them unsolvable with any technology. Rarely are solutions directly compared in terms of the number of FPs. Yet even minor reductions in FPs yield huge savings in effort over time, and modern analytics reduces FPs by double digit percentages. Not valuing incremental improvements is usually challenging for people, not businesses. Business thrives on incremental improvements.
These anecdotes show that there is a perception of value problem. But what does value even mean? Let’s begin by taking a look at a dictionary definition of the word:
value verb
valued; valuing
Definition of value (Entry 2 of 3)
transitive verb
1: to consider or rate highly : PRIZE, ESTEEM
values your opinion
2a: to estimate or assign the monetary worth of : APPRAISE
value a necklace
b: to rate or scale in usefulness, importance, or general worth : EVALUATE
The only definition out of the three to mention monetary worth is 2a. We can of course calculate what we’re spending on security, but that’s just a tally for bookkeeping and for peer benchmarking.
The cost of salaries, hardware, software and services are the parts that make security but don’t express the sum. In many ways, good security is an intangible asset. It cannot be amortized for example. Neither does adding together the sum of what we spend on security make it tangible, just as there is no direct correlation between the cost of a set of cards and how well someone plays with them. But intangible assets are valuable, otherwise how could they make up 90% of the market value of companies listed in the SP500 in 2020?
This points to a different issue – not that security doesn’t have value – but that we have yet to work out how to measure that value.
But surely you say, the value of cybersecurity can easily be measured – have I been breached?
The fallacy there is in thinking 100% security is achievable, especially continuously, or even always desirable.
We have come to accept that, all things being equal, and considering the challenges arising from scale, complexity and pace of change, a perfect state of being secure does not exist in an active and evolving system. Your exposure to risk, and your ability to mitigate those risks, are in constant flux, so even if you are able to achieve something approximating 100%, it will be a temporary circumstance. Cyber defense and offense are in an adaptive cycle resulting in continuous iterative evolution. There is no status quo, no “normal.” Instead we now focus on being resilient to cyber attacks through rapid response, limiting the scope of breaches, and frictionless recovery.
Another way of valuing security instead arises out of a feeling – the feeling that you are able to safely and securely operate the way you need to (as per dictionary definition #1 “to consider or rate highly”). Essentially, valuing security means having confidence that it is effective.
And to cultivate a feeling of confidence requires a sense of what is being done, why it’s being done and how well it’s being done. It requires making security visible, even when there’s no breach, And that’s where measuring and metrics come back in.
Sadly though, as an industry, we have historically been really bad at providing insightful and actionable security metrics. The most obvious example is that we continue to mistake operational metrics for risk or value metrics. In the C-Suite, the perception is that security metrics are too technical in nature – and more damning, do not reflect the likelihood of being breached, or the consequences if we are. Adding business context to technical KPIs as in Gartner’s CARE framework is a valiant attempt at making bad data usable, but only skirts around the lack of good metrics.
A part of the problem can be explained by the immaturity of security. We have yet to establish a shared taxonomy and ontology, even though with efforts like MITRE ATT&CK we are getting closer. We also have not established scientific methods of assessing risk, a necessary first step if we want to improve business methods. But here we are also starting to see green shoots, for example with projects like the FAIR framework. Slowly we are beginning to move from ritual and superstition through art and to science.
I will be writing much more about how to measure, assess and value security on these pages, and look forward to our journey together to learn more about key performance and risk indicators, risk quantification and security benchmarking.
