How to Measure Anything in Cybersecurity Risk

As I said, this book reads like an education in quantitative modeling and how to apply the methodology to cybersecurity. It truly challenges the current common practices in use to develop expert opinion-based risk frameworks. Here is a snippet from the book:

“So let’s be clear about our position on current methods: They are a failure. They do not work. A thorough investigation of the research on these methods and decision-making methods in general indicates the following: There is no evidence that the types of scoring and risk matrix methods widely used in cybersecurity improve judgment. On the contrary, there is evidence these methods add noise and error to the judgment process. Any appearance of “working” is probably a type of “analysis placebo.” That is, a method may make you feel better even though the activity provides no measurable improvement in estimating risks (or even adds error). There is overwhelming evidence in published research that quantitative, probabilistic methods are effective. Fortunately, most cybersecurity experts seem willing and able to adopt better quantitative solutions. But common misconceptions held by some—including misconceptions about basic statistics—create some obstacles for adopting better methods. How cybersecurity assesses risk, and how it determines how much it reduces risk, are the basis for determining where cybersecurity needs to prioritize the use of resources. And if this method is broken—or even just leaves room for significant improvement—then that is the highest-priority problem for cybersecurity to tackle!”

The authors lay out the book in three sections:

• Part I sets the stage for reasoning about uncertainty in security. It outlines terms on things like security, uncertainty, measurement and risk management. Plus, it argues against toxic misunderstandings of these terms and why we need a better approach to measuring cybersecurity risk and, for that matter, measuring the performance of cybersecurity risk analysis itself. Finally, it introduces a simple quantitative method that could serve as a starting point for anyone, no matter how averse the person may be to complexity.
• Part II delves further into evolutionary steps we can take with a simple quantitative model. It explains how to add further complexity to a model and how to use even minimal amounts of data to improve those models.
• Part III describes what is needed to implement these methods in the organization. It addresses the implications of this book for the entire cybersecurity “ecosystem,” including standards organizations and vendors.

The cybersecurity community suffers from not having standard evaluation metrics, like earnings before interest, taxes, depreciation and amortization (EBITDA). The authors try to bring some discipline to terms by offering standard definitions coming from the quantitative analytics field. From the book:

• Definitions for Uncertainty and Risk, and Their Measurements Uncertainty: The lack of complete certainty, that is, the existence of more than one possibility. The “true” outcome/state/ result/value is not known. Measurement of Uncertainty: A set of probabilities assigned to a set of possibilities. For example: “There is a 20% chance we will have a data breach sometime in the next five years.” Risk: A state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. Measurement of Risk: A set of possibilities, each with quantified probabilities and quantified losses. For example: “We believe there is a 10% chance that a data breach will result in a legal liability exceeding $10 million.”

They also walk the reader through established methodologies like: Monte Carlo simulations, Bayesian interpretation, risk matrix, loss exceedance curve, heat maps, chain rule tree, beta distribution changes, regression model predations, analytics maturity mode, power law distribution, subjective probability, calibration, dimensional modeling, expected opportunity loss, bunch of guys sitting around talking, expected value of prefect information, NIST and ISO. They explain how, in Excel, so they are truly practical. They also lay out survey results from attitudes toward quantitative methods, global information security workforce study, and stats literacy and acceptance studies.

This work follows other work like Factor Analysis of Information Risk (FAIR) which is a well-recognized value at risk (VaR) framework. They outline another Monte Carlo–based methodology and tools like those developed by Jack Jones and Jack Freund. Another similar work is The Wisdom of Crowds by James Surowiecki.

Finally the book has some great online resources. You can find eight sample downloads of the methods explained, as well as webinar/blog info.

Conclusion

How to Measure Anything in Cybersecurity Risk is an extension of Hubbard’s successful first book, How to Measure Anything: Finding the Value of “Intangibles” in Business. It lays out why statistical models beat expertise every time. It is a book anyone who is responsible for measuring risk, developing metrics, or determining return on investment should read. It provides a strong foundation in qualitative analytics with practical application guidance.

Bottom line: The authors lay out a solid case for why other industries with the similar challenges of lack of quantifiable, standardized or historical actuarial table-like data are able to use classic statistical modeling and methodologies to measure risk in a qualified, repeatable way. Definitely worth considering.

Book Review by Rick Howard

The Cybersecurity Canon Committee inducted the first edition of this book into the Hall of Fame back in 2017. Steve Winterfeld did the initial book review and Bob Clark interviewed the authors at the time for the Cybersecurity Canon Gala awards show that year (See References below). The authors, Douglas Hubbard and Richard Seiersen, published a second edition in 2023 and I wanted to take a look at the updated material. I reached out to Richard Seiersen, a friend of mine now (he and I met at the gala and even presented together at the RSA Conference in 2018 on this topic) and asked him to summarize the update. Here’s what he said.

• A new and simpler “Rapid Risk Audit” along with the one-for-one substitution
• New statistical methods for quick estimates
• Updated research on the impact of data breaches
• New Bayesian examples
• Additional methods on decomposing risk by subsystem and simple adjustments for the effectiveness of controls
• Simple methods for combining the estimates of multiple experts in a way that outperforms individual experts
• New methods using the R statistics language
• New guest contributors
• A forward by Jack Jones himself!

All that’s true. In terms of outline, they replaced Chapter 3 (Model Now) with a new chapter called The Rapid Risk Audit and sprinkled in new sections where needed:

• A Taxonomy of Measurement Scales
• More Hints for Controlling Overconfidence
• Beyond Initial Calibration Training: More methods for improving subjective judgment
• An Example from Little Data: Does Multifactor Authentication Work?
• Other Ways Bayes Applies
• More Advanced Modeling Considerations
• Functional Security Metrics Applied: Boom!
• Wait-Time Baselines
• Security Metrics with the Modern Data Stack
• Modeling for Security Business Intelligence
• Integrating CSRM with the Rest of the Enterprise

And before I start to throw my opinions around about this edition, let me say up front that these guys have found some fantastic quotes about probability and risk forecasting to begin each chapter that brought me great joy. Here are three of my favorites:

• “Bayesian inference is the reallocation of credibility across possibilities … Credibility is synonymous with probability.” –John K. Krushke’s “Doing Bayesian Data Analysis”
• “It is unanimously agreed that statistics depends somehow on probability. But, as to what probability is and how it is connected with statistics, there has seldom been such complete disagreement and breakdown of communication since the Tower of Babel.” —L. J. Savage, American mathematician
• “The most important questions of life are indeed, for the most part, really only problems of probability. —Pierre‐Simon Laplace, Théorie Analytiquedes Probabilités, 1812

The first edition of this book, and another Canon Hall of Fame book, “Measuring and Managing Information Risk: A FAIR Approach” published in 2014 by Jack Freund & Jack Jones, introduced me to the idea of better risk forecasting. The industry has been using Risk Matrix Heat Maps as a best practice since the early 1990s to convey cyber risk to senior leaders and I used to be one of those guys. But, as Hubbard and Seiersen point out in both editions, statisticians have written reams of research papers showing that heat maps are just bad science for this task.

For the uninitiated, a heat map puts all the bad things that can happen to your organization on an x-y coordinate system. The x-axis tracks how likely it is that the bad thing will happen from “unlikely” on the far left to “highly likely” on the far right. The Y axis tracks the potential damage from the bad thing from “not much” to  “existential threat to the business.” The really dangerous risks float up and to the right of the chart. Heat map designers color code the matrix so that risks high and to the right are red, risks in the middle are yellow, and risks low and to the left are green. That’s why they call it a heat map.

Out of all the reasons that heat maps are bad science, two strongly resonated with me. The first is that heat maps use ordinal scales (high, medium, low, likely, and unlikely) that nobody really understands. When I ask you what “Likely” means, do you think to yourself, “Well, that’s almost a sure thing” or do you say to yourself, “Well, that’s better than 50/50.” The research shows that even if I tell you what I think it means (close to 100%), your own personal bias kicks in and you might use your own number (50/50).

The second is even a bigger flaw.  A risk matrix doesn’t provide business leaders a way to judge if some risk placed high and to the right is within their risk tolerance. The map represents the risk as a scary thing (fear, uncertainty, and doubt) that needs addressing, not as a business risk that leaders need to make a decision about.

For both of these reasons and others, the authors proved to me that the better chart for this task is something called a Loss Exceedance Curve and I have spent almost a decade trying to learn how to practically build these things myself. More on that project in a bit.

A Loss Exceedance Curve shows the probability of losses exceeding different values. It’s another x-y coordinate system but this time, the x axis shows the estimated dollar losses from zero dollars on the very left to something very high on the right (like $500M for example). The y axis shows the probability. The curve snakes through the grid matching probabilities to the dollar amounts. You might see that there is a 30% chance that the company could lose $100K in the next year due to a cyber event. You might also see that there is a 5% chance of losing $1M.

Loss Exceedance Curves correct my two biggest objections to heat maps. Readers of the chart know exactly what the labels mean. They are precise. The impact is that leaders can make judgements regarding their own risk tolerance. CEOs might say that they can live with the 5% chance of losing $1M but that the 10% chance of losing $100K is too high. In that case, they would direct the CISO to bring that probability down.

The book’s main thesis is that in order to build these charts, calculating probabilities and estimating dollar loss are two essential skills. They advocate for the idea of using the math principles behind the Bayes Algorithm for calculating probabilities, and they suggest using simple Monte Carlo simulations within a spreadsheet to estimate dollar losses.

The Bayes Algorithm is the underlying bedrock idea for the entire book. It is a statistical method that updates beliefs based on new evidence using conditional probability to make more accurate predictions. In other words, you make an initial estimate regardless of how broad it is using in-house experts. Over time, you collect more evidence that will allow you to adjust the estimate up or down based on the information collected. Each time you adjust the estimate, the answer brings you closer to the actual risk.

Note: For a deeper review of the history of Bayes’ Algorithm and how it has been used in the past 200 years to solve highly complex problems, see Sharon Bertsch McGrayne’s 2011 book, “The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy.” I summarized some of those stories in the appendix to my book called “Cybersecurity First Principles Appendix – Bayes Success Stories” (See References).

A Monte Carlo simulation is a computational technique used to model and analyze complex systems. You build a reasonable model that provides a random answer within the parameters you specify (like there is a 15% chance of losing between $100K and $500M). You run the model 10,000 times and collect the data. You use that data to build the Loss Exceedance Curve. The authors walk the reader through a practical example of just how to create a Loss Exceedance Curve and provide other more complex models on their book website.

I’m not a math guy and frankly, whenever a statistician starts throwing around the names of various distributions and why you should use one over the other (like Triangular, Binary, Normal, Lognormal, Beta, and Power Law), I want to throw myself out the window. But even I understood the author’s explanation of why you would use a  lognormal distribution over a normal distribution for these calculations; essentially, the lognormal distribution can’t generate a zero or negative amount, but it has a tail to the right that allows for the possibility of extremely large outcomes. The great news about this is that spreadsheets do all the work for you. You just have to use the correct formula.

Hubbard and Seiersen spend time explaining how to improve the ability of experts to make better risk forecast estimates. They call it calibration. They recall the research done by  Dr. Philip Tetlock in his 2005 book, “Expert Political Judgment: How Good Is It? How Can We Know?” about, in part, why some forecasters are better than others. I would also point readers to Tetlock’s follow-on 2015 Cybersecurity Canon Hall of Fame book, “Superforecasting: The Art and Science of Prediction” that refines the thought. Hubbard and Seiersen recommend various strategies to make anybody better at this task and provide practice exercises to see how good you are.

They quote the famous British statistician George E. P. Box, “All models are wrong, but some are useful”. The rest of the book is dedicated to making their recommended models more useful.

That is why the Canon Committee selected the first edition of the book for the Hall of Fame. It is a new and better way to think about risk for your organization. The second edition expands on those ideas. I endorse completely the committee’s hall of fame induction of the book. That said, I do have some quibbles.

First, this is not an easy read. It’s not a book where you grab the audio version, take the dogs around the block, and hope to learn the bulk of it. You have to sit with it, read it and re-read it, and try the examples. The authors assume you know the math and don’t waste a lot of time explaining things. You can tell they tried to improve on that between the two editions, but it’s still dense if you’re not a math person comfortable with probabilities. I’m not, so I had to wade through it. I got most of it but the advanced stuff in the later chapters was beyond me.

Second, the supplemental materials were not that helpful. I applaud any author that supplies supplemental material on a web page somewhere and Hubbard and Seiersen do that in spades. They provide numerous spreadsheets with practical examples of the models they describe in the book. But, they offer no explanation of how they work. The book refers to them but doesn’t explain them and, like I said, I’m not a math guy, a spreadsheet guy, nor am I a probabilities guy. The spreadsheets didn’t match exactly what they were talking about in the book either so they weren’t that much help to me. Your mileage may vary.

Lastly, my biggest criticism of the book, and for “Measuring and Managing Information Risk” too, is that I kept waiting for the chapter at the end that showed how to practically use these techniques in the real world. Don’t get me wrong, they show how to build a loss exceedance curve (and I spent the 2023 holiday break learning how to do it myself) and how to make the models better, but they didn’t spend any time describing how you might use these techniques to build a better deck for the board or senior leadership. Once you get done with all of the math, how does the answer help you convey risk to the leaders of the company? That chapter doesn’t exist. So, I wrote it myself (See Chapter 6, Cybersecurity First Principles in the references below).

Those complaints aside, Hubbard and Seiersen have written a must-read cybersecurity book. The second edition has only made the material better. If you’re still using heat maps to convey risk to senior leadership and you haven’t read this book yet, you have a giant hole in your education. Stop what you’re doing right now and read this book.