I was watching the opening night events of the Sochi Olympics last night while I was finalizing my presentation for RSA Conference this month in San Francisco. My presentation is on what the right metrics are to measure security and risk. At the same time, I am watching figure skating and womens mogul skiing. It struck me that in all of these things subjective judging of performance is just so imprecise.
When you watch a skier go down a hill and measure exactly how long it takes and whether they missed any gates, it is cut-and-dry to see who did it faster. But when you have to judge how well they are handling the bumps, how cool their jump was and things like that, beauty is really in the eye of the beholder. The same thing in figure skating. I can tell if someone falls or doesn't land cleanly on a jump. But a triple lutz from a double axil? How much to award for grace and presentation? Beauty is in the eye of the beholder.
The same goes for security and risk. One man's risk is another man's disaster. What you may think important to measure and manage really isn't important to another organization. I first ran into this in the vulnerability scoring area. Just because you rate a vulnerability critical, don't think everyone else will. If that critical vulnerability is on an unreachable server, it isn't so critical. If there is nothing of value on the device with the vulnerability and it doesn't lead anywhere else, again, it is not critical.
Another issue from my RSA presentation is what is important depends on who you are. C-level and board members aren't really interested in the nuts and bolts of security metrics for the most part. They want it boiled down to are we at risk? What is the risk and what can we do to lower that risk reasonably? Confusing them with lots of metrics or facts on things they really don't care about only turns them off and confuses the issue.
What we need in IT security and in many of these Olympic events is to make the judge less important than the athlete. Let's not leave it up to someone's interpretation, prejudices and pre-conceived notions. A standard set of criteria that are less subjective and more objective makes everyone's lives easier.
I know what you are saying. Shimmy, you are crazy. There are just certain things that we can't reduce to metrics. The world doesn't work that way. Maybe you are right, the world doesn't work that way and you can't reduce everything to numbers. But that doesn't mean we should stop trying. The more we can make objective and the less subjective, the better our systems will be.
Whether we are talking about Olympic competition or measuring and managing our risk, the more objective the better.
As co-founder and Managing Partner at The CISO Group, Alan Shimel is responsible for driving the vision and mission of the company. The CISO Group offers security consulting and PCI compliance management for the payment card industry. Prior to The CISO Group, Alan was the Chief Strategy Officer at StillSecure. Shimel was the public persona of StillSecure as it grew from start up to helping defend some of the largest and most sensitive networks in the world.
Shimel is an often-cited personality in the technology community and is a sought-after speaker at industry and government conferences and events. His commentary about the state of security, open source and life is followed closely by many industry insiders via his blog and podcast, "Ashimmy, After All These Years" (www.ashimmy.com). Alan is now also a regular contributor to The CISO Group’s security.exe blog and podcast. Follow him on Google.
Alan has helped build several successful technology companies by combining a strong business background with a deep knowledge of technology. His legal background, long experience in the field, and New York street smarts combine to form a unique personality.
Disclosure: The CISO Group sells a software-as-a-service PCI compliance application called SAQPro. The company is independent and does not represent any other vendor's products as a reseller.
Policy on comments: Respectful discussion is welcomed! However comments that use inappropriate language, consist of name calling or personal attacks, or include accusations of wrongdoing are not appropriate. Those comments will be deleted or edited.