Judging in the Olympics and IT Security: Subjective vs. Objective

I was watching the opening night events of the Sochi Olympics last night while I was finalizing my presentation for RSA Conference this month in San Francisco. My presentation is on what the right metrics are to measure security and risk. At the same time, I am watching figure skating and womens mogul skiing. It struck me that in all of these things subjective judging of performance is just so imprecise.

When you watch a skier go down a hill and measure exactly how long it takes and whether they missed any gates, it is cut-and-dry to see who did it faster. But when you have to judge how well they are handling the bumps, how cool their jump was and things like that, beauty is really in the eye of the beholder. The same thing in figure skating. I can tell if someone falls or doesn't land cleanly on a jump. But a triple lutz from a double axil? How much to award for grace and presentation? Beauty is in the eye of the beholder.

The same goes for security and risk. One man's risk is another man's disaster. What you may think important to measure and manage really isn't important to another organization. I first ran into this in the vulnerability scoring area. Just because you rate a vulnerability critical, don't think everyone else will. If that critical vulnerability is on an unreachable server, it isn't so critical. If there is nothing of value on the device with the vulnerability and it doesn't lead anywhere else, again, it is not critical.

Another issue from my RSA presentation is what is important depends on who you are. C-level and board members aren't really interested in the nuts and bolts of security metrics for the most part. They want it boiled down to are we at risk? What is the risk and what can we do to lower that risk reasonably? Confusing them with lots of metrics or facts on things they really don't care about only turns them off and confuses the issue.

What we need in IT security and in many of these Olympic events is to make the judge less important than the athlete. Let's not leave it up to someone's interpretation, prejudices and pre-conceived notions. A standard set of criteria that are less subjective and more objective makes everyone's lives easier.

I know what you are saying. Shimmy, you are crazy. There are just certain things that we can't reduce to metrics. The world doesn't work that way.  Maybe you are right, the world doesn't work that way and you can't reduce everything to numbers. But that doesn't mean we should stop trying. The more we can make objective and the less subjective, the better our systems will be.

Whether we are talking about Olympic competition or measuring and managing our risk, the more objective the better.