RSA: Metrics are key to measuring security effectiveness

Gathering metrics to measure the effectiveness of an enterprise security strategy can be a difficult and somewhat imprecise task, but that’s no excuse for not trying, said IT managers at RSA Conference 2006 in San Jose, Calif., this week.

That’s because metrics are the only way to truly tell if enough money is being spent on the company’s security effort, they added.

“Start using metrics to make security decisions, and don’t get too hung up on the quality of the data, and don’t get too hung up on complicated methodologies,” said John Meakin, group head of information security at Standard Chartered Bank in London. “Just start doing it,” he said.

Security experts have for some time advocated the use of metrics to get a more measured view of IT operational risks and the controls needed to mitigate them. The topic is important at a time when companies are coming under increasing compliance pressures that require them to demonstrate due diligence when protecting their data assets. The idea of metrics is to give companies a way to prioritize threats and vulnerabilities and the risks they pose to enterprise information assets based on a quantitative or qualitative measure.

Setting such priorities can help companies target their IT security resources far more effectively, Meakin said. Standard Chartered, for instance, has been moving to a risk-based approach to vulnerability management over the past three years. As part of the effort, the company has classified all of its core information systems on a value scale of high, medium and low based on the importance of those systems to the business and the disruption or loss that would result from a security failure on any of them.

The bank has developed similar measures for threats and vulnerabilities and the likelihood of them being exploited against each of these systems.

The approach has given the company a much clearer picture of enterprisewide IT risks and the controls needed to mitigate them, Meakin said, It has also helped the bank to target its security resources much better, he said. For instance, about three years ago, the bank was considering encrypting all confidential traffic moving over one of its wide-area networks because of security concerns. But a metrics-based risk assessment showed that such encryption was overkill, he said. “I would say one of the worst things I could do is spend too much money on security,” he said.

Zions Bancorporation in Salt Lake City started using metrics as part of its security efforts about four years ago. The goal was to move from a “subjective, gut feel of risk” to a more measured view of the threats and vulnerabilities the bank faced and the controls needed to mitigate them, said Preston Wood, the bank’s chief information security officer.

The metrics have allowed the bank to get a clearer picture of the effectiveness of its tactical and strategic security efforts, Wood said. “It’s very much making sure you spend just enough” on security, he said, “Not more, not less.” Metrics have also been useful in getting business groups to understand the nature of the IT security risks they face and the controls that are needed to address them, Wood said.

But arriving at quantitative or qualitative measures for security can be a big challenge, given the dynamic nature of threats and vulnerabilities and the difficulty involved in attaching a definite value to information assets, Meakin said.

Enterprise Management Associates., a Boulder, Colo.-based consulting firm, last May released a white paper highlighting some of these challenges while suggesting an approach to gather risk metrics.

“Risk management calculations can embrace a wide breadth of detail, from multifactor calculations that can reflect key aspects of the risk posture of the enterprise as a whole to detailed measures of compliance in specific respects,” the paper said. Because of this, risk measures can be daunting to gather, said Enterprise Management Associates.

“But there’s no excuse not to start doing it,” said Dan Geer, chief scientist at Verdasys, a security vendor in Waltham, Mass.

The goal is not so much about arriving at specific numbers for measuring risk, but about getting a feel for what’s important and why, he said. “This is an idea whose time has come,” Geer said.

Despite the challenges, it’s possible to begin gathering and using metrics to more effectively manage security efforts, he said. “I’m fairly certain that A is better than B and that B is better than C. I’m not sure if I can say A is 3.2 B, and that B is 6.9 C,” and that shouldn’t be the goal, Geer said.

The key is not to make the whole process overly complicated, said Pete Lindstrom, an analyst at Spire Security in Malvern, Pa.

Metrics are “simply a probability based on legitimate experience of your network that some bad activity is going to occur,” he said. “They help you understand what controls are most effective in reducing risk.”