One of the most difficult aspects of managing risk in information assurance (IA) is that our statistical information is so poor. We don't know about security breaches that we have not noticed; we don't report all the breaches that we do notice to any central collection point; and we use dreadful methodology for collecting information using poorly-constructed surveys that have tiny percentages of respondents, no internal validation, and no follow-up verification.
One of the most difficult aspects of managing risk in information assurance (IA) is that our statistical information is so poor. We don't know about security breaches that we have not noticed; we don't report all the breaches that we do notice to any central collection point; and we use dreadful methodology for collecting information using poorly constructed surveys that have tiny percentages of respondents, no internal validation and no follow-up verification.
On a practical level, the question arises of just exactly what we should be measuring (such as how to define security metrics) as ways of understanding and managing security issues.
Dr. Gary Hinson, CISSP, CISA, CISM, MBA of Isect wrote an excellent paper entitled "Seven myths about information security metrics" that was originally published in the ISSA Journal in July 2006. Hinson thoughtfully and articulately challenges these seven common assertions (quoting the headings):
1. Metrics must be objective and tangible
2. Metrics must have discrete values
3. We need absolute measurements
4. Metrics are costly
5. You can't manage what you can't measure and you can't improve what you can't manage
6. It is essential to measure process outcomes
7. We need the numbers!
In his section on "Some pragmatic design considerations for information security measurement systems," Hinson discusses key issues (again, quoting his headings):
1. Which things are we going to measure
2. How will we measure things?
3. How will we report?
4. How should we implement our measurement and reporting systems?
5. Setting targets
One of Hinson's references is to the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-55, "Security Metrics Guide for Information Technology Systems." Since his article was written, this publication has been revised.
NIST SP 800-55 Revision 1 is entitled, "Performance Measurement Guide for Information Security." Published in July 2008, this 80-page document was written by Elizabeth Chew, Marianne Swanson, Kevin Stine, Nadya Bartol, Anthony Brown and Will Robinson.
The authors summarize the benefits of using measures (§3.2) as follows (they explain each of these points at length):
• Increase accountability
• Improve information security effectiveness
• Demonstrate compliance
• Provide quantifiable inputs for resource allocation decisions.
The document provides thought-provoking analysis of the organizational implications of developing and using security metrics, including a perspective on U.S. federal government pressures such as the Federal Information Security Management Act (FISMA). Section 5 has a valuable schema for developing appropriate metrics and Section 6 makes practical suggestions for implementing the data collection for those metrics.
Another useful reference for everyone interested in security metrics is the recent (March 2009) draft publication entitled "Directions in Security Metrics Research" (NIST Interagency Report, NISTIR 7564) by Wayne Jensen. This short (26 page) paper has an intriguing abstract:
More than 100 years ago, Lord Kelvin insightfully observed that measurement is vital to deep knowledge and understanding in physical science. During the last few decades, researchers have made various attempts to develop measures and systems of measurement for computer security with varying degrees of success. This paper provides an overview of the security metrics area and looks at possible avenues of research that could be pursued to advance the state of the art.
Jensen discusses metrics at a deeper level than the other papers: his concern is with fundamental issues. For example, his Sections 3 and 5 have these topics:
3.1 Correctness and Effectiveness
3.2 Leading Versus Lagging Indicators
3.3 Organizational Security Objectives
3.4 Qualitative and Quantitative Properties
3.5 Measurements of the Large Versus the Small
4. Possible Research Areas
4.1 Formal Models of Security Measurement and Metrics
4.2 Historical Data Collection and Analysis
4.3 Artificial Intelligence Assessment Techniques
4.4 Practicable Concrete Measurement Methods
4.5 Intrinsically Measurable Components
I encourage readers to plunge into this fascinating area of research and to engage in discussions in their favorite professional arenas. I think that the papers briefly pointed to in today's column provide an excellent basis for those discussions.