One of the most difficult aspects of managing risk in information assurance (IA) is that our statistical information is so poor. We don't know about security breaches that we have not noticed; we don't report all the breaches that we do notice to any central collection point; and we use dreadful methodology for collecting information using poorly constructed surveys that have tiny percentages of respondents, no internal validation and no follow-up verification.

On a practical level, the question arises of just exactly what we should be measuring (such as how to define security metrics) as ways of understanding and managing security issues.

Dr. Gary Hinson, CISSP, CISA, CISM, MBA of Isect wrote an excellent paper entitled "Seven myths about information security metrics" that was originally published in the ISSA Journal in July 2006. Hinson thoughtfully and articulately challenges these seven common assertions (quoting the headings):

1. Metrics must be objective and tangible
2. Metrics must have discrete values
3. We need absolute measurements
4. Metrics are costly
5. You can't manage what you can't measure and you can't improve what you can't manage
6. It is essential to measure process outcomes
7. We need the numbers!

In his section on "Some pragmatic design considerations for information security measurement systems," Hinson discusses key issues (again, quoting his headings):

1. Which things are we going to measure
2. How will we measure things?
3. How will we report?
4. How should we implement our measurement and reporting systems?
5. Setting targets

One of Hinson's references is to the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-55, "Security Metrics Guide for Information Technology Systems." Since his article was written, this publication has been revised.

NIST SP 800-55 Revision 1 is entitled, "Performance Measurement Guide for Information Security." Published in July 2008, this 80-page document was written by Elizabeth Chew, Marianne Swanson, Kevin Stine, Nadya Bartol, Anthony Brown and Will Robinson.

The authors summarize the benefits of using measures (§3.2) as follows (they explain each of these points at length):

• Increase accountability
• Improve information security effectiveness
• Demonstrate compliance
• Provide quantifiable inputs for resource allocation decisions.

M. E. Kabay, PhD, CISSP-ISSMP, specializes in security and operations management consulting services. CV online.