* More tips for gathering information after a security breach
In this continuing series of articles on Computer Incident Response Teams I am reviewing principles and practical pointers for effective response to security breaches and other operational difficulties in information technology management. Today I’m continuing my focus on critical distinctions that your CIRT members should keep in mind when gathering data.
DISTINGUISH OBSERVATION FROM HYPOTHESIS
Sometimes when people are careless or untrained, they don’t distinguish between what they saw and an idea that might explain what they saw. In the example (see previous newsletter) about a supposed flaw in a firewall, the person speaking seemed to take the flaw for granted; that was an assumption. A similar problem can occur when someone thinks that _maybe_ there’s a flaw in the firewall and then proceeds as if that were true without testing their hypothesis. “And so maybe they exploited a flaw in the firewall, so we should patch all the holes right away.” Putting aside for the moment the advisability of patching holes and firewalls, merely hypothesizing an exploit doesn’t make it true. Maybe it’s a good thing to patch the firewall, but it doesn’t follow that it’s the top priority right now simply from having thought of the idea. CIRT staff should be careful to think about what they’re hearing and note explicitly when people are proposing explanations rather than reporting facts.
CHALLENGE YOUR HYPOTHESIS
I hope you will forgive me, Dear Reader, for a brief foray into the philosophy of science. I do have a reason to bringing it up.
In the 35 years I have been teaching college courses, I’ve taught biology, genetics, biochemistry, embryology, physiology, applied statistics, programming, software engineering and information assurance. All of these subjects have involved a concept that some students have struggled to grasp: science depends on _disproof_, not proof. Empirical science (in contrast to logical systems such as mathematics) does not offer “proofs” in an absolute sense. Instead, a scientist formulates a hypothesis, defines a set of conditions and observations with predicted results and sees if there are grounds for rejecting randomness as a simple explanation of the deviation of the observations from the predictions. In many cases, scientists will assume the _absence_ of a relationship or phenomenon (thus “null” hypothesis). Many experiments assume the absence of the interesting stuff and try to see if there are grounds for _rejecting_ this simple explanation: “There’s nothing there.” Science works by DISPROVING hypotheses. Explanations that cannot, by definition, be disproved are not part of a scientific effort.
Even more confusing for people who habitually think in terms of absolutes, even accepting the null hypothesis doesn’t necessarily mean that there’s nothing there. We may be measuring or counting too few occurrences to spot the cases that will challenge the nonexistence of the phenomenon. There may also be confounding factors that obscure a real phenomenon.
But rejecting the null hypothesis does not, however, prove that any _specific_ alternate hypothesis is necessarily correct. The evidence just restricts the _range_ of reasonable hypotheses. We knock out explanation after explanation until what’s left is a smaller set of explanations. In science, the best we hope for is not truth in an absolute sense but an operational equivalent to truth: useful enough to use for now.
OK, so now I want to bring this back to network management and the CIRT. When your CIRT members develop hypotheses, they have to try to shoot them down. Trying to show that an idea is _correct_ is – ironically – the wrong approach to testing hypotheses. Just as in quality assurance, we have to come up with ways of showing that our explanation is wrong. If we fail enough times to disprove an hypothesis using genuine, thoughtful, intelligent tests of our ideas, maybe we’ve got something useful after all.
* * *
If you would like to see and hear a narrated PowerPoint lecture that expands on the topics covered in these two articles, you can download a WinZIP compressed file that is used in the Norwich MSIA program from:
https://www2.norwich.edu/mkabay/msia/public/Problems.zip
After you extract and run the PPT file from this archive, just press F5 (on a Windows system) to run the show and hear the commentary.
