* Unusual activity can be a security tip
I recently came across a story in Risks that reminded me that some computer failures can be spotted using outlier analysis or even just common sense.
Back in November 2003, Risks contributor Danny Burstein sent in a report about a medical testing equipment failure that illustrates a common failing among computer users: not using common sense.
It seems that “about 3,000 people got opposite results when they were tested for gonorrhea and chlamydia over an 18-month period. Because of a faulty diagnostic machine in Cranbrook (southeastern British Columbia), positive and negative test results for the two sexually transmitted diseases were reversed.”
Peter Neumann’s summary continues, “About 3,000 people were tested. The 83 that were positive were incorrectly told they were clean. The 2,900 or so that were negative were told they were positive and were given the standard treatments.”
Burstein and Neumann correctly note, “One Would Have Thought that someone in the medical office or the lab or the insurance or the pharmacy or somewhere…, looking at 3,000 test results, would have quickly noticed that instead of finding a positive rate of 3% these tests were coming back at 97%.” [Risks 23.19]
The case is a reminder that system and network managers must analyze outliers. Outliers are unusual events.
Examples include the biggest users of network bandwidth, the user with the highest rate of growth in network disk storage, the department with the highest number of calls per capita to the help desk, and the workgroup with the sharpest inflection point (change in slope) in their total mainframe CPU utilization growth curve.
In research, it is a truism that once the basic model has been tested and currently available alternative explanations for observations have been disproved, the next phase of work is to analyze “residuals.” Residuals are the deviations from expectations based on the current model. Residuals are the veins of observation in which we can mine additional insights into reality.
The people who were processing the reversed data in the Canadian medical-equipment case should have been interested in the unusual ratio of infected vs. uninfected patients. Even the first dozen cases or so should have alerted a responsible supervisor that there was a problem. For example, if the expected occurrence rate of infection was normally 3%, the non-infection rate was 97%. So the likelihood of having 10 uninfected people in the first 10 results would be (0.97)^10 = 74%. Looked at another way, the likelihood of having at least one infected person in the first group of 10 results would be 26% (1 – 0.74 = 0.26). The likelihood of having two or more infected results out of 10 would be only 0.72%. (The derivation is left as an exercise for the reader. Hint: Calculate the probability of at least one infection out of nine patients and then multiply the probability that a 10th patient is infected).
So an alert statistician would have seen by the second “infection” in the series that there was something odd about the results and possibly saved more than 2,900 people from being treated for diseases they didn’t have – and would have gotten quicker treatment to people who were really sick.
I remember one Monday morning 20 years ago when I was checking the weekly status reports for clients at the service bureau where I was director of technical services in the 1980s. I notice a sharp inflection in the disk space utilization for one of our clients over the last week: they were increasing their usage about 10 times faster than ever before and much faster than anyone else on the system. Investigation revealed that a programmer had REMmed (commented) out the PURGE commands for hundreds of temporary files used in the nightly batch programs as part of a diagnostic run – and then forgotten to take out the REMs. There were now thousands of these files accumulating in the client’s account for no good reason, costing them money and putting our disk capacity at risk. So one simple question, “What’s causing this outlier?” saved us a great deal of trouble.
Don’t ignore outliers.
