* Computers aren't always right
In addition to teaching information assurance courses at my university, I also teach database design every year and sometimes teach systems engineering courses. In all of these courses, at some point I emphasize the importance of integrating plausible limits into the computer-human interface to reduce the effects of unthinking human credulity.
As someone who has been programming computers since 1965, I am always amazed at how readily naive users will accept utter nonsense simply because it is presented by a computer.
I began using slide rules in 1962 and still have and use my sickly greenish-yellow Pickett “Vector-Type Log-Log Dual-Base Speed Rule.” A necessary skill in using these calculating devices is the ability to keep an estimated order of magnitude for the answer in one’s head. Alas, I fear that mental arithmetic is a lost art for most people.
I’ve often told students of an incident at a grocery store years ago where a charming child told me that my tiny order would cost over $20; I firmly asserted that it would cost around $7 and told her to check the bill. Sure enough, my paltry selection cost something like $7.23. She stared at me in utter amazement and asked in disbelief, “How did you _do_ that??” Clearly, Isaac Asimov was prescient when he wrote “The Feeling of Power” in 1958 – a story about a world where everyone had forgotten that computation could be accomplished without computers.
In a spectacular demonstration of slavish obedience to computers, a pair of nitwits demonstrated the crucial role of observation and thought when using mission-critical technology. Peter G. Neumann wrote in Risks 20.14 (the item is dated Dec. 28, 1998):
“A German couple drove their BMW with great confidence under control of its computerized satellite navigation. Indeed, they drove it past a stop sign, down a ferry ramp, and into the Havel River in Caputh, near Potsdam/Berlin, Germany. The computer system reportedly neglected to tell them they needed to wait for the ferry. Ship traffic was stopped for two hours, but the couple was OK.”
In February 1999, the Risks Forum Digest had a cute story from Carnegie Mellon University Professor Philip Koopman, who lost his photocopier privileges for one of his graduate courses because the administrators reported, straight-faced, that he and his students had made 4,294,967,026 copies in two weeks.
They knew this because a computer told them so.
A quick calculation would have shown the administrators that even at 10 copies per minute, it would take more than 816 years of continuous operation day and night without interruption to print that many copies (remember that a year has about 365.25 days). If we estimate 250 pages per inch of thickness, that number of pages would stand over 271 miles high. Wouldn’t even a few moments of common sense have pointed to a system error rather than abuse of photocopier privileges as a better explanation of such a preposterous volume?
In Valparaiso, Ind., someone pressed the wrong key in the municipal-tax program in 2005 and accidentally altered the property value for a house originally evaluated at $121,900 so that it was appraised at $400 million. No one noticed. The tax bill went from $1,500 to $8 million, causing a significant increase in the anticipated municipal tax revenues. Although the faulty tax bill was corrected, the town planners had already lowered the property tax rate to take into account the imaginary $8 million windfall and therefore faced a budget deficit for municipal services and schools.
Human beings should not allow the origin of information to overwhelm rationality. Whether it is government propaganda, commercial advertising, the results of an electronic voting machine devoid of auditability or any critical data baldly presented by a computer, information must be evaluated for credibility. Is it consistent with known limits of the process being described? Does it conform to reasonable predictions? Can it be checked or tested independently? Basically, does it make sense?
Computers should not be approached as worthy of unquestioning faith. Let’s not substitute credulity for credibility.
