Seven Million Muscovites and an Elephant

In his marvelous book Against the Gods: The Remarkable Story of Risk, Peter L. Bernstein tells the story of a Soviet statistics professor in World War II who refused to go to his neighborhood bomb shelter during the frequent German air raids. “There are seven million people in Moscow,” he would tell his concerned friends. Why should a bomb pick him, specifically, out of that crowd? Then one night he showed up in the shelter, to the surprise of his neighbors. “There are seven million people in Moscow,” he explained, “and one elephant. Last night they got the elephant.” Our reaction to identical probabilities can be distinctly colored by subjective interpretations. The probability of the professor being hit by a bomb was about the same as the elephant being hit by a bomb. But where the professor felt statistics were on his side as one of seven million, the elephant was unique. And always, someone has to be the elephant. “We pay excessive attention to low-probability events accompanied by high drama,” Bernstein writes, “and overlook events that happen in routine fashion.” We take reasonable precautions in the design of our networks to mitigate the risks from software, hardware, and link failures. We analyze MTBFs. We conduct regression, interoperability, and performance testing before putting anything new in the network. We build in as much redundancy as budget and reason allows. We lock doors and install firewalls and IDS to keep the evildoers out, both physically and virtually. Yet the biggest source of outages in most networks is not hardware or software or link failures. It is the person sitting in the operations center or engineering office legitimately logged into a router, doing authorized configuration changes. And most network operators do shockingly little to manage the human risk. I’ll have more to say about human risk in tomorrow’s post.