Controlling outages through high availability and monitoring
Last Tuesday, September 1, Google’s web-based email went offline. This follows another outage in May. The September outage was caused by engineers taking some servers offline and inadvertently overloading other servers as a result. May’s outage was caused by a traffic routing error. There were previous outages earlier in May, one in mid-April, and another in March.
What does this mean?
For one thing, both Google and Hotmail promise 99.9% availability (also known “3 nines”). As good as that sounds, 3 nines doesn’t matter during that other one-tenth of one percent. 3 nines also equates to nearly 44 minutes of unscheduled downtime per month, or 8 hours and 45 minutes per year. The more nines, the more uptime you have – with 4 nines only 52 minutes of downtime per year, and the vaunted 5 nines delivering 5 minutes 16 seconds of unscheduled downtime in a year’s time. To be considered truly available, 3 nines really doesn’t cut it. (And this is only regarding unscheduled downtime, not maintenance windows!)
While I don’t know how Google structures its operations, there are some other areas to look at here. How does one control unscheduled downtime? Building redundancy, a.k.a high availability, into one’s operations certainly can help. If Google had backup servers they could use or had clustered their servers, that may have prevented the last outage.
Another approach that can be helpful is monitoring your production environment to know what’s going on. If you could see that other servers were starting to get overloaded, you could proactively bring on other servers or reassign the workload, perhaps using virtualization to do this quickly.
Monitoring tools and virtualization are not new technologies. Back in the days of “big iron” and mainframes, IBM had VM as an operating system and both they and third parties had monitoring tools. Today we also have virtualization technologies and monitoring tools. This may or may not get you to 6 nines of availability, but being able to proactively monitor your production environment and architect high availability can make a big difference.
