Why a two hour Gmail outage doesn’t mean the cloud stinks for the enterprise

Analysis
Sep 2, 20093 mins

The move to the cloud could be job security for network professionals

Yesterday’s Gmail outage was caused by overloaded servers and a misconfiguration of networking gear, reports IDG News Service. The outage caused an I-told-you-so effect from IT professionals who worry that the cloud is nowhere near ready as a prime time, production environment. But I beg to differ. Outages occur, even e-mail outages, even on private infrastructure. As one cloud computing user said to me today, “That’s why they have a statistic called mean-time-between-failures.”

The outage was caused by changes Google made in an attempt to better load balance, the company said. It had had altered traffic flow on some routers and servers designed to direct Web queries to the appropriate Gmail server. Oops. These changes accidentally overloaded the system after workers took some Gmail servers offline to perform routine upgrades, according to Google’s official Gmail blog.

Ben Treynor, site reliability Czar wrote on the Gmail blog. “At about 12:30 p.m. Pacific a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!”. This transferred the load onto the remaining request routers, causing a few more of them also to become overloaded, and within minutes nearly all of the request routers were overloaded.”

Google’s mistake should not serve as warning that the cloud is inherently unreliable unless you can say with absolute authority that no one in your IT department would ever make a configuration mistake that could bring down your internal e-mail. But the Gmail problem is interesting in that it points out how critical the network becomes when users put their apps and data in the cloud. While server admins may be looking at cloud computing wondering if it will take away their jobs, network guys should not be in the same boat. As applications move to the cloud, a company’s connection to its data will become more critical. The job of architecting solid, failure resistant networks won’t be going away anytime soon.

Like this post? Visit the Google Subnet home page for more news, blogs and podcasts. Sign up for the weekly Google newsletter. (Click on News/Google News Alert.) More blog posts from Google Subnet:
  • 10 cool things you didn’t know about Google [slideshow]
  • Apple may decide to ok Google Voice on the iPhone afterall
  • Can Android apps hold a candle to iPhone apps?
  • Google wants an act of Congress to fix orphan books mess
  • EU to investigate Google Book deal
  • Google Chrome Pre-fetching
  • Live by the Meme, Die by the Meme