Besting the storm

Weather Underground uses Linux clusters and low-cost bandwidth to keep from being pelted by traffic storms.

Introduction WINNERS BY INDUSTRY Education | Financial services | Government | Healthcare | Manufacturing | Transportation | Hospitality | More WINNERS BY TECHNOLOGY Applications | Convergence | E-commerce | Management | Security | Servers | Storage | WAN | Wireless Download complete list of winners John Gallant on being an All-Star

For storm chasers, the excitement is in facing Mother Nature's wrath and coming out unscathed. The paradigm is similar for the IT team at Weather Underground, a Web-based weather service that experiences internal storms whenever bad conditions hit. For example, the number of page views on Weather Underground topped more than 14 million - triple the average - on Aug. 29, the day Katrina hit New Orleans. With users requesting animated radar maps that can be as large as 800K bytes, bandwidth requirements surged to more than five times the average, says Chris Schwerzler, IT operations manager at the San Francisco company.

To survive such traffic storms, the IT team devised a flexible infrastructure that can scale easily and deliver cost-effective bandwidth-intensive data as needed. Computing clusters built out of low-end, Intel-based servers running Linux give the Weather Underground processing oomph, while connectivity from Cogent provides big bandwidth at reasonable cost. Without this infrastructure, Weather Underground would not be able to deliver high-quality weather content and still be profitable, Schwerzler says.

"It would be hard to advertise and break even for the cost of the bandwidth," he says.

Weather Underground's efforts to create an inexpensive, scalable infrastructure to deliver weather data via the Web began in early 2000, coincident with a National Weather Service (NWS) policy change. Rather than restrict availability of that information to a few companies through private contracts, the NWS made the data widely accessible. "There was suddenly the availability of a lot of interesting radar data, and it was our project to develop an infrastructure that could scale and bring that data to a large number of end users," Schwerzler says.