WaMu banks on MPLS

2003 User Excellence Award winner Washington Mutual powers up for rapid expansion with a massive Multi-protocol Label Switching network.

Bankers as a group tend to be risk-averse. So in June 2002 as Washington Mutual moved toward finalizing contracts for a huge Multi-protocol Label Switching-based network, CIO Jerry Gross was getting nervous.

The nation’s seventh-largest bank was poised to commit millions to the project, a gamble of sorts on a technology used more by big carriers than corporations. In fact, WaMu wanted MPLS for the same reasons a Tier 1 ISP does – it scales readily; segments easily, with connections and teardowns handled automatically; and handily accommodates big surges in demand, says Nabil Badr, the company’s resident protocol genius, who architected the network.

Confident as Gross might have been about his backbone choice, he was worried about the stability and viability of the primary carrier with whom he was negotiating – MCI, then known as WorldCom. With word on the street putting MCI in deeper financial trouble than its executives were letting on, Gross flew to the carrier’s New York headquarters for assurances. Over dinner, then-CFO Scott Sullivan convinced Gross that the rumors surrounding the carrier were baseless and that everything would be fine.

So Gross was probably more shocked then most when, less than a week later while watching CNBC in his Seattle office, he saw law enforcement officials leading Sullivan away in handcuffs. “I’ve been involved in some white-knucklers in all my years in business, but this one took the cake,” Gross says. It didn’t help that soon after WaMu Chairman Kerry Killinger stopped by Gross’ office to ask, “How you feeling this week?”

With MCI in such dire straits, Gross needed to get even more pragmatic and contingency-minded than he had been regarding this massive network overhaul. Gross and his network staff still believed that the carrier was the best in the business to provide MPLS. ISP UUNET, acquired by MCI, pioneered the technology in the 1990s. It uses MPLS to run its own massive IP backbone, which comprises much of the public Internet.

So rather than ditch MCI in favor of AT&T or another carrier as the primary service provider, WaMu structured its MPLS contracts assuming MCI would go broke and cease operations. That is, it spelled out terms for shunting all WaMu traffic on MCI’s backbone to back-up carrier AT&T. The MCI and AT&T contracts also contain such unique terms and conditions as the ability to change the mix of services without penalty at any time, Gross says.

The financial services company had plenty of incentive to get this MPLS project right and minimize risk. WaMu’s business spans consumer banking, mortgage lending and insurance in the U.S., with assets totaling more than $286 billion. The MPLS network, connecting 3,000 locations and the 60,000 people who work at them, would help fuel unprecedented business opportunities. WaMu, for example, relied on the MPLS network in opening 32 bank branches in Chicago in one day in August – a figure of which the company is justifiably proud.

“Our network takes shocks like adding that many branches, and still the needle stays on ‘normal,’ ” Badr says. “MPLS can take it.”

WaMu wins the 2003 User Excellence Award for successfully undertaking this massive enterprise MPLS deployment to meet the company’s rapid and widespread expansion, while significantly boosting the efficiency of day-to-day operations. Conservative cost estimates for such a large-scale MPLS deployment range in the tens, if not hundreds, of millions of dollars. While WaMu officials won’t divulge budget information, they did verify that first-year ROI was a 25% reduction in IT operational costs, a figure they forecast will remain constant over the next three years.

Company at a crossroads

While few companies can warrant such a massive network overhaul, WaMu was at a crossroads. Gross describes the situation he encountered his first week at WaMu, in August 2001: Intranet pages took three minutes to load; the network went down a lot, but no one could explain why; and remote connectivity was so bad that he opted to use Hotmail when working at home rather than try to access the corporate e-mail server.

Then consider the thousands of bank tellers trying to get account information, or the multitudes trying to close on the best interest rate, or the financial services company’s inability to handle its own internal business processes. Applications had been developed and then shelved because the data infrastructure couldn’t support them. Long outsourced to IBM, that data infrastructure had been intended to support nothing more than general office applications such as file serving, e-mail and accounts payable.

Part of the problem was that the old network couldn’t scale enough to meet WaMu’s rapid expansion – an acquisition binge started in the late ’90s wrapped up with purchases of three financial institutions in the past year. In the past six years, the number of WaMu employees more than doubled from 27,000 to 60,000. The number of locations also more than doubled, to 3,000 nationwide.

“It was a burning platform,” Gross says. “There was a basic need for a scalable, stable, hardened network that could support WaMu’s growth.”

And although WaMu has scaled back acquisition plans, continued growth there would be. The company planned to open as many as 300 financial centers through 2004. WaMu was willing to prioritize IT spending on a companywide data network that would span local, metropolitan and wide areas – and eventually voice and wireless data traffic. Plug and play of new sites was an imperative, given the company’s growth projections.

“MPLS was the choice for flexibility and scalability, which makes for a better total cost of ownership,” says Badr, noting the company also considered fully meshed hub-and-spoke and ATM-based backbones.

And so WaMu created a six-tier architecture using the MCI and AT&T services and network gear from Cisco and Juniper.

The new LAN architecture provides 75,000 Ethernet ports (a mix of 10M, 100M and 1G bit/sec links) at branch banks, financial centers, home loan sites, corporate office buildings and for automatic teller machines. Each WaMu site connects to one of multiple metropolitan-area networks (MAN) for connectivity to the WAN. A WaMu-developed node, called a supercore, further consolidates MAN traffic across any of three different SONET OC-3 lines, which operate at 155M bit/sec. From the supercore, traffic passes onto any one of nine MPLS nodes, which in turn collapse into a three-node backbone. Back office, system-to-system and dial back-up layers round out the six-tiered WaMu network (see “Tiered MPLS“).

Design and testing began in the fall of 2002, with implementation following through this year.

Supercore to the rescue

WaMu holds network specifics close to the vest, not willing to share too much information for competitive reasons. But Mike Spalter, team leader for WaMu’s Infrastructure Services Group, did explain how the company ensured such a smooth rollout. WaMu partitioned the supercores from each other and from the rest of the network until they were certain each was stable, he says.

Also critical was teamwork among WaMu’s IT staff and the MPLS players. Spalter describes 60-person meetings at which all the vendors were represented so blame couldn’t get shifted and everyone could agree on cooperative action. Offloading a lot of responsibility to its vendors helped WaMu achieve a “remarkably” smooth and quick implementation, the IT folks say.

To ensure the network continues operating seamlessly, WaMu holds monthly quality improvement meetings during which vendors present their network statistics in front of each other. This format, Spalter says, introduces an even higher level of accountability and spreads the risk around. “It gives the frontline vendors the tools they need to manage the back office,” he says.

WaMu and its carrier partners manage the MPLS network from three network operations centers (NOC) – one each owned and operated by WaMu, MCI and AT&T. All the major vendors and service providers have full-time staff working in WaMu’s NOC.

Leveraging the infrastructure

Gross and the network team are under no small pressure to leverage the MPLS backbone. And while it’s not a front-burner action item, WaMu is looking at the MPLS network as a means to handle voice-over-IP traffic across the corporation and to be able to create virtual call centers rather than invest in physical infrastructure for customer service. Gross also sees the MPLS backbone as a means to tie local wireless LANs to the MAN and WAN, and even broadcast interest rate changes to bank customers or inform them of other market news.

But for now, Gross and his team have their hands full. The good news is, what looked like a huge risk in June 2002 already has begun paying off for WaMu.

“Downtime is all but gone,” says Gross, citing a network uptime statistic of 99.997%, up from 97% when he joined the company. “And we can deploy new capabilities much faster. Our operational excellence wouldn’t be possible without IP applications. How do you calculate the cost of not having an intranet – millions of dollars? MPLS has transformed how we grow and how productive we are.”

Sweeney is a Los Angeles writer and editor who has covered IT and networking for 20 years. He can be reached at terry@tsweeney.com .