How to roll out 180,000 VoIP phones

Craig Hinkley is in the thick of a 180,000 IP-phone rollout at Bank of America, which is replacing more than 460 PBXs with a centralized Cisco VoIP system that serves 6,000 U.S. branches and back office locations. Hinkley, named last year to our list of the Most Powerful People in Networking, took time out from flipping on the VoIP switch to give a project update to Network World Senior Editor Phil Hochmuth. The following is an edited transcript.

The bank announced plans to roll out VoIP in 2004. How is your deployment structured, and what stage are you at?

It's been going extremely well. In essence, what we've done is create three programs under our transformation umbrella: branch/retail, enterprise and call center. We've been moving at a steady pace on our retail branch deployment. We've [also] done a good number of enterprise locations this year. For our call center deployments, we're going to go into a pilot in the next four to five weeks before the end of the year. Then we'll be ramping that program up in 2007.

There are different scopes and capabilities in each of those major segments of our business. Our branch enterprise represents 6,000 stores we need to go and complete the transformation in. We focused on that first in order to make sure we get those numbers rolling.

How far along are you in that transformation?

In the branches, we have around 800 stores we've converted. In our enterprise deployment, we have around 50 enterprise locations we've completed today … enterprise locations are back-office type facilities. [More than 20,000 IP phones are deployed overall].

How do you approach branch office and enterprise deployments? These must have very different requirements.

Yes, that's fair to say. The standard Bank of America branch has a small number of phones, vs. a back-office location, where you might occupy one, two of five floors of a multitenant building with over 150 people per floor.

One of the things we focused on in the first part of our deployment program was building the transformation process, what we call the "T-minus schedule." So when we do a cut-over, we work backwards towards that T-minus date. At a certain point in time, we go out to a location and do the survey of what's out there. We order new circuits or new capacity that's necessary, and build out the templates we're going to use in that location. If there's any pre-staging work that has to be done, we do that during standard outage windows at the bank [such as off-hours]. We do all that preparation work, and we combine that on the people side and then bring the training to bear.

Before we actually convert the branch, we provide training to the team leader for that banking center. So that person becomes the trainer of the trainers. Then that bank person turns around and does some training internally. On the night of the transformation, we have the SWAT team [from EDS, the bank's network integrator] that comes in. They close the branch at night, and when it opens in the morning, it's using a brand-new technology standard for our VoIP deployment. Associates are on-site the next day to answer questions or concerns that arise the next morning with the new technology. So we do all that. We do it flawlessly, and we're going to repeat it 6,000 times.

And in the enterprise locations?

The main difference between an enterprise and branch is the number of phones involved. But what it comes down to is understanding in those enterprise locations, the interdependencies of how phones are set up to interact with each other. For instance, line coverages, who is part of a hunt group on a floor — there is more potential variation in an enterprise site in terms of how the phones are used. One admin might be supporting five or six managers in that area, and might have coverage for someone else on another part of the floor. So the key part is making sure we have the appropriate assessment process to capture all of those requirements, so we can translate them into a standard VoIP solution.

Other than that, you have the benefit of having a team working in a confined space. From a transformation perspective, when you hit one location, you get a larger volume of numbers converted -- which can be a bad thing, or it can be a good thing. You can get more done by having a team in that concentrated area.

So with 800 branch conversions completed, what have you seen and learned so far?

With something of this magnitude, you're not just changing the technology in the back room. What you're doing has an impact on the way associates are using technology every day — it's a little more disruptive. So we've had some lessons learned around the training, and how we communicate the use of the technology, and where we're moving to some more standard business-process-driven capabilities. We're making sure associates understand the training around how we're now using the phone system to execute business processes, and not just as a phone with basic features and functions.

The key thing we quickly learned at the start of the project was we had to unravel the heritage PBX environment that for years was just providing phones and dial tone. We had to understand how a phone system was being used to support the business. And then from there, we had to understand what the standard telephony integration models were going to be. If we just went in and replaced what was there … we would just end up replicating the 460 PBX deployments we have today. That was a key moment for us, to say, we need to make sure we're building this to support our business model and strategies across the whole U.S. branch franchise.

Do you use a lab that replicates a branch setup for testing and planning?

We have a lab where we build out a standard solution. Our approach is to have standard templates and standard solution sets that we deploy. We're not trying to make each deployment unique. We're very much as Six Sigma company — so variation is the enemy. We don’t want variation of anything in the VoIP project. Delivering ubiquitous capabilities allows us to be consistent with our customer interactions. We're looking to drive the appropriate technology to support the business, and provide a common set of interactions with the customer. So as they interact with our associates, and our associates use the technology to fulfill business transactions, then it's consistent.

What are some examples of the business transformation you're gaining with IP telephony?

What's been interesting is it's some of the little things that I didn't realize people would be so delighted about. For example, something as simple as a missed call indicator. For an associate to realize they have a missed call, and to go into the phone and see the number of the call, and being able to call back that number — which is typically a customer. People have asked me what is the killer application? I say, by deploying a VoIP system, we're getting just out-of-the-box features that people take for granted with VoIP. But when you come at it from the context of coming from a traditional key system, or PBX, this is all new.

Being able to go into a Web page, and configure speed dials. And if I'm out of the office, the ability to go to a Web site and forward my phone to my cell phone. That's a quantum leap forward for a lot of people. … For us, it's realizing that it's more than dial tone, and utilizing some of the out-of-the-box features with VoIP have produced a positive effect. Even linking it to the corporate directory, and being able to dial any associate in the bank from a phone. All of that has been positive.

As one of the early-adopters for large-scale VoIP, has anything changed in the technology since you started the project?

From a technology perspective, we were very comfortable with the stability, and availability and reliability of the Cisco platform when we selected it two years ago. If we weren't we wouldn't have made that decision. With that said, they've added capabilities and features into the product.

What's changed is you see a lot more VoIP stories in the marketplace. Were we the early adopter, or the front-runner? We certainly made a strategic decision two years ago to go to VoIP. I believe based on the size of our deployment, we needed that [large amount] of time because we had such a massive transformation to undergo.

At the same time you're seeing in the industry, changes such as what we consider non-traditional telecom companies coming into VoIP -- our friends at Microsoft for example. There's discussion around converged services and unified communications. So that, to me, is the next phase of IP telephony. The buzzword is "interaction management." As Bank of America looks broadly across our customer base, we're managing relationships and interactions. So how do we manage those interactions if it's in a banking center, a call center, or via telephone or e-mail or instant message. It's not so much VoIP.

Are unified communications part of your VoIP deployment?

We have not deployed any real unified communications capability. We're deploying our VoIP, our basic IP telephony platform, and we're focused on that. We've got a multiyear strategy around all of this. We're now looking at our road map to layer in the other capabilities as we get the deployment penetration into the branches and the enterprise. We're looking at how we move those capabilities to meet the business needs. We have some of our internal business partners who are looking for certain IP telephony capabilities. We're looking to provide things such as flexible workplace programs. Or how do we enable an associate to have mobility that provides a single telephone number? We need to provide that advanced capability to complete our transformation program.

As you turn on more sites, how is this affecting the overall IP network?

Certainly, it's affecting the network in that we're carrying more traffic over our internal communications infrastructure. As part of the deployment process, we're ensuring we have the appropriate bandwidth capacities available on the connections to our branches. Right now we have a dual OC-48 backbone with certain segments running at OC-192.

We're deploying a standards-based solution into the branches, so the bandwidth needs for each branch is pretty well known. Typically, we're using the existing WAN circuits to a branch. If it’s a Frame Relay circuit, we may lift up the [committed information rate] in that connection.

How is your new centralized IP voice mail system used differently from the bank's past messaging technology?

Not many people ask about voice mail throughout this whole project. It's interesting. In our branch environment, voice mail was used. But unfortunately, it was not a well-integrated solution. As we move forward, having that voice mail capability at the branch and being able to send out a single message to all our VoIP branch locations — that's a key benefit being used by our division operation executives who run the branches; they're able to send out a message of the day. From their perspective, voice mail is a key way of delivering a very hands-on message to associates in the branch.

What is the cost impact of the project?

The reason we're undergoing our VoIP transformation, besides the ubiquitous capability and standardization, is to gain cost efficiency. We are seeing cost efficiency out of our converged VoIP environment. We're seeing it operationally, and with things such as moves/adds/changes. As you provision the environment, we're seeing reduced costs for supporting that.

Can you say that a given branch, prior to a VoIP cutover, had X amount of cost associated with it, and after, the cost is Y?

When it gets down to that branch level, I'd say we do not have a good quantifiable number. But we do have solid numbers at the top of the house around the VoIP project, in terms of costs, expansion and the associated savings. Unfortunately, we can't get into how much we're saving. But I can say that if it was not generating significant cost savings, the bank would not be executing such a large technology transformation.

With Bank of American being in such as competitive market, do you keep up with what other big banks are doing with VoIP and IT? Ever pop into a CitiBank and see what phones are on the desks?

Obviously, Bank of America operates in a very competitive marketplace. I'm confident that we have market intelligence around what our competitors are doing. We do look to understand how they are using technology as an enabler within their businesses. We do that to make sure that we're staying competitive, but other than that, we're also focused on making sure we're being innovative with technology and that we're continuing to be seen as a leader.