Over the past decade Bank of America has grown by leaps and bounds internally and through an array of mergers and acquisitions. From a technical standpoint, that growth has created a complex and disparate set of data centers, computing architectures and vendor relationships.
For CTO David Reilly, there was an obvious goal: Standardize on more efficient infrastructure. For a company that spends $3 billion investing in new technology development each year – nearly double the amount it did five years earlier – any reduced expenditures translate directly to improved bottom line profitability for the bank. Transitioning to a shared virtualized computing platform not only drove savings in the IT organization, but net profit for the bank. But soon Reilly realized that standardizing and virtualizing was not enough. He wanted to start all over again.
Do like Facebook does
Bank of America had to get to a point where it is as agile, flexible and efficient as young, web-scale companies, Reilly believed. “It’s not enough to make what we’ve got better,” he concluded. “We actually have to get to a point where we could almost completely discard (our infrastructure) and move to a brand new, greenfield environment, unencumbered by any of the history or legacy of our existing footprint.”
Reilly isn’t alone in thinking that legacy infrastructure just isn’t cutting it anymore. In 2011 social media juggernaut Facebook started a community foundation named the Open Compute Project (OCP), which aims to create a set of standards for commodity infrastructure components that organizations can assemble together themselves. That’s opposed to buying racks and stacks of infrastructure from the traditional IT vendors.
Reilly liked the idea. But just ripping and replacing was not an option. Bank of America has 100,000 technology and operations workers and contractors serving 19 million mobile banking customers – growing at 5,500 per day. Reilly’s technology infrastructure team manages 169 petabytes of data. The legacy system couldn’t just be shut down.
So about 18 months ago Reilly created a team that began rethinking everything – from how a machine was built to how it was managed. “Software-defined technology was the way to go,” he says, harkening to the philosophy of the OCP – whose board includes executives from Facebook, Intel, Goldman Sachs and the co-founder of Sun.
The new cannot look like the old
Bank of America took a two-pronged approach to starting what Reilly calls the bank’s greenfield project. One path used a vendor’s proprietary infrastructure stack (Reilly would not disclose which provider). The other followed standards being developed by OCP and other open source projects.
Both embrace software-defined infrastructure and follow core guiding principles: Compute, storage (and eventually network) resources are provisioned via APIs; applications use containers; and data centers are built in modular components.
+ ALSO ON NETWORK WORLD Bank of America sees software-defined data centers as 'irresistible' +
“We’ve held a very hard line to say that the new cannot look like the old,” he says, even delaying the project at times to comply. The goal is to one day have a system built entirely on commodity servers that are uber-flexible: They could act as compute servers one day, a network switch another and be part of a pool of storage resources if needed. Containerized, micro services-designed applications automatically provision infrastructure resources they need without requiring any human hands in the process.
The proprietary stack is running about six-months ahead of the open source one because technology innovation by private vendors has been faster than the methodical yet essential work of the open source community.
About 95% of the applications that have migrated to this new greenfield environment have thus far been written to the proprietary stack. But Reilly hopes in the coming years as the open source innovation picks up, that will become the bank’s dominate platform. Open source software-defined networking is one of the laggards, Reilly noted.
All eight lines of business within Bank of America are already using the new platform including wholesale banking, development and quality assurance. There are plans to migrate additional grid-based workloads, while pockets of the wealth management division have use cases for it.
“We are now convinced that this software-defined model, where an application presents a manifest for the assets and facilities it needs, it acquires those assets with a call, uses them, then lets them go when they’re done so they can be used by someone else – that’s going to work for something like 80% of our technology workloads,” Reilly says. The other 20% of workloads will run on more statically provisioned machines and a mainframe that the bank still uses.
Matt Eastwood, senior vice president at IDC, says financial services firms are the next logical candidates to embrace this type of technology after the web-scale businesses. Businesses that do drug discovery, oil and gas exploration, aerospace and defense could all have workloads that favor a dense, scale-out, cluster-model compute environment.
“Any cost savings on the CAPEX and open side of the equation can be re-invested back into the computing environment in an attempt to get a competitive strategic advantage in the market,” Eastwood says.
Already Bank of America’s greenfield system has created enormous efficiencies in how application developers do their jobs. Reilly positions developers as pseudo micro-CIOs who serve their individual business units, and are charged back based on infrastructure usage. “A developer will move to this environment initially to get the price benefit,” says Reilly. “But the developer stays because of the flexibility and agility.”
Change isn’t easy
Oddly enough, Reilly says the biggest challenge in executing this greenfield program has not been a technical one. “This is not a criticism of the way we’ve been doing