Citigroup controls network resources

How Citigroup tackled network change and configuration management and won.

In August 2004, Citigroup’s Enterprise Systems Service team, which is part of the company’s Technology Infrastructure division in New York, realized that proprietary tools and manual efforts could no longer keep the threats caused by inconsistent configuration-management practices at bay. The team then led a two-phase project for the $17 billion company.

As simple as it may sound, maintaining an accurate, up-to-date record of network-device inventory, operating system and configuration becomes exponentially more challenging as devices multiply, vendors vary and data collected from the devices differs. Add to that challenge the numerous changes that occur on any given day – some of which may require distributing a patch to several routers and switches, for example – and IT managers face potential network failure, customer-service worries and imminent security threats.

According to market research firm Enterprise Management Associates, 60% of network downtime is caused by human error during device configuration. There’s also potential for error when real-time emergencies such as viruses or worms occur. To address the complexity of the problem, network change- and configuration-management vendors typically automate the process of collecting multivendor configurations and maintaining them in a database.

Citigroup’s project found the Enterprise Systems Service team exploring niche vendors with products that promised to eliminate the manual effort of collecting configurations in heterogeneous enterprise networks. These vendors put into software the dirty work – telneting into devices and scraping configurations, for example – typically performed manually by network operations staff. Such tools also incorporate configuration details garnered from equipment vendors, which reduces the need for device-specific experts within a single IT shop. “We were looking for a product that would provide all the reporting, governance, inventory and configuration features such as rollout and rollback, which we did manually, as well as some best-practice workflow and processes,” says a Citigroup IT official who, because of corporate policy, cannot be identified.

Following a $1.5 million investment in software, hardware and overall manpower costs for Phase 1, the company reports it began seeing benefits within three months of installing network change- and configuration-management software from AlterPoint. Within six months, the company significantly reduced the time it required to manage access lists across devices – from four or five staff members working for three days to one staff member working for three to seven hours. And Citigroup reduced manual remediation from 75% to less than 1%. Citigroup says the gain is 400% to 500% improvements in staff scalability.

Although it used AlterPoint DeviceAuthority as the core, the IT team built its own Web-based portals to address Citigroup-specific features and workflows that were not strategic for AlterPoint to develop or not planned until a later stage in the vendor’s road map, the IT official says.

Citigroup built extensively around the AlterPoint product to provide the overall solution that internal clients needed, leveraging the vendor API. The driving factor overall was to find a scalable solution, ideally one that was comprehensive enough to reconsolidate related features that Citigroup had decoupled to address the limits to scalability of industry and internally developed tools, Citigroup says.

Citigroup, which supports more than 44,000 network devices worldwide, was faced with “complexity inflation” and a lack of scalable management tools to keep complexity in check, Citigroup says.

Among Citigroup’s specific challenges were regular maintenance processes requiring multiple staff members and far too many man-hours. “We were spending 90% of our time dealing with compliance, making sure our processes and devices were compliant to regulatory and internal Citigroup mandates, in an ISO 9000-like environment.”

ISO 9000 is a worldwide quality standard, and certification requires businesses to have documented, repeatable processes for ensuring that they deliver quality products. Citigroup wanted a better method to keep its devices in line with not only ISO 9000 but also the Sarbanes-Oxley Act, other information security requirements and internal security policies.

Compliance wasn’t Citigroup’s only concern. The Enterprise Systems Services group also wanted to associate the company’s network devices to the business and tag assets with priorities relevant to Citigroup clients, both internal and external.

“We needed to be able to do business-tagging in relation to the clients that go through the device so we could quickly answer questions, such as ‘Is that device part of Tier 1, 2 or 3?’ to determine how critical the device is,” the IT official says. “That way we could more quickly determine where we could shut off the valve during a worm or other attack, before it affected the organization and the clients.”

With some 48 criteria to consider – including business, technology, product-support features and user reaction – Citigroup decided on AlterPoint’s DeviceAuthority Suite for its diverse device support and vendor stability.

The suite includes a server, a set of adapters and an Open Database Connectivity-compliant database. It has two application components, the Audit Module for inventory reporting and the Update Module, which automates mass configuration changes across any range of devices. The suite supports more than 1,000 network devices from 25 manufacturers, and audits in real time any change made to any of those devices.

Citigroup installed DeviceAuthority on servers in its geographically dispersed data centers and kicked off an internally developed device-discovery process, which reported back hardware and software configuration data from every device to be managed. This process compared devices from Citigroup’s internal network-asset inventory with those in its network fault-monitoring systems, and reconciled the devices’ configurations in the different systems against the devices themselves using DeviceAuthority Suite’s APIs.

With some 28,000 of its 44,000 devices being managed with DeviceAuthority, Citigroup says it’s poised to enter Phase 2 of the implementation, which will broaden the scope of the AlterPoint software to include other router and switch vendors such as Nortel and Juniper Networks, and potentially will add other IP-based devices that support the network, such as load balancers and compression appliances.