Citigroup tackled a Herculean task of subduing the multi-headed Hydra that emerges when large enterprise IT shops try to scale network change and configuration management software - and the financial services company came out on top. The financial services company submitted its project to Network World's Enterprise All-Star Awards competition , which honors 50 companies and their groundbreaking technology projects, and earned a spot among the winners for its large-scale rollout.

In August 2004, Citigroup's Enterprise Systems Service team, which is part of the company's Technology Infrastructure division in New York, realized that proprietary tools and manual efforts could no longer keep the threats caused by inconsistent configuration-management practices at bay. The team then led a two-phase project for the $17 billion company.

As simple as it may sound, maintaining an accurate, up-to-date record of network-device inventory, operating system and configuration becomes exponentially more challenging as devices multiply, vendors vary and data collected from the devices differs. Add to that challenge the numerous changes that occur on any given day - some of which may require distributing a patch to several routers and switches, for example - and IT managers face potential network failure, customer-service worries and imminent security threats.

According to market research firm Enterprise Management Associates, 60% of network downtime is caused by human error during device configuration. There's also potential for error when real-time emergencies such as viruses or worms occur. To address the complexity of the problem, network change- and configuration-management vendors typically automate the process of collecting multivendor configurations and maintaining them in a database.

Citigroup's project found the Enterprise Systems Service team exploring niche vendors with products that promised to eliminate the manual effort of collecting configurations in heterogeneous enterprise networks. These vendors put into software the dirty work - telneting into devices and scraping configurations, for example - typically performed manually by network operations staff. Such tools also incorporate configuration details garnered from equipment vendors, which reduces the need for device-specific experts within a single IT shop. "We were looking for a product that would provide all the reporting, governance, inventory and configuration features such as rollout and rollback, which we did manually, as well as some best-practice workflow and processes," says a Citigroup IT official who, because of corporate policy, cannot be identified.

Following a $1.5 million investment in software, hardware and overall manpower costs for Phase 1, the company reports it began seeing benefits within three months of installing network change- and configuration-management software from AlterPoint. Within six months, the company significantly reduced the time it required to manage access lists across devices - from four or five staff members working for three days to one staff member working for three to seven hours. And Citigroup reduced manual remediation from 75% to less than 1%. Citigroup says the gain is 400% to 500% improvements in staff scalability.

Although it used AlterPoint DeviceAuthority as the core, the IT team built its own Web-based portals to address Citigroup-specific features and workflows that were not strategic for AlterPoint to develop or not planned until a later stage in the vendor's road map, the IT official says.

Citigroup built extensively around the AlterPoint product to provide the overall solution that internal clients needed, leveraging the vendor API. The driving factor overall was to find a scalable solution, ideally one that was comprehensive enough to reconsolidate related features that Citigroup had decoupled to address the limits to scalability of industry and internally developed tools, Citigroup says.