Automating change

The New Data Center demands easier, more reliable change-management processes.

There's an adage that says change is the only constant. Still, every IT executive knows constant change wreaks havoc on a complex IT environment. In fact, IDC and Gartner report that 70% to 80% of IT-related problems are directly attributable to changes made to the environment.

Mark Etherington, global head of distributed computing at JPMorgan Chase, New York, sees it firsthand. His firm makes thousands of changes to its 30,000-server environment each month. Like many financial firms, JPMorgan sometimes imposes change freezes at the end of the month or quarter to ensure optimal systems availability.

"When we have a change freeze, we see a reasonably dramatic drop-off of issues in the environment," Etherington says. "So we're faced with a conundrum. Do we prevent change to increase availability, or do we work out how to manage large numbers of changes better? The way to attack this problem is to make change a safer practice."

The problem is compounded as firms move to New Data Center technologies, such as virtualization. "As you build a more complex infrastructure to support things like server virtualization, you may think it's OK to be sloppy, since virtualization guarantees the service availability to some degree," says Richard Potocki, department manager of IT operations at Erie Insurance, in Erie, Pa. No one would notice if 25 out of thousands of servers fail because virtualization would cover for them. "But . . . an environment that allows me to be that sloppy has to be very complex. To manage that complexity, to make sure it works properly, you need to have really good change management," says Potocki, who has automated change management across his 285 servers.

Good change management relies on automation, specifically automating the change-management process while following the best practices laid out within the ITIL, users say. Strict IT business processes implemented via automation can increase the success rate of change, thereby reducing the number of changes necessary, eventually resulting in increased service levels across the board.

But getting to that point isn't easy. Many tools provide some automation but not of the entire change-management process. Ultimately, end-to-end change-management capabilities should come from larger firms, such as BMC Software, CA, HP, IBM and Symantec. Each of these companies, for example,is integrating the appropriate technology, often gained through acquisition, into their product lines.

Before choosing a change-management product, get a handle on current processes to ensure that they are as efficient, manageable and auditable as possible, users say. "You have to understand what you want and put it in the right context. Otherwise, you tend to lose focus," says Stephen Ashton, London CIO at Dresdner Kleinwort Wasserstein, a global investment bank that runs 10 data centers.

ITIL comes in by spelling out best practices for six main IT business processes - configuration, incident, problem, change, service/help desk and release management - aimed at providing operational efficiencies. A truly automated change-management process includes elements of each of these six. For example, you can't make a change without first addressing how it would affect the environment, and you can't do that unless you know what elements are in your environment and their dependencies. This is configuration management, for which ITIL specifies the use of a configuration-management database (CMDB).

Similarly, many changes are designed to fix problems in the environment, so processes and tools that tie into incident, problem management or help desk systems are key. And once a change has been approved, tying it to an automated release-management system ensures that its rollout is controlled, tested and auditable. Each ITIL best practice provides key functionality and accountability to the overall change-management system, which is paramount for organizations facing compliance issues.

CMDB tools guide Most big systems vendors are working toward fully integrated,end-to-end change-management database platforms.
ProductDiscovery methodFully integrated (from approvals to rollback)?Differentiator
BMC's Change and Configuration ManagementAgent-based and agent-lessNewly available, with its Closed Loop capability; fully integrated.CMDB is integrated with Remedy Service Desk; includes storage discovery; mainframe discovery planned.
CA's Harvest, AllFusion Endevor, Unicenter Software Delivery, Clarity Project ManagerAgent-lessDue in 2006.Includes automated project and portfolio management via Clarity.
HP's OpenView Change and Configuration Management, Open-View Service DeskAgent-based and agent-lessAcquired Peregrine technology; not fully integrated.Strong workflow via Peregrine; active CMDB technology uses desired state model to ease rollback, standardization.
IBM's CCMDB, plus Tivoli Configuration Manager and Tivoli Provisioning ManagerAgent-based and agent-lessDue June 30.Collation adds auto-discovery and application mapping; emphasizes WebSphere workflow.
Symantec's Veritas Configuration Manager, Veritas Provisioning ManagerAgent-basedStill integrating Relicore technology (Veritas Configuration Manager) with Veritas Provisioning Manager for auto rollout.Relicore is the market-leading CMDB.

"If you look at the procedures investment banks have gone through with Sarbanes-Oxley compliance, you find things where you thought you had controls, but you haven't got evidence. We've been working on the levels of application, general and entity-level controls, and the ITIL framework is a good response, especially in change management," Ashton says.

Users say automating change can't work without a CMDB, because automation requires a good map of your environment (see NDC Insight story). In a sense, the CMDB is a huge database that keeps track of every configuration item - be it hardware, such as a router, or software, such as an application release - in an environment. It also maps the dependencies among these items. So, if an application running on one server depends on a database on another server and uses a security appliance running elsewhere, then the CMDB shows all of those connections.

The problem has been getting that level of information in a single view and keeping it current. Today, there are several CMDB wares that provide automated discovery of configuration items and automated mapping of application dependency. These include products from BMC, CA, Cendura, HP, IBM, nLayers, Mercury Interactive, Symantec and Tideway Systems.

The tools differ, however, in their breadth and methodology. For example, CMDBs can use agent-based or agent-less discovery. Agent-based systems typically gather more detailed diagnostics and can be a good choice for critical infrastructure, JPMorgan's Etherington says. But in cases in which business units may be adding gear without IT's knowledge, agents are at a disadvantage. The CMDB can't work properly if agents never get installed. So the agent-less method is the choice at JPMorgan, which is conducting a large-scale discovery using Tideway's Foundation, he says.

Others say a hybrid approach, in which some discovery is performed via agents while the rest is done agent-less, is best. For example, IBM's Change and Configuration Management Database (CCMDB) uses Collation's agent-less discovery and combines that data with information gathered from its various enterprise monitoring systems, which are agent-based. This lets users receive more in-depth information from critical pieces of the infrastructure while ensuring that no piece goes undiscovered. Still the agent-less crowd contends that full agent-less systems are best, because they save IT the hassle of having to deploy agents on various boxes throughout the environment.

When it comes to breadth, most CMDBs can discover and map a wealth of server-based software, says Paul Schaapman, manager of technical services at the VA Farm Bureau in Richmond, Va. "But there are some applications they can't detect yet, especially the legacy ones. If I have a legacy app written in COBOL sitting on CICS, it's not going to get discovered. Nobody's considering the mainframe or AS/400s."

Joe Kennedy, vice president of IT architecture and planning at State Street Corp. in Boston, says he uses mValent's Integrity change management tool and agent-less CMDB technology within the Wealth Management division of the bank. The tool discovered most of the environment except for a proprietary accounting engine.

"If you're running Weblogic, WebSphere, JBoss , Apache - you just point it at the box, and it pulls out the configuration information for you," Kennedy says "But with the accounting engine, mValent spent a couple of weeks to write an adapter for us. Now, it pulls information and compares configuration files across the different instances of our accounting platform. But it took a little work."

As for the black hole of mainframes, a few vendors are addressing it. IBM contends, however, that its use of agent-based technologies, combined with Collation's agent-less discovery, covers all the bases. Meanwhile, BMC rolled out bolstered storage discovery capabilities last month and plans to release support for mainframe discovery in the fall, it says.

Once a CMDB is in place, organizations can tackle automating the rest of the change-management process. Each change can be analyzed using the CMBD's application dependency mapping features to ensure that the administrators responsible for supporting each item affected by a change are informed and included in the change-approval process. The best tools automate the whole process, from initial request through final approval. In some cases, such as with tools from smaller server-focused vendors, final approval can actually kick off an automated software delivery component, in which the tool then rolls out the change. The best tools also offer some kind of audit capability that tracks the change rollout and its success.

Erie Insurance uses BMC's Support Magic tool to automate discovery and application mapping, as well as approvals. Once a change is approved, the tool alerts technicians that they can roll out the change. When they've finished, they go back into Support Magic, answer a few simple questions, and based on their responses, the tool calculates a success rate, Potocki says. "Now the techs don't feel like they are being judged by management," and we get a reliable audit trail, he adds.

The whole point is to more easily and reliably document each change, track its approval and rollout, and then audit its success - all in an effort to improve overall service delivery to the business.

"Really, the objective is to measure what we do so we can improve," Schaapman says. "Setting these processes up positions us for that. That's our main driver."

Cummings is a freelance writer in North Andover, Mass. She can be reached at jocummings@comcast.net.

< Previous story: The many sides of application optimization | Next story: SOA governance: Preventing rogue services >

Learn more about this topic

Mercantile exchange uses automated change management

03/20/06

Insurer conquers change management

02/20/06

Automating change management

03/07/05

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:

Copyright © 2006 IDG Communications, Inc.

IT Salary Survey: The results are in