Upgrade ingenuity

To fully appreciate the network project that won insurance giant Royal & SunAlliance USA the 2002 User Excellence Award, put yourself in the shoes of Roger Thibodeau, chief network executive. Your brand new CTO, Chris Heeley, was hired in May 2001 to oversee strategic infrastructure planning. Heeley almost immediately must break his sacred keep-it-simple project management rule with a $3.2 million doozy for which he's named you project manager. You've got to upgrade 7,000 Windows 95 PCs to XP while your network staff simultaneously rolls out Active Directory as the core of a new identity management system, switches to a new software distribution server and implements self-service password management, among other tasks.

Your first thought was to get help from the outside. But a cost estimate based on rates and timelines from your usual contractors tallied $1.5 million with a sometime-in-2003 completion date for the operating system cutover alone. You've only got until December 2002 to complete the project, from planning through completion - with a scant four months to upgrade 7,000 PCs.

You're certain that the network can be your workhorse. With a little customization done by your staff on XP's self-installation feature, Remote Install Service (RIS), the network can push the install program to the clients without CDs. (You shove from your brain the risk in choosing XP, which at the moment - September 2001 - Microsoft hasn't released for production yet. You've beta-tested XP, and know you want the operating system for its remote control and other built-in features. Plus, Microsoft has revealed that it will shortly cease supporting your other option, Windows 2000.)

But who would initiate the install on each machine and do the backup, restore and troubleshooting throughout your 94 offices, including the Charlotte, N.C., headquarters? You and Heeley conclude each user is the most expedient choice, although many R&SA USA employees are not especially computer-literate. The process must be so simple that users can upgrade their PCs themselves with minimal oversight, regardless of technical prowess.

Mission impossible? The network infrastructure team proved not. Not only did R&SA USA's team conduct a massive self-service operating system cutover, but in the process, it also certified that the company's 450 applications would work with XP, inventoried IT resources and instituted a corporatewide software license compliance program.

Better still, the project converted business managers and end users into partners for software license compliance and application training. This while making infrastructure folks into corporate stars as employees gained a better understanding of IT processes.

And not one person involved filed a complaint against the project to the CIO or other senior managers, Heeley says. End users actually begged for their turn at the upgrade, Thibodeau adds.

For its creativity in crafting an almost painless operating system cut-over process that saved $1.5 million in contract labor fees while improving security and turning end users into IT partners, R&SA USA wins the 2002 User Excellence Award.

Picking the winners The User Excellence Award, in its 18th year, honors user organizations that demonstrate exceptional use of network technology to further business objectives. Network World editors select winners from a pool of entries submitted by user organizations or vendor representatives. More

From liability to asset

The plan was either ingenious or a death trap. "We had considerable opportunity to fail. It could have been the shortest job of my career," Heeley quips. Begging off for more time was not an option. The CIO mandated the fixing of problems discovered during a security audit conducted in the fall of 2001 by R&SA USA's London-based parent, the seventh-largest property and casualty insurance company worldwide. R&SA conducted this audit as part of a global effort to implement international computer security best practices, which has led the company toward security seal-of-approval, ISO 17799 certification. Corporate gave R&SA USA until December 2002 to comply with the auditor's recommendations.

A new client operating system fixed a major flaw discussed in the audit: Windows 95's weak passwords, bypassed simply by clicking cancel. Thibodeau cites other reasons for the move to XP - support for new equipment such as digital cameras for insurance adjusters and built-in remote control features, good for assistance from the help desk and application training from business managers.

Building a software mall

A wayward software distribution program, Novadigm's Enterprise Desktop Manager (EDM) software, also needed immediate fixing. EDM, purchased in 1997, had two nasty habits. For one, it overwrote Dynamic Link Libraries (DLL) used by older DOS and mainframe applications each time it pushed out new code, such as antivirus updates. That meant after each EDM session, mission-critical applications on people's desktops would crash, Heeley says. Second, when updating, EDM assumed control of target PCs at the next login, even when that meant pushing an enormous file at a user dialing in from the road for a quick e-mail check.

"It was so bad that someone actually put a sign on one server that said, 'Turn me off on Thursday so I don't get EDMed,' " Heeley recalls.

In August 2001, while in discussions with Novadigm's CEO over fixing EDM, Heeley learned of Novadigm's new product, Radia. Like EDM, Radia pushes out software updates, but gives users three times to refuse a download. More importantly for this project, end users can download their own applications, per policies and licenses, from an intranet "software mall."

R&SA USA took stock of all software and hardware assets using Peregrine Systems' Asset Management software. It discovered 450 applications to place in the mall. "We found software and media that we didn't even know about," says Tricia Laurion, a network engineering project manager. Before this, R&SA USA had no such inventory. The inventory even became the basis of another money-saving project. Thibodeau says he is negotiating higher-volume agreements on licenses for duplicate mapping software and other titles that individual offices buy.

So Radia solved two problems for R&SA USA: the software license control issue named in the security audit, and the challenge of how end users would reinstall applications during the upgrade.

With the technology means for the self-service cutover materializing, Thibodeau and Heeley turned their attention to the business side of the house. They enlisted the support of Karen Martin, who manages the 26 facility managers overseeing R&SA USA's remote sites. With her backing, those facility managers became project touchstones, preparing their sites for XP deployment with duties such as scheduling upgrade days, arranging for teleworker updates and being the XP guinea pigs. "She made the facility manager's role as site liaison part of their jobs, included it in their performance review. We could never have done this without that kind of support," Thibodeau says.

Martin adds that her crew was an obvious choice because many are technically adept. Some are even former IT members from the days before R&SA USA outsourced its help desk to IBM, when all large offices had a full-time IT staffer.

By October 2001, the infrastructure team had completed the major steps of the self-service plan and had begun work on the several large issues remaining. Remembering the pain of EDM, Thibodeau created a team to certify that any application put into the Radia software mall would work with XP and Internet Explorer 6.0. One of the team's tasks would be fixing DLL conflicts, Thibodeau says. But should applications still break after installation, Radia's "self-healing" feature would repair broken applications automatically the next time a user logged on to the network.

Getting active

R&SA USA also had to roll out Microsoft's Active Directory because Radia relies on the directory service to store a user's identity and software entitlement information. The company had used Active Directory in a limited fashion to support a few Windows NT application servers and about 400 Win 2000 machines, but certainly didn't store in it the meaty data to determine every user's software entitlements. If that data was anywhere it was strewn across directory systems - Novell Directory Services (NetWare is used for print and file sharing) and X.500 for Unix systems - or stored within mainframe programs, Lotus Notes or other applications.

This Active Directory rollout also became the foundation for an identity management initiative. Active Directory will become the U.S. company's metadirectory - eventually tying into a global metadirectory - storing employee passwords, locations, job roles and hardware inventories, all of which help determine software entitlements.

Eventually, Active Directory will tie into R&SA USA's PeopleSoft human resources application so as the HR database is updated, the changes are passed to Active Directory. In turn, Active Directory will update the user's profile and alert other systems involved. R&SA USA has made its software distribution investment do double duty, gaining identity management without buying another, complex provisioning platform often costing $1 million to implement.

But first, R&SA USA had to solve a troublesome password management problem. Users had on average six passwords apiece for accessing network operating systems and applications, Heeley says. They would need all of them to reinstall their applications. But the IT team knew many wouldn't remember all their passwords, given that at the time password-related issues accounted for 40% of R&SA USA help desk calls, he adds.

Heeley searched out password management products, and in April, less than a month before the first self-installation upgrade, settled on M-Tech Mercury Information Technology's P-Synch. Tapping into Active Directory, P-Synch could synchronize passwords so that one would suffice. It would then enable ongoing self-installation resets, reducing help desk calls and related outsourcing costs (see "Identity management begins with the humble password").

Only three questions

The team turned to IBM's System Migration Assistant (SMA) for a back-up and restore tool friendly enough for end users to control during the cutover. "We had to wipe and load. Microsoft doesn't let you upgrade from Windows 95 to XP," says infrastructure team member Mike Johnson, of the necessity to format each hard drive as part of the upgrade process.

SMA backed up to and reinstalled from a network server. Concerned about security, the team configured SMA to upload data files to user-specific home directories on the server. SMA also saved the user's "personality elements" such as screen savers, Thibodeau says. The latter "was such a hit with the users [that] they were more apt to tolerate blips during the conversion." The network team further customized the off-the-shelf product, configuring it to run in batch mode, bypassing the tool's graphical user interface. The team wrote scripts to automate responses to SMA prompts and altered numeric messages to human-friendly messages, "so users would know they were getting a good backup," Johnson says.

A custom script also made the XP install a truly unattended process, answering XP's pop-up questions to keep the install from pausing.

Ultimately, end users launched the entire process - backup, hard drive format, password synchronization, operating system upgrade and installation of mandatory applications - by answering three simple questions from a custom script: What type of PC do you have: desktop, laptop or home PC (to determine if VPN or remote services were needed)? Where are you located (to determine what server to use)? And, what's your main network ID and password? Known to all, that password would then sync with others.

The process took about two hours, during which IT trained users on XP, Radia and, if one of the company's nearly 1,200 teleworkers, also on XP's VPN-IPX and VPN-IP services.

After training, users followed written instructions to restore data by typing in a .bat command and downloading programs from Radia. Along with the restore, Radia automatically downloaded corporatewide mandatory programs such as word processors and e-mail clients. IT typically needed only one person onsite to be the "XP coach" for user questions. In all, only 26 IT people, working part-time on this project, took part in the actual cutover.

Servers for all

R&SA USA needed to modify the network only slightly. At headquarters, the infrastructure team installed an Active Directory server, a main Radia Manager server and an Oracle database for inventory information on its standard fare, Win 2000 rack-mounted Compaq servers.

The team issued each remote office its own low-end Compaq, Win 2000 Server to house Radia's software mall, forgoing back-up servers to rely on those connected via the WAN for failover. Now IT in Charlotte can upload applications or patches to their local Radia Manager. Then, in off-peak hours, the Radia servers will synchronize. Local servers also house XP's RIS. So downloads, from the XP install to ongoing antivirus updates, are pushed to desktops over high-speed LAN links, not the slower WAN.

While network executives know they can't be separate from the business, rarely is the converse true. By creating a process that eases the pain of a cutover, then trusting employees to do their part, IT gave colleagues a rare glimpse into its world.

"Employees now have a greater appreciation for what has to take place in IT," Martin says, adding that ultimately the people, not just the technology, caused success.