By Neal Weinberg
Network World, 01/04/99

Mark McGwire isn't the only St. Louis pro who in 1998 delivered an awesome performance under incredible pressure. Scott Richert and his 18-person network team at Unity Health spent the year racing against the clock to merge the IS infrastructures at six hospitals into a seamless whole.

Unity Health, created by the 1995 merger of St. John's Mercy Health System, St. Anthony's Medical Center and St. Luke's Episcopal Presbyterian Hospitals, faced the stark reality that the software controlling its basic business functions was not Year 2000-compliant. That meant Unity Health hospitals would be unable to admit patients on Jan. 1, 2000. And after Jan. 1, 1999, the hospitals' computer systems would be incapable of scheduling checkups and other appointments for the following year.

The remedy was to construct a new enterprise network to support the STAR suite of Year 2000-compliant, hospital-specific applications from HBO & Co. in Atlanta. The project involved rewiring entire buildings with enhanced CAT 5 cabling, replacing terminals and undersized PCs with 4,500 new Windows 95 desktops and building an ATM metropolitan-area network to link five of the hospitals and a business office to a new data center.

Working long hours that sometimes stretched through the weekend, Richert's team met all its deadlines, overcoming obstacles that included an electrical fire, a WAN outage caused by a downed power line and a baffling router software bug. Five hospitals went live on the new system in 1998, and the remaining one has a drop-dead date of April 1, 1999. That's because the last hospital to get upgraded can limp along for a few months with a manual system for scheduling most Year 2000 appointments, but its obstetrics unit wants to be able to schedule deliveries nine months down the road using the new system.

"Some people would probably consider it suicide speed, but the reality is that we had to maintain the pace because of Year 2000 issues," says Jim Wesley, Unity Health chief information officer.

For his accomplishments, Mark McGwire already is recognized in the Baseball Hall of Fame. For the STAR project, Richert and his team can claim their own lofty prize - Network World's 1998 User Excellence Award.

The payoff

While hard return on investment numbers are difficult to come by - given the emergency nature of the project, Unity Health administrators aren't leaning on the IS department to provide a detailed ROI analysis - it's clear the $12 million IT investment is paying dividends.

New admissions software slashed the time it takes to admit a patient from 18 minutes to 7 minutes, creating the double benefit of improving customer service and using staff time more efficiently, Wesley says. Similarly, new table-driven forms reduced the percentage of patient bills that contain errors from 40% to 7%. That means far fewer phone calls to former patients to clear up billing errors, and that in turn means a major cost savings.

Sheri Beekman, director of accounts receivable, says the merger meant her group had to deal with four legacy systems, each with its own staff and call center. The process of filing insurance claims was so complex Unity Health used a separate clearinghouse to handle that function.

With the STAR system, the clearinghouse is no longer needed. Beekman consolidated call center staff around the integrated system and reduced staffing by about 15%. Bills are getting out in four days rather than 14, and collections are up 20%.

STAR also lets Unity Health take advantage of the economies of scale gained by the merger. The company wins significant discounts on bulk purchases of everything from PCs to pillows, Wesley says.

Wesley adds that he's barely scratched the surface. Applications being installed now - such as billing, payroll, accounting, admissions and purchasing - are basic to conducting business. Between now and the end of 1999, the IS team will be occupied with finalizing the implementation of those applications throughout all six hospitals and ensuring the Year 2000 problem is solved.

After that, the real fun starts. Wesley plans to focus on applications that improve operating efficiency and provide a competitive advantage. Among the items on the drawing board are:

• Rolling out telemedicine applications, including the capability to send radiology images across the network.

• Adding voice traffic to the ATM and frame relay networks.

• Giving patients smart cards that contain their medical histories.

• Creating a gigantic data repository that will hold patients' lifetime medical records.

• Implementing expert systems that can assist physicians with diagnoses.

A STAR is born

The STAR story begins with the August 1995 merger, which joined St. John's' two hospitals, St. Luke's' two hospitals and St. Anthony's; a sixth hospital was purchased early in 1996. Unity Health also includes senior health centers, a mental health treatment center, physical therapy and sports medicine facilities, a pain therapy center, a child development center, dialysis centers and a nursing home. It is the second largest hospital network in the St. Louis area, with more than $1 billion in assets, 12,000 employees, 3,000 beds and a total of 53 locations where patients receive some type of medical treatment.

In 1996, hospital administrators decided to create a single back-end system to enable the various Unity Health locations to present a single face to patients and physicians. Wesley, an industry veteran familiar with STAR, was hired late that year expressly to create a centralized IS department that could make that goal a reality. He is using STAR for all hospital systems except radiology, pharmacy and lab, where he is taking a best-of-breed approach.

Most of 1997 was spent in the planning phase. Richert, a network administrator at one of the hospitals, came up with a blueprint for the new network. His plan was accepted, and he began assembling his staff in May 1997.

"We had to reinvent everything," Richert says. The hospitals had a melange of technologies. Some had green screens, some 286 PCs; some had substandard wiring, others no wiring at all. There were four separate data centers. Plus, there were cultural differences: Some hospitals locked down PCs and didn't give users free access, others didn't care what software users loaded.

Wesley says the initial plan was to try and salvage as much of the infrastructure as possible. But the IS team finally came to the conclusion that ripping out the old wiring and starting from scratch was, in most cases, the only way to go. That meant a 40% increase in the project costs over the original estimate, Wesley says. But given the critical nature of the project, he says the hospital administration has been willing to provide the needed capital.

The first thing Richert and his team did was settle on some standard technologies to apply across the board. For most decisions, Richert went with the tried and true, rather than gambling with an untested vendor or an unproven technology. He bought desktops from IBM, chose NetWare for file and print services, and selected Cisco for routers and switches.

Cisco won the nod over Cabletron and Bay Networks (now part of Nortel Networks) based not on price, but on service, support and reliability, Richert says. However, Cisco offered an unexpected price concession.

Richert says he initially planned to run shared 10M bit/sec Ethernet to the desktop and to buy shared hubs from Bay. But Cisco wanted all of Unity Health's business, so it offered LAN switches for the same price as the Bay hubs. As a result, Unity Health now has switched 10M bit/sec Ethernet to the desktop.

Unity Health built a new data center, anchored by seven IBM RS/6000s. Because St. Louis is in a flood zone and is subject to tornadoes, the data center is on the second floor, protected by three-inch thick tornado-proof glass. Employees have to get through two levels of security to reach the data center.

It was the decision to go with ATM on the WAN that gave Richert the most pause. In 1997, not many companies in St. Louis and not many hospitals anywhere were willing to trust their mission-critical traffic to the public ATM cloud. Needless to say, it is especially important for a hospital network to be up and running 24-7.

"Yeah, it was a risk," Richert says. But he needed something that was flexible, scalable and able to accommodate future moves, such as the planned relocation of the data center. "ATM seemed like the way to go," he says.

Unity Health became one of the first customers of SBC Communications' ATM service. Although it took a few weeks to iron out the bugs, the service is performing well, linking hospitals that are 25 miles apart to the data center via a 30M bit/sec trunk.

However, Richert realized a fiber cut could

put the entire Unity Health network out of commission. So he hired Teleport Communications Group (TCG), which AT&T recently bought, to provide a second 30M bit/sec ATM connection. He's doing some rudimentary load balancing between the two, using one ATM link to support IP traffic and the other for IPX.

Last winter, when a truck knocked down a utility pole and put TCG's ATM service offline for 18 hours, users at Unity Health didn't know the difference because the routers are configured to provide complete failover capabilities.

Richert's ATM decision gave rise to an unexpected bonus. Various physicians' offices connect to the network via frame relay lines. But Richert was able to ditch the six T-1 frame relay trunks to the data center by using frame relay-to-ATM interworking to carry the traffic over Unity Health's ATM circuit, a modification that saves $9,500 per month.

The choice of Novell likewise worked out well for Richert. One of the key expectations of the STAR system was that doctors and patients would be able to move freely from one facility to another and

be recognized by the network. So a patient could

go to one hospital, get a referral to a second

hospital and not have to go through the entire admissions process all over again. And a doctor could log on anywhere and have access to the same applications.

Novell Directory Services (NDS) allows Richert to put those features in place. The combination of a

central naming database under NDS and a product called SFLogin from Netoria, an Orem, Utah-based company that makes network utilities, lets users log on from any station and gain access to any application for which they're authorized. And with NDS, it takes only eight network administrators to keep track of more than 4,000 users.

Richert is also using NDS for NT to manage a small pilot program for Microsoft Exchange e-mail, and NWAdmin is helpful in managing remote access for users who dial in to a modem pool.

One of the more vexing problems associated with STAR is the constant upgrades. Already Richert has upgraded each user's 35M-byte STAR client software package 10 times.

Manually updating several thousand PCs would be a nightmarish task, so Richert uses Novell's ZENworks to push client updates to the desk-

tops. He's also using ZENworks to provide the

"network menu" - a graphical user interface

that follows individual users wherever they go on the network, listing the applications available

to them.

High-speed problem solving

When it comes to implementation schedules, Wesley is like a general, driving his army of 220 staffers at breakneck speed. "The last 30 days before you go live, there are a lot of 70- and 80-hour weeks," he says. On the final weekend, staffers can be found working straight through, catching naps in chairs. "It becomes a monumental effort," he says.

The first hospital to go online was St. Anthony's, which needed $1 million in new construction alone. Electricians raced to install 1,000 new jacks in three months as the desktop team followed close behind installing new PCs. The network went live at 2 a.m., Feb. 1, 1998, and Richert's staff was there waiting for the phone to ring. It never did.

With one successful install under its belt, the team added a second hospital in June and two more in November.

The November implementations went so well that a special help desk, which was supposed to fix problems during the first two weeks of operation, was shut down after five days. The fifth hospital is going live the first week of January, Richert says.

As with any project of this magnitude, there were some rocky moments. The network is typically up and running several weeks before a hospital goes live, so there's time to test the applications. Just before St. Luke's was scheduled to go live in June, users reported the system was slowing down. Richert started measuring network response time and found delays were increasing, from between 2 msec and 4 msec to between 40 msec and 50 msec.

His team traced the problem to a bug in a Cisco 4700 router that left cache memory unprotected, so that another application was writing over it.

After Richert made several frantic phone calls to Cisco, it was determined late on a Friday afternoon that there was a Cisco engineer in California who could solve the problem and write a fix on Monday.

Richert demanded faster action and prevailed upon Cisco to have the employee work on the fix into the wee hours of Saturday morning. The Unity Health team then stayed through the weekend to install the patch and ensure everything was back to normal. "The whole place smelled like Chinese food for three days," Richert says.

Another time, a small electrical fire in a wiring closet caused some smoke damage to a router and the team had to go in, determine what was damaged and fix it.

Those small setbacks aside, Richert and his crew are feeling pretty good about their accomplishments. "We've got the highway laid, now we can run all kinds of trucks on it," Richert says.

No rest for the weary

Looking ahead to 1999, Richert's team won't have much of a chance to take a breather. He has to get the last hospital online by April, and the entire data center is moving in the second quarter. Additionally, 1,100 users who handle various back-end functions need to be brought into the new data center.

Richert still has to fine-tune the STAR system to make absolutely sure everything is Year 2000-compliant. And there are plans to link more physicians' offices and remote workers to the central database.

Plus, Richert wants to deploy sophisticated network management tools so he can move toward service-level agreements. Part of that plan means getting more out of ZENworks, using it to lock down desktops for easier management, obtain inventory information and handle some help desk functions remotely.

At this point, the network is overbuilt for the applications it is supporting. But Wesley is eyeing all the new applications he can implement in the future and is ever-mindful of Unity Health's expansion plans. "I don't want to be the one who goes back to my boss and says, 'You can't buy that seventh hospital because we don't have the network capacity or the processing capacity.' "

With the infrastructure Richert's team put together, it doesn't appear Wesley will face that problem.