At a Boston hospital, lessons learned from Slammer

While organizations around the world this week scrambled to disinfect and patch systems that had been hit by the dangerous new W32.Blaster Internet worm, John Halamka could sit back and relax.

“I’m proud to say we don’t have a single copy (of Blaster) in the hospital,” said Halamka, CIO of Beth Israel Deaconess Medical Center (BIDMC) in Boston, a Harvard University research institution.

BIDMC’s good news on Blaster came amid reports of the new worm’s continued spread on Wednesday and the appearance of new worm variants on the Internet.

It was also a marked contrast to the scene at BIDMC in January after the SQL Slammer worm crippled the hospital’s computer systems for about six hours, forcing medical staff to resort to paper-based records to track patients.

With Slammer, Halamka’s staff patched their Microsoft SQL Server software prior to the worm’s release. However, like many Microsoft customers, BIDMC was blindsided by an overlooked component on Windows XP desktop machines called the Microsoft Data Engine 2000 (MSDE). MSDE was also vulnerable to SQL Slammer and the worm was able to infect computers in the hospital’s research labs and private offices. It then flooded the rest of the network with traffic, according to Halamka.

So when Microsoft warned customers in July of the RPC (Remote Procedure Call) vulnerability – the one later exploited by W32.Blaster – Halamka and his staff weren’t taking any chances.

BIDMC staff updated their network firewall configurations to block the ports, such as 135 and 4444, which were identified by Microsoft as avenues that could be used to exploit the new vulnerability, according to Halamka.

The hospital also promptly updated its intrusion detection systems with the appropriate signatures to detect and warn about traffic associated with scans for vulnerable systems, according to Kristofer Karas, senior security engineer at BIDMC.

Then came the patching.

Not wanting to fall into the same trap as with the Slammer worm, Halamka developed an aggressive schedule for patching servers and desktop machines at BIDMC.

On the server side, the IT staff held what Halamka called an “all nightmare-athon” patching session in late July, applying the relevant Microsoft patches to the hospital’s 130 Windows servers.

For desktop machines, Halamka’s staff used Microsoft’s Systems Management Server product to distribute software patches to 4,500 desktop machines running various flavors of Windows.

Once the worm emerged on Monday, BIDMC relied on Network Associates’ NAI Total Virus Defense antivirus product to distribute updated virus definitions to users’ desktops, Halamka said.

Speaking on Wednesday, Halamka said the combination of approaches, using firewall, intrusion detection systems (IDS), antivirus and patches, worked.

The air-tight approach even allowed BIDMC’s network to withstand exposure to Blaster from two laptop computers that were infected through employees’ home Internet connections and then connected to the hospital network.

“Those systems were easily identified and rectified and there was absolutely no spread of the worm within our network,” Karas said.

Other hospitals in the vicinity of BIDMC did not fare as well, he said.

BIDMC’s firewall blocked Blaster attack traffic that emanated from those institutions and was sent to BIDMC over LMANet, a high-speed data network that connects the various research hospitals in Boston’s Longwood Medical Area, Karas said.

BIDMC’s experience illustrates that firewalls, antivirus and IDS software are only part of the solution, according to Deb Peinert, vice president of education for the Information Systems Security Association.

Organizations need to have policies in place in advance of new vulnerabilities and virus breakouts to plug holes throughout their networks, Peinert said.

CERTs and solid user education about virus exposure, in addition to patch management software and a solid understanding of antivirus and firewall technology, can help organizations like BIDMC prepare for the worst, she said.