After a security breach

Every network is vulnerable to some type of attack. The important thing is how you respond.

CIOs and network administrators offer tips about how to respond following a network security breach.

Plan for a smooth recovery

An investigation found that the rootkit, an attacker tool for compromising computer systems without detection, hadn't been touched since it was installed in October 2003. That made it highly unlikely personal records were ever copied off the server. Still, Kerntke persuaded senior administration to err on the side of caution and go public with the breach.

The frenzy had finally died down that Friday, but at 7 p.m. as he neared his driveway in his Chevy Tahoe, he got another call from his public relations manager. A channel 3 news crew was waiting for him back at the data center and needed him to show them around.

"I never thought when I took this job that I'd be on TV," says Kerntke, who not only kept his job despite the breach, but also earned accolades from school administrators for his ability to communicate the extent of the damage to a non-technical news audience and to be available for interviews at odd hours.

Facing the limelight is part of the way IT executives' jobs are getting more challenging as a result of new rules to report private data breaches. There's also the other work involved - the investigations, repairs and notifications arising from data breaches that expose personal information. In all, 80 such breaches went public between Feb. 15 and Sept. 29, according to the Privacy Rights Clearinghouse.

While IT executives don't seem to be losing their jobs over the rising number of publicly reported breaches, their companies are experiencing severe losses, starting with an exodus of customers and customer loyalty. According to a September survey of 10,000 adults conducted by the Ponemon Institute, a privacy research organization, 19% of respondents ended their relationships with companies reporting breaches, and 58% say they have lost trust.

Publicly held companies also suffer a 5% stock drop in the wake of such a disclosure, according to the 2003 study "The Economic Cost of Publicly Announced Information Security Breaches" published in the Journal of Computer Security. And the cost of informing affected parties also is expensive, ranging anywhere from $15 to $35 per victim, according to Jonathan Penn, principal analyst for identity and security at Forrester Research.

But organizations can reduce their overall losses by reporting breaches in a timely manner and offering whatever help they can to the affected parties, Penn says. On the other hand, organizations can compound their losses by covering up and delaying reporting, such as the case with ChoicePoint, whose stock dropped by 15% after fraud in its system exposed 145,000 credit identities in February. And health maintenance organization Kaiser Permanente was fined $200,000 in August for a three-month delay in reporting an exposure of patient data posted on a publicly accessible Web site used for help desk support.

Mary Henderson
POOR RESPONSE Kaiser Permanente • The Northern California HMO was fined $200,000 in August for failure to properly investigate and report a mistake that exposed a handful of patient lab records. Mary Henderson, vice president of IT compliance, was not able to determine who was responsible.
Shelton Waggener

GOOD RESPONSE UC-Berkeley • When a laptop was stolen from the graduate department that contained personal information about 98,400 alumni, Shelton Waggener, director of central computing, launched an informational Web site, now emulated by other universities responding

to data breaches.
John Shaughnessy
POOR RESPONSE CardSystems • Visa’s vice president of risk management, John Shaughnessy, opted to pull the plug on payment card processor CardSystems after it violated card industry payment standards set ahead of time to prevent breaches such as the one that exposed 40 million cards in June.
Michael Kerntke
GOOD RESPONSE University of Connecticut • Following the June 20 discovery of a rootkit on a system housing personal information on 72,000 employees, students and alumni, U-Conn CIO Michael Kerntke opened his data center to the media and explained to them, in plain English, the level of risk to the data on the compromised server.

Start with standards

The best response plan starts with documented compliance to security standards mandated by a particular industry. If a company hasn't met these standards and a breach occurs, the company faces regulatory action.

Failure to adhere to security best practices also could result in corporate liability in the advent of an exposure, as in the case of BJ's Wholesale Club, which faces $13 million in outstanding claims by credit card-issuing banks trying to retrieve the costs of fraudulent purchases tracked back to accounts copied out of BJ's systems. According to a Federal Trade Commission complaint, the retailer violated common security practices, including failing to encrypt data, holding data it shouldn't have and failing to take proper measures to prevent unauthorized access. In a written statement, BJ's responded that no conclusive evidence of a breach was found.

A similar violation of payment-card industry standards might force CardSystems Solutions out of business. In June, CardSystems reported that identity thieves had hacked into a database containing 40 million credit card numbers.

The company admitted the data had been improperly kept. As a result, CardSystems has lost two of its three biggest card associations - Visa and American Express - and is awaiting a verdict from MasterCard. According to Penn, Visa and American Express had legal reasons to pull the plug on CardSystems. If they hadn't, they also could be held liable, he says.

"We had no choice but to drop CardSystems as an approved processor," says John Shaughnessy, senior vice president of operations and risk management at Visa. "They were in clear violation of our payment-processing standards."

Not to mention the CardSystems blunder also cost the card associations and the issuing banks millions of dollars in reparation.

Determine the scope

For example, Visa investigators have spent numerous hours uncovering the scope of the damage for its issuing banks by monitoring for and tracking fraudulent transactions back to the CardSystems origination point, Shaughnessy says.

Shaughnessy also had the burden of supporting an outside FBI investigation into the criminal activity of the hacker, which is still ongoing. As such, he was required by the FBI to keep the breach under wraps so as not to scare off the attacker. But he lost the luxury of time when, on June 17, the story broke prematurely in the Wall Street Journal.

"It's important to have an emergency response plan in place ahead of time," Shaughnessy says. "You must be prepared to track down who's impacted, and already have in place who's responsible to do what, because when something like this happens, you don't want to think about what to do and who to contact under pressure."

In another case, involving a stolen laptop at the University of California at Berkeley in March that contained the unencrypted records on 98,400 alumni, the IT department coordinated the investigation with university police. But because the computer was being used to aggregate data from various sources around campus to analyze graduation rates, the problem was in reassembling the data in question, which took the graduate department several business days, according to Shelton Waggener, director of central computing at Berkeley.

"The policy challenge is substantial here, because this machine was in compliance with the latest patches and security updates. It wasn't hacked. It was ripped from its mooring," Waggener says. "We were also dealing with the functional owners of the laptop who were requesting the data, and trying to determine new policies around data storage and access so we could prevent this from happening again."

Waggener chose to investigate first, then inform his administration and begin the process of reporting. But if you're in a highly regulated industry, the impetus is to report first, investigate second.

"Laws are much more specific that you report immediately when it comes to exposure of patient health information," says Lynne Randolph, a spokeswoman for the California Department of Managed Healthcare (DMHC). The DMHC fined Kaiser Permanente in part because of the HMO's lag in reporting the security breach.

From the start, Kaiser thought it was doing the right thing by investigating the posting of patient information before reporting. And that investigation was plagued from the start, says Mary Henderson, vice president of IT compliance at Kaiser. The exposed data was pertinent only to the Northern California regional office, which didn't involve the central compliance office until after it conducted its own investigation. And the data in question - numbers identifying patients, and in four places real lab results embedded in sample troubleshooting forms - was nowhere to be found online.

"The No. 1 concern for us was to mitigate immediate damage to our customers," Henderson says. "So our first task was to sort through hundreds of pages of system documentation to see if there was any identifying patient data in the sample screen shots and reports embedded in training materials we use to help troubleshoot report help desk calls."

The only evidence to go on were copies of the non-compliant training pages posted on two mirror sites that linked to the blog of a former employee who had reported the violation to the Office of Civil Rights in January. So Kaiser also spent time contacting the hosting providers of the mirror sites to get the material taken down.

In the end, Henderson's team was never able to determine who was responsible for the posted data, so no jobs were lost over this. But the lack of evidence, including the timeline of the exposure, made Kaiser look bad enough for regulators to levy the fine.

Sound the alarm

Timely and rigorous notification is also critical in minimizing your losses, says Forrester's Penn, who urges compliance managers to go beyond legal requirements and include attempts to make the potential victim whole, such as offering credit monitoring when called for. According to the Ponemon survey, 52% of those receiving notifications of a privacy data breach thought the notice was difficult to understand, while 39% felt the message was not honest and believable.

Notification and reparation were the most difficult parts of the process, Berkeley's Waggener says.

For starters, he explains, the graduate student information dated to 1997, so it was hard to find most of the affected parties. Ultimately, his team was able to

e-mail about one-third of affected alumni. Then the university began paper mailings and set up a Web site and a 24/7 call center to reach the rest.

For the first two weeks, call center lines lit up 1,000 times a day, Waggener says. Concerned alumni asked what this actually meant to them, how the data was used and what they could do about it. Blogs and e-mail threads referred to the breach. Much of the posts were full of misinformation, making clarity among call center operators even more important.

"We had to write scripts on the fly and update them constantly due to the misinformation floating around out there. Not to mention, the criminal investigation into the stolen laptop was still in motion," Waggener explains.

To quickly respond to the call traffic, Waggener's response team selected an outside call center, which escalated to in-house responders when needed. This is another thing he wished he could have done better. Setting up the call center during the emergency and without a negotiated contract was chaotic and costly.

So another lesson learned was to have a negotiated contract in place with a call center for emergencies like this, he says.

Improve operations

UC-Berkeley's ID Alert Web site outreach also is another valuable tool Waggener plans to keep. Already, he says, a half a dozen other universities have requested permission to use Berkeley's ID Alert site for their own education.

Which is the final point IT managers make about recovering from a data breach: Learn from the experience and use it to make improvements. Use the incident to re-educate and enforce data safety practices in all personnel, advises Kaiser's Henderson, who used the experience to get funding for Web site security audits and to start an encryption program for laptops.

Quantifying the problem
Check the Privacy Rights Clearinghouse’s chronology to learn what businesses have reported data breaches, the type of breach and the number of individuals affected.

Since its breach, the University of Connecticut has been examining ways to reduce its reliance on Social Security numbers for student identifiers. It's also been auditing servers that contain and transmit sensitive information, and implementing more stringent network and server access controls.

"You'll never have a risk-free environment, because there will always the human element," Henderson says."

So when the unforeseen happens, act responsibly. Investigate. Inform affected parties and properly disclose to your regulators. Then use it as a learning opportunity to enforce better practices and security standards."

Radcliff is a freelance writer in Northern California. She can be reached at

Learn more about this topic

BJ's settles case with FTC over customer data security


Troubled CardSystems to be sold


Stolen laptop recovered


Kaiser Permanente patient data exposed online


Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: 10 new UI features coming to Windows 10