Technical change proved relatively easy; educating users was the hard part.
Five years ago, Argonne National Laboratory was in the midst of a cybersecurity crisis. The Energy Department research center was failing audits, and management was under pressure to fix the situation. Today, Argonne gets top marks from security auditors. Scott Pinkerton, communications infrastructure department manager, tells Network World Senior Editor Carolyn Duffy Marsan how the organization turned things around.
What’s your role at the lab?
The voice and data infrastructure is my primary responsibility. I own much of the network-based security including firewalls, intrusion-detection systems and virtual private networks. I work closely with the cybersecurity program manager, who is in the same division.
What percentage of your time is spent on security issues?
We just had a large Department of Energy audit this summer, so during that time the percentage was fairly high, about 40% to 50% of my time. In a normal year, it’s probably down in the 15% to 20% range.
How many people do you have on your staff involved with cybersecurity?
I have a staff of 23 people, of which three are involved in cybersecurity. They handle firewalls, VPNs, intrusion detection and [Cisco] NetFlow data analysis.
Can you give a brief description of your network?
We support around 3,000 people at the lab and 12,000 computers. We have several high-performance clusters of machines. It’s an IP network. We have a lot of wide-area networking connectivity. Right this second, we probably are using six OC-192s. We have a 10G campus infrastructure. We have traditional TDM-based voice. We have an active VoIP pilot program in one of our newest buildings, the Center for Nanoscale Materials.
| ||||||||||||||||||||||||
What does your security architecture look like?
We run a distributed model with multiple firewalls cooperating to provide a perimeter protection scheme. We recently added Tier 2 or divisional firewalls at the project or building level. All the places at the lab that have sensitive technologies or a preponderance of personally identifiable information get a second level of firewall protection. We use primarily Cisco firewalls, intrusion-detection systems and VPNs.
Are you spending more time on security issues than two years ago?
No. Our cybersecurity crisis occurred in 2001. That was the low water mark of our cybersecurity profile. We had suffered through quite a number of audits that were very negative. The federal government’s inspector general gave us a bad audit. We had poor audit results from the Energy Department’s Office of Assessment. We were under a lot of pressure. We were having weekly meetings with the Associate Lab Director in 2001.
What steps did you take to resolve your cybersecurity crisis?
Before 2001, we didn’t have many technical solutions. Firewalls, intrusion-detection systems, VPNs – a lot of that technology didn’t exist. However, what I would say was the most telling changes all revolved around culture. We started a cybersecurity architectural review committee to figure out a paradigm for separating our network services. We had a slew of town hall meetings throughout that 12-month period. We did an enormous amount of education about what these changes would mean to [the scientists.]
Dropping in the firewalls wasn’t that painful. Intrusion-detection systems weren’t that painful. Tuning the intrusion-detection system wasn’t that painful. It was educating the user community that was hard. We did a complete revamp of our IT training program. We have a yearly IT refresher course that everyone has to go through that was completely redesigned. We did a tremendous amount of outreach to our system administrators and now we keep track of them.
When were you out of crisis mode?
By 2002, we had started to turn the ship around and we were starting to get passing grades on audits. At lease from the point of view of the senior management of the laboratory, we were out of trouble.
What have you done differently since the crisis was over?
Since 2002, every day we’re still asking ourselves what we can do more, better, smarter. We’ve done a number of things that are very interesting. We re-adapt our intrusion-detection systems every 60 minutes based on the current state of the firewall. We’ve done a lot to integrate [our devices.] We’ve been very creative on how we administer network control. Since we’re similar to a college environment, it would be too challenging to force an agent on every device. So we scan every 10 seconds for new devices on the network. We keep track of everyone’s presence on the network.
What new security initiatives do you have planned?
We’re pushing a federated approach to sharing data to help improve cybersecurity. We’re asking if it would be valuable for Oak Ridge National Lab or Lawrence Berkeley National Lab if Argonne gave them a summary digest every 30 minutes of the IP addresses that have been hostile here. We’re seeing that the people that are hostile at one [Energy Department] lab end up being hostile at others.
Do you feel more confident about security than you felt several years ago?
Yes, because we’ve been doing fairly well on the audits. However, I’d say that the battle never rests. The threat model is forever evolving. I’m still nervous.
What issues still keep you up at night?
Personally identifiable information is a new problem. For [the Energy Department], it’s on the political radar screen. Any incidents involving PII are highly scrutinized. We have a ton of new oversight about what we are doing on the PII front.
What lessons have you learned from your experience improving security at Argonne?
Communication and education have to be number one. You have to listen
| Title: | Communications infrastructure department manager (since August). |
| Organization: | Argonne National Laboratory |
| Responsibilities: | Voice and data networks, network-attached security devices. |
| Annual budget: | $5.5 million |
| Staff: | 23 |
| Previous jobs: | Various positions at Argonne, including networking section manager and computer systems engineer. Also served as staff engineer at Martin Marietta Astronautics. |
| Education: | Master’s degree in computer science from the University of Colorado in Boulder, bachelor’s degrees in computer science and math from Bowliing Green State University. |
| First PC: | Radio Shack TRS/80 with attached cassette deck storage. |
| First experience on the Internet: | News groups (netnews). |
| Home network: | Firewall with four-port switch, wireless access point, two adult computers, two kid computers, two network-attached printers and Tivo with a wireless dongle. |
What does your security architecture look like?
We run a distributed model with multiple firewalls cooperating to provide a perimeter protection scheme. We recently added Tier 2 or divisional firewalls at the project or building level. All the places at the lab that have sensitive technologies or a preponderance of personally identifiable information get a second level of firewall protection. We use primarily Cisco firewalls, intrusion-detection systems and VPNs.
Are you spending more time on security issues than two years ago?
No. Our cybersecurity crisis occurred in 2001. That was the low water mark of our cybersecurity profile. We had suffered through quite a number of audits that were very negative. The federal government’s inspector general gave us a bad audit. We had poor audit results from the Energy Department’s Office of Assessment. We were under a lot of pressure. We were having weekly meetings with the Associate Lab Director in 2001.
What steps did you take to resolve your cybersecurity crisis?
Before 2001, we didn’t have many technical solutions. Firewalls, intrusion-detection systems, VPNs – a lot of that technology didn’t exist. However, what I would say was the most telling changes all revolved around culture. We started a cybersecurity architectural review committee to figure out a paradigm for separating our network services. We had a slew of town hall meetings throughout that 12-month period. We did an enormous amount of education about what these changes would mean to [the scientists.]
Dropping in the firewalls wasn’t that painful. Intrusion-detection systems weren’t that painful. Tuning the intrusion-detection system wasn’t that painful. It was educating the user community that was hard. We did a complete revamp of our IT training program. We have a yearly IT refresher course that everyone has to go through that was completely redesigned. We did a tremendous amount of outreach to our system administrators and now we keep track of them.
When were you out of crisis mode?
By 2002, we had started to turn the ship around and we were starting to get passing grades on audits. At lease from the point of view of the senior management of the laboratory, we were out of trouble.
What have you done differently since the crisis was over?
Since 2002, every day we’re still asking ourselves what we can do more, better, smarter. We’ve done a number of things that are very interesting. We re-adapt our intrusion-detection systems every 60 minutes based on the current state of the firewall. We’ve done a lot to integrate [our devices.] We’ve been very creative on how we administer network control. Since we’re similar to a college environment, it would be too challenging to force an agent on every device. So we scan every 10 seconds for new devices on the network. We keep track of everyone’s presence on the network.
What new security initiatives do you have planned?
We’re pushing a federated approach to sharing data to help improve cybersecurity. We’re asking if it would be valuable for Oak Ridge National Lab or Lawrence Berkeley National Lab if Argonne gave them a summary digest every 30 minutes of the IP addresses that have been hostile here. We’re seeing that the people that are hostile at one [Energy Department] lab end up being hostile at others.
Do you feel more confident about security than you felt several years ago?
Yes, because we’ve been doing fairly well on the audits. However, I’d say that the battle never rests. The threat model is forever evolving. I’m still nervous.
What issues still keep you up at night?
Personally identifiable information is a new problem. For [the Energy Department], it’s on the political radar screen. Any incidents involving PII are highly scrutinized. We have a ton of new oversight about what we are doing on the PII front.
What lessons have you learned from your experience improving security at Argonne?
Communication and education have to be number one. You have to listen to a lot of people. You have to let them communicate their concerns and worries about these types of changes. Developing willful partners with your user population is important. Talk to your peers to see what other people have done. Don’t try to re-invent the wheel yourself. Also, know thy network. We have NetFlow data so we can understand what traffic is on our network.
