IT managers and analysts are expressing surprise at the amount of time it appears to be taking officials at the City of San Francisco to regain full control of the city’s FiberWAN network after a disgruntled network administrator allegedly locked access to it by resetting administrative passwords to its switches and routers.
With few details publicly released on what exactly happened, many are suggesting that the network lockout and the city’s response to it point to a failure to implement and manage fundamental security controls.
Terry Childs, a network administrator at San Francisco’s Department of Telecommunications and Information Services (DTIS) was arrested on July 13 for allegedly tampering with the network, which carries almost 60% of the city government’s traffic. He is also alleged to have planted network devices that enabled illegal remote access to the FiberWAN network.
He was jailed last week on $5 million bond after refusing to divulge the passwords he had used to block access to the network. Child, 43, pleaded not guilty to the charges against him at a hearing in San Francisco Superior Court last week. He is scheduled for a bail hearing tomorrow. If convicted on all charges, Childs faces a maximum of seven years in prison.
As of late Monday, the city’s efforts to repair the network still remains a “work in progress” according to Ron Vinson, deputy director of the San Francisco Department of Telecommunications and Information Systems (DTIS). Despite around-the-clock efforts with vendors and staff the city is still only in the process of determining “what level of illegal access or what level of tampering” Childs was responsible for, Vinson said.
According to Vinson, the WAN is operating normally, but the city has not yet regained full administrative of all routers. What remains unclear also is whether IT officials have discovered all of the network devices that Childs is alleged to have illegally installed on the WAN, Vinson said. He did not say what exactly the network devices were or what they did, but claimed that Childs had gone to great lengths to hide them from detection. “We have 60 plus departments that are clients of ours,” Vinson said and the task now is to find out if Childs managed to install the devices at any of those departments, he said.
“That is why we need to do a system-wide analysis of where we are with access. We don’t know what he had access to,” Vinson said. He added that the focus right now is on damage containment. “We want to make sure we maintain full operability if [Childs] was to be released on Wednesday,” and were to try to gain illegal access to the network, he said. “We want to make sure we are up and running.”
Vinson however did not provide any details on what exactly Childs did or the extent to which he may have compromised the network. He said that the reasons for the slow recovery would become apparent once those details were publicly released.
Meanwhile, news of the city’s continuing struggles, combined with a relative lack of publicly available details on what exactly happened, is fueling questions and theories about what may have happened in some quarters.
“I am completely floored that it is taking so long to restore access to the equipment,” said Jim Kirby, senior network engineer with Dataware Services, a Sioux Falls, SD-based service provider. “Unless they have some crazy uptime requirement that prevents them from rebooting gear, it’s hard to understand.” In most cases, he said, passwords can be reset with a reboot and some keyboard combinations.
But, he added, with some networking gear from Cisco Systems Inc. it is possible to issue a command that renders password recovery impossible, making it necessary to send the equipment back to Cisco to be “re-flashed to new.” “Even so, it’s merely a matter of replacing locked gear with new gear created from the backup configurations they surely must have somewhere. One box at a time, if needed,” Kirby said. The apparent fact that San Francisco city officials took longer than 48-hours to restore access says that they were not following “even basic network administration standards,” Kirby said.
Similar sentiments were expressed by Johannes Ullrich, chief technology officer at the SANS Internet Storm Center in Bethesda, Md. “It is odd that it took so long to recover. The only reason I can think of is that they did not have good configuration backups,” Ullrich said. Though it is unclear what networking equipment was involved, typically it is possible to reset passwords with physical access to the equipment though all configuration changes are lost, he said. Good configuration management practices also have allowed for unauthorized changes such as those made by Childs to be detected quickly thereby limiting damage. “In addition, it can help with recovery as it is easier to revert to an old known-to-be-good configuration,” he said.
Lou Michael, director of network and infrastructure services in Virginia’s Arlington County department of technology services said that based on media reports alone, he did not want to come down against the IT officials in San Francisco. “But it is a long-standing best practice to make sure that access passwords are never in one place or with one individual,” he said. “I have worked on systems where root level, hidden accounts were maintained so that if a rogue individual changed all the passwords or attempted to lock the accounts they would not know of these accounts existence,” he said.
He theorized that even if Childs had used just one password to lock others out of the network, San Francisco IT officials would probably need to initialize and rebuild each locked device to gain access to them. “Depending on how many people you have working on it and when you are allowed to bring the device down, it could take awhile,” he said.
The entire incident points to a management failure at DTIS, said Arshad Noor, CEO of StrongAuth, Inc. a vendor of compliance and ID management products based in Cupertino, Calif. For starters, a single employee should never have had exclusive control over the WAN, in the manner Childs appears to have had, Noor said. There appears not to have been much of a configuration-management or a change-management process or a disaster recovery plan, judging from the recovery time, he theorized.
“All in all, the IT management of the city are responsible for this mess, because it was within their authority — and in their mandate — to avoid this situation, but they did not,” Noor said. “While Terry Childs might pay for this situation through jail-time or fines, management cannot be absolved of their responsibility for the situation,” he said.
