Cisco IOS scare stirs questions

A serious vulnerability in Cisco’s IOS software has yet to yield publicly disclosed exploits, but the incident does punctuate the difficulty of properly maintaining IOS-based gear — especially for those running older versions, experts say.

A serious vulnerability in Cisco’s IOS software has yet to yield publicly disclosed exploits, but the incident does punctuate the difficulty of properly maintaining IOS-based gear – especially for those running older versions, experts say.

It also has amplified concerns that Cisco might be trying to make IOS do too much.

Cisco notified customers July 16 that had it discovered a flaw affecting all devices running IOS software, whereby an attacker could disable a router by sending specially crafted IPv4 packets to the device. The next day Cisco posted a patch – which requires a router to come offline temporarily while rebooting – and a separate work-around that can be deployed without restarting equipment.

On July 18, it was reported that ready-made code written to exploit the IOS flaw was circulating the Internet and could be used by fairly unsophisticated users to crash routers.

However, no ISPs or companies have reported being hit as a result of the Cisco vulnerability.

“This bug definitely has the potential to be a significant threat to businesses,” says Pat Donahue, a network administrator with ACMI, a maker of medical equipment that has offices across the country.

Donahue says the ubiquity of IOS and the ease with which the bug could be exploited prompted him to act quickly. He downloaded the patch from Cisco and had all his equipment upgraded within days.

“I felt much safer knowing that the software running the router itself is not vulnerable, rather than relying on access lists that may or may not work or could be modified or removed at some later date,” he says.

Other companies reported taking immediate action.

“For SBC, it was a total of about 600 pieces of equipment that were affected,” said an SBC spokesman. “We immediately started rolling out that patch.”

The upgrades were done on a rolling basis so no service was interrupted, and all equipment was patched as of July 18.

Others were taking a more measured approach.

“We’re looking at it,” says Dick Emford, lead network analyst for plastics manufacturer Newell-Rubbermaid, about the IOS vulnerability. “But we haven’t yet assessed how bad it might be for us.”

Rubbermaid’s network links many manufacturing sites in the U.S. over a WAN, but few Cisco nodes face public or unprotected Internet connections so it is not an urgent concern, Emford says.

“We’re monitoring this vulnerability . . . but we’re not too concerned about it,” says Phil Go, CIO at Barton Malow. The Chicago construction company uses Cisco routers to link three offices across the country with IP voice and data. Go is confident because his firm uses access control lists to keep traffic on those routers limited to internal corporate voice and data packets, and few ports face the Internet.

Hesitancy in upgrading routers is common, says one veteran industry watcher.

“Lots of companies, when they buy routers, don’t want to upgrade them [later],” says Frank Dzubeck, president of consultancy Communications Network Architects. “They think they’re buying an appliance, not a software product. . . . And if they get three or four [IOS revisions] behind, there’s much trepidation when the time comes to apply a major patch like this.”

Smaller shops, in particular, might be slow to apply patches, or are taking extra time to determine if gear is at risk, experts say. This is because upgrading IOS on older routers that have been chugging along for long periods of time without maintenance can be a tricky proposition.

“It [is] more difficult for organizations that don’t have an engineer on staff” to upgrade an affected IOS device, says Toby Velte, CTO of software company Solv Technology, and author of the book Cisco: A Beginners Guide.

The upgrade process involves downloading the patch, applying it to an IOS version, sending the image to a router, then restarting the device “which kills all open connections,” he says. This procedure could be trouble for organizations that cannot avoid router downtime, or those that have lots of routers but few IT staff members proficient in IOS.

“If you get the wrong IOS on the device or screw something else up along the way, the device will be down until you get it right,” Velte adds.

Becoming proficient in IOS can take hours of reading, coursework and certification, and years of experience. IOS is an intricate operating system that can support many things besides routing packets – such as security, VoIP, Wi-Fi and quality of service.

Dzubeck says that a decade of evolution and consistent re-crafting of IOS might lead to the code becoming prone to failures.

“IOS is really an amalgamation of a number of things,” Dzubeck says, adding that as the code gets on in years more vulnerabilities could pop up.

But Velte disagrees that “feature bloat” is to blame.

“While the code is getting larger by including more features, the hardware capabilities have more than kept up,” Velte says. “When some software trees get older, such as Unix, they get more hardened and reliable. A recent copy of IOS is greatly improved over a five-year-old version. The presently discovered vulnerability could have been in IOS for years. So it wasn’t IOS bloat that caused the problem to appear. It was the level of sophistication of those that probe for such vulnerabilities.”

Jim Duffy, managing editor of Network World’s The Edge, contributed to this story.