Cisco routers caused major outage in Japan: report

Cisco routers were the source of a major outage May 15 in an NTT network in Japan, according to an investment firm bulletin.

Between 2,000 and 4,000 Cisco routers went down for about 7 hours in the NTT East network after a switchover to backup routes triggered the routers to rewrite routing tables, according to a bulletin from CIBC World Markets. The outage disconnected millions of broadband Internet users across most of eastern Japan.

Cisco says it could not say which specific router models were involved.

"Cisco is working closely with NTT East to identify the specific cause of the outage and help prevent future occurrences,” a Cisco spokesman said in an e-mailed reply. “At this time, Cisco and NTT have not determined the specific cause of the problem."

NTT East and NTT West, both group companies of Japanese telecom giant Nippon Telegraph and Telephone (NTT), are in the process of finalizing their decisions on a core router upgrade, according to the report.

The routing table rewrite overflowed the routing tables and caused the routers’ forwarding process to fail, the CIBC report states.

“Clearly, this failure doesn't reflect well on (Cisco) and at the very least highlights the need for two vendors,” states CIBC analyst Ittai Kidron in the report. Kidron states that NTT West is evaluating Juniper core routers while East evaluates the Cisco platforms.

“That said, we don't expect the failure at NTT East to influence its decision with respect to its choice of core router vendor,” Kidron states in the bulletin. “In fact, as router capacity was partly responsible for the failure, it is possible the outage could accelerate NTT's transition to Cisco's newer core router, the CRS-1.”

NTT was one of the initial testers of the CRS-1 when the product was launched three years ago.

“We don't believe the decisions would change based on this event,” Kidron concluded. “Juniper still remains a leading contender at NTT West and Cisco at NTT East.”