Verizon outages affected several states for two days

Verizon said it resolved a two-day problem in its Signaling System 7 network that affected calls coming to and from competitive local exchange carrier networks this week.

Outages dropped calls between Verizon and CLEC networks in California, Florida, Idaho, Illinois, Indiana, Michigan, Ohio, Oregon, Washington and Wisconsin. Other states with some impact included Arizona, Nevada, and North and South Carolina.

The regions represent the former GTE footprint of Verizon’s network.

One user in Washington said he continually received “all circuits busy” messages from Monday into Tuesday.

“There was a problem that emerged on Monday morning and carried on through Tuesday,” a Verizon spokesman from the Pacific Northwest said. “It was a problem in Fort Wayne, Ind., that impacted the SS7 handoff between TNS and Verizon.”

TNS is Transaction Network Services, a Reston, Va., company that provides SS7 services on behalf of carriers. TNS has 100 carrier customers for SS7 services. It did not reply to an e-mail query by press time.

The CLECs most affected were Level 3, Time Warner Telecom and XO Communications. Verizon said the problems were resolved on Tuesday night at 11pm EST. XO confirmed that.

“If a customer is still experiencing an issue they may need to reboot” their telephony system, an XO spokesman said.

Verizon said the Fort Wayne facility could not handle the call capacity.

“We believe that the Indiana quad was under-engineered from a capacity perspective, and that the four links ran at a high rate,” a Verizon spokesman from Texas said. “A maintenance event [one link going down] triggered a large volume of resets that forced the processors to overload and go offline.”

Two switches need to be augmented with additional link capacity to TNS, which TNS will have to order and install, the spokesman said. In addition, Verizon has requested a patch from Alcatel to prevent processors from going offline.

Verizon and Alcatel engineers completed cabling preparation work Tuesday night in Indiana for the implementation of next-generation boards, the spokesman said.

“These next-generation boards have a more efficient buffer technology that will allow cushion until additional link capacity can be obtained by TNS,” he said. “The current plan is to install these new boards in the Indiana [signal transfer points] tonight during safe time.”

The spokesman later retracted his statement about the Indiana facility being "under-engineered" and said Verizon is continuing to work with the equipment manufacturer and TNS to "understand the root cause" of the outage.

The customer in Washington is now evaluating how to reroute around such a failure should it happen again.

“I shudder to think about losing over a day of business,” he said. “A small business is not always going to be thinking of alternate ways of reaching customers. If you don’t do anything else, at least have an 800 number or a number that resides on another network.”