Earthquakes, fire and lightning: Must be a NEBS test

The blue and gray chamber looks harmless enough. Narrow windows embedded in 12-foot-wide, 18-foot-high walls tempt visitors to peer in at bare cement blocks, a bright yellow video camera and a shiny metal apparatus that looks like a flute. Then you step in and take a whiff.

“Pretty strong, huh?” says Clayton Forbes, senior project engineer at National Technical Systems (NTS) in Boxborough, Mass. “That’s from the units that didn’t make it. The flames and smoke got ’em.”

The NTS fire test chamber, where a charred metallic and gassy odor lingers, is one in a battery of stations used to determine whether telecom gear is safe and sturdy enough to be used in the central offices of RBOCs and other big U.S. carriers, and, in some cases, military networks. The flute-like instrument in the chamber is actually a methane gas torch that gets stuffed into the guts of telecom boxes to find out if they can contain a blaze. The yellow camera takes thermal images, especially useful when cabinet doors and black smoke make viewing the burning innards otherwise impossible.

Welcome to the largely behind-the-scenes world of NEBS testing.

While the acronym NEBS, short for Network Equipment Building System, is familiar to many in the industry, far fewer know what it takes for a switch, server or other box to gain NEBS compliance.

Carriers require equipment makers that hope to win their business to first run their devices through a gauntlet of about 30 NEBS tests – not government mandates – defined in documents GR-63 for environmental requirements and GR-1089 for electrical requirements.

The NEBS concept got its start in the 1970s at Bell Labs, and the documents now are maintained by Telcordia, which also conducts NEBS testing. While the NEBS criteria is designed for carrier equipment makers, testers say some enterprise network equipment gets NEBS certified in cases where vendors sell gear to carrier and corporate markets.

At NTS, telecom gear needs to survive the flame test and an earthquake simulator that tops 7 on the Richter scale, plus make it through chambers that determine whether equipment can handle extreme hot and cold temperatures as well as lightning strikes and high altitudes. Sometimes testers, armed with shotguns, even take boxes out back and try to shoot through their cabinets with No. 6 steel shot from 50 feet away.

Vendors pay tens of thousands to hundreds of thousands of dollars per year to have their products tested.

Jim Press, a colleague of Forbes’ who runs the electromagnetic compatibility lab at NTS, says he remembers hotshot entrepreneurs a couple of years back coming in to test all sorts of new devices that could push information down pipes faster.

“They’d tell me, ‘We’re going to ship this month,'” he says. “I’d tell them, ‘You’ve got months of testing and design work ahead of you,’ and they’d be shocked.”

The overall NEBS testing process usually takes at least a couple of months and is extremely challenging, as manufacturers seek to strike a delicate balance between sealing in emissions, blocking out contaminants and allowing enough airflow to keep systems cool in rooms that typically don’t have sprinkler systems.

“Fire testing is pretty nerve-wracking,” acknowledges Paul Elias, CTO with Covaro Networks, a start-up that recently got its CC-16000 Ethernet services box NEBS 3 certified, the most stringent level defined.

While the flame and earthquake tests are crowd pleasers, the more subtle electromagnetic interference and discharge tests are among the toughest to pass, says Elias, who has spent his career at small and large telecom companies. “In 20 years of doing this I’ve never seen a first-pass success,” he says.

Regardless of how well an engineering team has built its product, Elias says NEBS testing remains a high-anxiety event given that any setback could cost a start-up two or three months in its design cycle and chew up 10% of a round of venture funding.

Outfits such as NTS aren’t out to ruin anyone’s day, even though they run torture chambers for equipment. In fact, testing companies work with clients to suggest how products can be redesigned or tweaked to pass the tests. One box that underwent testing at NTS several weeks ago was patched with aluminum foil and masking tape to show the equipment maker where the box needed to be tightened up.

Labs use videotapes of boxes going up in smoke or dancing wildly on the earthquake simulator for educational purposes and to wow lab visitors.

“What we say here is: ‘Where else can you purposely break hundreds [or thousands] of dollars worth of equipment and get paid for it?'” Forbes says.

“What usually happens is they do things themselves the first time around, then come back and get us involved from the start the next time,” says Press, who has spent six years at NTS and is a 19-year veteran of the electromagnetic compatibility field. “If you want to get into the NEBS world, there’s a lot of homework to do.”

While NEBS testing might seem like overkill, those involved in it insist otherwise. Verizon spends more than $1 million per year to evaluate between 2,000 and 3,000 products for use in its 9,800 central offices and other facilities, says Chuck Graff, director of NEBS compliance and quality assurance at the carrier.

The company reviews NEBS reports provided by vendors and has discovered 320 errors in test reports over the past three years, which if unfound could have resulted in network outages, he says. Verizon requires that its vendors meet a level of five-nines reliability in order for the carrier to adhere to state public utility commission service levels.

Many consider Verizon to be the toughest carrier to satisfy – service providers that request NEBS certification often have special requirements beyond the base standards, though they are trying to lessen such differences to make life easier for equipment makers. And the events of Sept. 11, 2001, only strengthened the carrier’s adherence to the standards. As was well documented in the wake of the terrorist attacks, Verizon’s 140 West St. switching office next to the World Trade Center remained in operation – thanks in part to diesel generators that took over when the DC power was lost – despite the network equipment there being covered with bricks, mortar and in some cases more than an inch of dust.

Graff says that every time it looks as though equipment vendors have gotten NEBS requirements down, change happens. For instance, with Verizon committing to a huge fiber-to-the-premises rollout, the carrier is increasingly dealing with video equipment makers, most of which aren’t up to speed on NEBS.

While NEBS testing will continue to challenge vendors, they might take comfort in the fact that Telcordia is mainly just fine-tuning the NEBS requirements, not planning any major new ones, says Rich Kluge, the company’s director of NEBS technical services.

“There’s no volcano test on the way,” he says.