Why Argonne has pulled the plug on VoIP -- for now

Scott Pinkerton has more than 25 years of IT experience, including more than a dozen with Argonne National Laboratory, where he currently serves as network services manager. He manages Argonne’s core network infrastructure, the one with capacity of some 80Gbps.

Given his network credentials, it should raise a big yellow caution flag for would-be VoIP implementers everywhere to learn that Pinkerton’s first VoIP foray went exactly nowhere. After a trouble-plagued pilot, he was forced to rip out his VoIP phones and replace them with TDM models. What’s more, his was a greenfield VoIP implementation, not a TDM replacement, and the network in question was running only voice, not converged voice and data.

This is the rather sad tale that Pinkerton told to attendees at the recent Network World IT Roadmap Conference & Expo in Chicago. “I often feel there’s a little more learning to be had from the rollouts that didn’t just go like clockwork,” Pinkerton said in a follow-up interview. “It really forced us to dig into things quite a bit deeper than we would’ve if it were just plug and play.”

Click to see: Argonne exterior

The setting

Located in Argonne, Ill., just south of Chicago, Argonne recently celebrated its 60th anniversary as a U.S. National Lab. It has a $475 million budget and about 2,900 employees, with more than 5,000 others who use its facilities throughout the year, conducting research on a range of technologies for the Department of Energy.

Argonne first looked at VoIP about two years ago, when it was replacing the main campus PBX. “Our confidence in VoIP wasn’t there,” at that time, Pinkerton said. But when a new building was going up to house Argonne’s Center for Nanoscale Materials, Pinkerton decided the time was right to try again, on a smaller scale.

The pilot project, which began around September 2006, involved 80 IP phones connected to three Cisco Catalyst 4500 switches and two 3560 switches. The Cisco switches, in turn, connected to a 10G Ethernet network core based on Systimax Solutions equipment.

Argonne installed the VoIP network on separate circuits from the building’s data network, which he said supported “a lot of oddball science,” including teraflop computing. “Our data networking is a little bit nontypical,” Pinkerton said. He expected that keeping the voice network separate would also simplify troubleshooting.

VoIP trouble spots

Argonne deployed three models of handsets and found they had different firmware versions because power-over-Ethernet standards had changed over time. Some of the handsets would work with the Cisco 3560 switches but not the 4500s. (Pinkerton did not want to disclose the name of the handset vendor because he is still in negotiations with the company over the matter, although he did say the handset were not Cisco’s.)

Many of the users whose handsets did work were complaining of jittery calls, button lag, random key tones during conversations and other problems. Pinkerton and his team checked network syslogs for error messages and double-checked QoS settings, but all signs pointed to the Cisco switches operating at 100Mbps full-duplex, as they should.

The problem turned out to be twofold. First, the gateway connection from Argonne’s traditional PBX to the VoIP network was configured to autonegotiate speed and duplex settings. That process wasn’t consistently working, and Pinkerton’s team ultimately had to force the connections to operate at 100Mbps full-duplex.

He faced the same problem with the handsets, which would go through the autonegotiate process but set themselves to 10Mbps instead of 100M. “It didn’t work until we forced to static 100M full duplex,” Pinkerton said. Only after many weeks did the vendor admit that the handsets were failing to properly autonegotiate, he noted. “We had a lot of vendor engagement that didn’t always immediately result in vendor solutions.”

The “death blow,” however, really had nothing to do with technology. “Users were unhappy with the ergonomics of the phone, how it felt on the shoulder,” Pinkerton said. Additionally, many complained about the quality of the speakerphone, which produced too much feedback. “We forever got call-quality complaints on the speakerphone.”