Why Argonne has pulled the plug on VoIP -- for now

After handset problems in its voice-over-IP pilot, Argonne National Lab turned to TDM, at least temporarily

Scott Pinkerton has more than 25 years of IT experience, including more than a dozen with Argonne National Laboratory, where he currently serves as network services manager. He manages Argonne’s core network infrastructure, the one with capacity of some 80Gbps.

Scott Pinkerton has more than 25 years of IT experience, including more than a dozen with Argonne National Laboratory, where he currently serves as network services manager. He manages Argonne’s core network infrastructure, the one with capacity of some 80Gbps.

Given his network credentials, it should raise a big yellow caution flag for would-be VoIP implementers everywhere to learn that Pinkerton’s first VoIP foray went exactly nowhere. After a trouble-plagued pilot, he was forced to rip out his VoIP phones and replace them with TDM models. What’s more, his was a greenfield VoIP implementation, not a TDM replacement, and the network in question was running only voice, not converged voice and data.

This is the rather sad tale that Pinkerton told to attendees at the recent Network World IT Roadmap Conference & Expo in Chicago. “I often feel there’s a little more learning to be had from the rollouts that didn’t just go like clockwork,” Pinkerton said in a follow-up interview. “It really forced us to dig into things quite a bit deeper than we would’ve if it were just plug and play.”

Argonne exterior

The setting

Located in Argonne, Ill., just south of Chicago, Argonne recently celebrated its 60th anniversary as a U.S. National Lab. It has a $475 million budget and about 2,900 employees, with more than 5,000 others who use its facilities throughout the year, conducting research on a range of technologies for the Department of Energy.

Argonne first looked at VoIP about two years ago, when it was replacing the main campus PBX. “Our confidence in VoIP wasn’t there,” at that time, Pinkerton said. But when a new building was going up to house Argonne’s Center for Nanoscale Materials, Pinkerton decided the time was right to try again, on a smaller scale.

The pilot project, which began around September 2006, involved 80 IP phones connected to three Cisco Catalyst 4500 switches and two 3560 switches. The Cisco switches, in turn, connected to a 10G Ethernet network core based on Systimax Solutions equipment.

Argonne installed the VoIP network on separate circuits from the building’s data network, which he said supported “a lot of oddball science,” including teraflop computing. “Our data networking is a little bit nontypical,” Pinkerton said. He expected that keeping the voice network separate would also simplify troubleshooting.

VoIP trouble spots

Argonne deployed three models of handsets and found they had different firmware versions because power-over-Ethernet standards had changed over time. Some of the handsets would work with the Cisco 3560 switches but not the 4500s. (Pinkerton did not want to disclose the name of the handset vendor because he is still in negotiations with the company over the matter, although he did say the handset were not Cisco’s.)

Many of the users whose handsets did work were complaining of jittery calls, button lag, random key tones during conversations and other problems. Pinkerton and his team checked network syslogs for error messages and double-checked QoS settings, but all signs pointed to the Cisco switches operating at 100Mbps full-duplex, as they should.

The problem turned out to be twofold. First, the gateway connection from Argonne’s traditional PBX to the VoIP network was configured to autonegotiate speed and duplex settings. That process wasn’t consistently working, and Pinkerton’s team ultimately had to force the connections to operate at 100Mbps full-duplex.

He faced the same problem with the handsets, which would go through the autonegotiate process but set themselves to 10Mbps instead of 100M. “It didn’t work until we forced to static 100M full duplex,” Pinkerton said. Only after many weeks did the vendor admit that the handsets were failing to properly autonegotiate, he noted. “We had a lot of vendor engagement that didn’t always immediately result in vendor solutions.”

The “death blow,” however, really had nothing to do with technology. “Users were unhappy with the ergonomics of the phone, how it felt on the shoulder,” Pinkerton said. Additionally, many complained about the quality of the speakerphone, which produced too much feedback. “We forever got call-quality complaints on the speakerphone.”

In the end, Argonne removed all the IP phones and replaced them with TDM models, which caused some amount of pain for Pinkerton and his team. “We weren’t quite prepared for a wholesale ‘cancel the whole darn thing,’” he told IT Roadmap attendees, noting he needed to pull more copper pairs to support the TDM phones.

Hard-learned advice

It’s no surprise, then, that his first words of advice for others embarking on VoIP projects is to have a good back-out plan should things go wrong.

Additionally, he said a good network baseline is critical. “Know intimately the round-trip time your packets will see,” Pinkerton said, noting this will help you determine if packets are out of order.

And, of course, don’t forget the ergonomics.

Even after all this, Pinkerton said Argonne hasn’t given up on VoIP and is planning another pilot for this summer, this time with a new generation of phones that supports Session Initiation Protocol, which is intended to make it easier for devices from multiple vendors to work together.

He also has more lofty expectations going forward, including lots of location- and presence-based features and integration of wireless services with the VOIP network.

At the same time, Pinkerton said he’s struggling to make the economic case for VoIP. “I love the reliability of my TDM network,” he told the attendees. He noted that Argonne has just nine PBX node rooms to cover its 1,500-acre campus because copper runs can extend for 4,000 feet.

Implementing VoIP will mean adhering to runs of no more than about 328 feet, which he said would mean installing lots more data closets and a huge investment. “That’s my biggest challenge,” he said.

Desmond is events editor for Network World and president of PDEdit, an IT publishing company in Southborough, Mass. Reach him at paul@pdedit.com.

Learn more about this topic

Video: Not all VoIP rollouts go as planned

Despite new building and fresh networking gear, a voice over IP implementation does not always go as planned - even for a high-tech outfit like the Argonne National Laboratory. Scott Pinkerton, manager of Argonne's core network infrastructure, explains the hurdles his group faced and the lessons learned from their failed VoIP project.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Now read: Getting grounded in IoT