Management applications measure packet loss, jitter and latency to zero in on problems plaguing converged networks.
Do you trust a computer to tell you how your CEO's new IP phone sounds?
Network testing and monitoring vendors are betting you will as they peddle new call quality management applications that pinpoint problems on converged networks. Despite increasing reliance on e-mail, voice remains executives' method of choice for closing deals, and businesses embracing VoIP can't afford to make assumptions about call quality.
Voice quality standards
Vendors such as Apparent Networks, Brix Networks, Empirix, Integrated Research, Qovia and Spirent are rushing to fill the void, often licensing algorithms for active testing and passive monitoring from call quality pioneers Psytechnics and Telchemy. "This is the beginning of a big push, though the standards for VoIP call quality measurement are still evolving," says Eric Siegel, senior analyst at Burton Group.
Frost & Sullivan reckons the emergent VoIP monitoring/management market hit $50.7 million in 2004, and expects it to increase about sixfold by 2008. IP telephony is exploding, and upfront network assessments will only take a VoIP implementation so far. Unlike data, it has to work perfectly out of the gate.
"VoIP can be made to run as well and as reliably and as clearly as the best traditional phone network, but it's not a static environment," says Pierce Reid, vice president of marketing for Qovia, a 3-year-old start-up dedicated to VoIP call quality. "It has entropy. This can be accelerated by the employee who decides to download 'Shrek 2' at lunchtime."
Keeping the canaries singing
Network professionals with converged environments liken IP telephony to the cages of canaries that used to accompany the miners below ground. The birds keeled over when conditions in the mine became unsafe. Voice is revealing network problems that used to go unnoticed on IP networks, and the standard data fix - more bandwidth - doesn't work.
"These real-time applications are showing us we have problems on the network end to end," says Walt Magnussen, director of telecom at Texas A&M University in College Station, which uses Apparent's AppareNet Voice software probe to support VoIP and video links to remote locations. "This new tool shows you where it is and what it is."
VoIP call quality management includes active and passive approaches, and particular products can encompass both. The active or intrusive approach is exemplified by British Telecom spinoff Psytechnics with its algorithms derived from years of subjective voice-quality testing. The method includes installing a thin client on various endpoints - such as phones, gateways and call servers - to take local readings and return a Perceptual Evaluation of Speech Quality metric.
"You insert a reference signal at one point, measure it at the destination, and through the use of algorithms calculate what the Mean Opinion Score would be," says George Hamilton, senior research analyst at The Yankee Group. "It's an active test, and it's more accurate." MOS is a metric of how good a voice call sounds (see story ).
This intrusive method allows for very targeted testing of specific network links and elements. The measurement goes end to end and establishes a baseline for comparison. Among the weaknesses, the simulated traffic eats up network bandwidth. Also, intrusive testing can be difficult when you don't control the entire path from one end to the other.
Telchemy championed a passive, non-intrusive approach that produces voice quality metrics extrapolated entirely from network statistics such as packet loss and jitter. The company's VQmon technology has been embedded in a range of third-party IP phones, gateways, probes, analyzers, switches and routers to enable real-time monitoring of VoIP call quality.
AppareNet Voice has been described as a hybrid of the non-intrusive and active approaches. The product generates test traffic and then uses software probes to listen to it, but doesn't require the installation of appliances or software agents at the remote sites.
"You just need to know the IP address at the other end, and you can do hundreds of sites an hour," says Gary Audin, president of Delphi, a network integrator in Arlington, Va. "Once you have to start putting agents in all the remote locations, it's a much bigger undertaking."
The remote locations in Texas A&M's VoIP network include extension facilities and experimental agricultural sites in Latin America, Africa and the Middle East. To help serve such areas, the university, an official Internet2 Technology Evaluation Center, also is working with the American Distance Education Consortium (ADEC) at the University of Nebraska and satellite provider Tachyon Networks on the prioritization of VoIP traffic over satellite links.
Texas A&M recently used AppareNet Voice to assess the voice capability of a link with an agricultural research center in Kenya, which starts with a terrestrial gateway connecting the U.S. and Europe and then moves onto a satellite link for the rest of the trip. The tool pinpointed a software problem in the infrastructure of the satellite provider, which could not come up with a cost-effective fix, forcing Texas A&M to look for an alternate carrier.
However, the university had better luck with the ADEC trials testing a path involving a satellite link out of San Diego, serving locations in Latin America. AppareNet Voice found a problem with a router, Tachyon replaced it, and the connection now supports VoIP.
AppareNet Voice sends out several hundred packets of various types to each router along the way and then uses the returning information to characterize the link and assess any problems.
"With this kind of tool, I can look into another carrier's network in a fairly non-intrusive fashion, without installing any software anywhere else," Magnussen says. "It's hard evidence for the vendor or service provider. The measurements aren't precise, but heck, for something you can come up with on demand in 4 or 5 minutes - it's pretty good."
Others feel the value of local appliances or agents throughout the network is worth the upfront time and expense. Apptis, a large network integrator in Chantilly, Va., uses the Qovia VoIP Monitoring and Management System internally and in VoIP implementations for clients.
"What struck us was its ability to do real real-time monitoring," says Mark Melvin, a senior solutions engineer at Apptis. "Qovia is monitoring the actual traffic stream, vs. looking at the call detail reports after the fact."
Melvin also applauds Qovia's distributed architecture. There is no agent running on the Cisco Call Manager server, which simplifies troubleshooting when a Call Manager process causes the problem. If an alert threshold is hit during a call, the troubleshooting can start before anyone complains.
The Qovia ION appliances can be software-upgraded remotely, and the reboot process is fairly quick. If the central Qovia server is brought down for maintenance, the remote appliances can continue to gather information and will report it as soon as the server comes back online.
"With the new scaled down ION appliances, Qovia becomes a much more scalable and cost-competitive solution to distribute across your entire environment, making it competitive with some software-only solutions," Melvin sums up.
The cost of quality
Voice quality testing and monitoring don't come cheap, which is one reason the vendors have, with few exceptions, focused on the service providers, testing laboratories and big integrators up until now. Some of the new enterprise-scaled products start in the $30,000 to $40,000 range.
"You can pay that much for a one-shot network assessment service, or buy our The Hammer VoIP Test Solution for Enterprises bundle and have it to use again," counters Jeff Fried, CTO for Empirix, which introduced its first enterprise-targeted products at VoiceCon last month.
Call center UpSource uses the Empirix product to test the stations of remote call center agents using softphones. UpSource CTO Mark Burns says he wanted something that could continually make assessments so new VoIP applications and VoIP capacity could be added on demand.
"We are in the business of solving customers' problems, so we can't have lots of complaints about voice quality as we move into convergence, any more than we can have lots of down time," Burns says. "These tools are important for the same reason that network redundancy is extremely important."
Qovia tells prospective customers that they can expect a good VoIP call quality management product to add 10% to 20% to the cost of a new VoIP system, and Melvin of Apptis says it is worth the premium. "The businesses that understand the value of network management will understand the importance and necessity
of these tools. As you move from traditional to IP telephony, there are a lot more pieces to manage," he says.
Initial customers tended to seek out VoIP call quality products in desperation, after an implementation had already gone bad. However, vendors detect a shift.
"We're starting to see more traction with new installations," says Nick Orolin, a vice president at Integrated Research, maker of Prognosis for IP Telephony for the Cisco environment. "Enterprises are budgeting for VoIP management upfront and bundling it in with initial deployments."
Big voice-equipment manufacturers such as Nortel are bundling call quality management into their VoIP platforms as well. Erlanger Health System, which started migrating its five-site Nortel shop to VoIP in 2001, quickly saw the value of enhancements in the Succession 3 switch that could feed jitter and packetization statistics and other key VoIP call characteristics into Nortel's Optivity Telephony Manager.
With these tools, a more voice-oriented stress test could be done before upgrading another group of phones to VoIP. "We get a good call quality baseline upfront, instead of just sticking a bunch of phones out there and then seeing what problems we encounter," says John Haltom, network director for Erlanger, a public teaching hospital affiliated with the University of Tennessee School of Medicine in Chattanooga.
Now on the Succession 4 IP PBX platform, Erlanger has begun a trial of the call- quality testing and monitoring capabilities in Nortel partner NetIQ's AppManager 6.0. Tapping into enhancements in the Succession 4 server and Phase 2 IP phones, AppManager 6.0 can provide deeper information about calls in progress.
"Now testing and monitoring technology is embedded in the phone itself, and the phone can report to the switch whenever it has an issue," Haltom says.
Erlanger has migrated 900 of its 1,300 desktops to IP phones, and 800 of them are Phase 2 models that contain the necessary processing power, RAM and software to participate in this type of proactive management.
"A lot of organizations can probably get by without proactive call quality management, but we are a hospital, so we have to plan for the worst and engineer our network a bit differently," Haltom says. "NetIQ's AppManager 6.0 is a true enterprise-wide network management platform that looks at voice quality management with the help of the switch and the phones, and reports alerts in any way that meets your needs."
A self-healing future?
Once VoIP call quality monitoring systems are in place, it is tempting to speculate about what a little artificial intelligence might add to the mix.
The idea would be to incorporate some expert systems technology into the solution and make VoIP networks self-healing. The gateways could take the call quality information and use it to move traffic around to the more optimal paths.
However, the cultural hurdles might be bigger than the technological hurdles.
"Integrated Research actually supports some very interesting automation, and a lot of people express an interest in the self-healing concept," Orolin says. But in the final analysis, no one is willing to stick their necks out and actually implement it and trust it. "I think rules-based automation is a good five years off."
Learn more about this topic
Breidenbach is a freelance writer in Nevada. She can be reached at firstname.lastname@example.org.