VoIP gateways side by side

To implement voice over IP to any extent you need a voice-over-IP gateway -- a seemingly magical box that transforms telephone calls between plain old telephone service (POTS) on the one side and an IP data network on the other. We brought three of the leading voice-over-IP gateways into the Mier Communications' lab in Princeton, N.J., and found out what makes them leaders in this new, but burgeoning, voice-over-IP marketplace.

---

---

We awarded the Network World World Class Award to Nuera Communications' ORCA GX-8. The ORCA GX-8 supports the best redundancy and the most straightforward and complete management capability of the products tested.

This was a close match, and Cisco's AS 5300 gateway, featuring the lowest per-channel price, finished just inches behind Nuera. Clarent's Gateway 400 also turned in a laudable performance and earned high ratings for its integral features, but slipped behind the other two, mainly in management and ease of use.

More than one dozen vendors of voice-over-IP gateways were invited to participate in this open competition. To qualify, the gateway had to support a minimum of one -- and up to four -- T-1s of voice traffic load. This equates to a capacity of roughly 100 concurrent real-time telephone calls.

Rating voice over IP

Our evaluation of voice-over-IP gateways was based on:

• Performance, which encompassed more than a dozen measurements and metrics including voice quality and interactive call quality under various settings; latency; bandwidth consumption; call set-up time; call completion rate; and redundancy failover.

• Features, including the scope of settings and parameters accessible to a user for adjusting and tuning the voice-over-IP network; the gateway's ability to set bits in the voice-over-IP packets for prioritization; and transparent support for fax and modem calls.

• Management and administration criteria, including intuitiveness and effectiveness of management interfaces -- whether graphical or command-line; real-time monitoring capabilities; scalability for centrally managing many gateways within the same call agent domain (per Media Gateway Control Protocol, or MGCP) or gatekeeper zone (per H.323); and additional management capabilities such as event, alarm or trap monitoring, and the generation of management reports.

• Configuration criteria, including telco interfaces supported; vocoders supported; modularity and density; and support for redundancy and component-failover configurations.

• Installation, ease of use and documentation.

Nuera out in front

We found that all three systems are highly reliable, deliver very good voice quality and interactive call characteristics, support most key features and can be adequately managed.

Nuera's ORCA GX-8

However, we view Nuera's combined gateway and call agent package as slightly ahead of the competition in several respects.

A call agent is a separate, stand-alone node defined in the MGCP specification that handles call routing and call setup. An MGCP call agent manages multiple gateways within a mutually exclusive area called a domain. The call agent functionally is akin to a gatekeeper, as defined in the H.323 specification, which oversees gateways within an administrative area called a "zone".

These differences in terminology underscore a major concern that permeates the voice-over-IP industry -- a proliferation of functionally equivalent standards.

The three voice-over-IP gateways we tested each employ different call-control protocols (see table). Nuera's product is based on MGCP. The vendor says that it also supports another, related standard called Session Initiation Protocol (SIP), which Nuera uses for communications across MGCP domains, and between call agents.

The Nuera call agent software runs on a dedicated processor running Hewlett-Packard's HP-UX. As with the other products, we tested the Nuera gateways with up to four full T-1s of traffic and we would "slam" the system by delivering calls to the gateway at a rate up to 48 calls per second.

The Nuera system didn't flinch at this load: The successful call completion rate always exceeded 99.9%, and calls were always set up and ringing at the other end in less than a second. What's more, voice quality was consistently very good to excellent (see graphic). Nuera's calls exhibited the lowest latency -- just 63 msec one way, end to end -- of the three gateways tested.

All three gateways consistently garnered voice quality ratings well above the 4.0 level that equates to toll quality. The ratings are based on blind panel assessments of male and female voice recordings that have been sent through the voice-over-IP systems, using a five-point ITU-specified rating scale and test procedure called the Mean Opinion Score (MOS). Even using various bandwidth-conservation techniques -- including low-bit-rate coders and voice-activity detection -- voice quality still scored well above toll-quality ratings.

Nuera's management consists of two clean Windows applications -- one for gateway management, the other for the call agent -- which ran together on a Windows NT management station. Together they offer comprehensive management of the voice-over-IP package. Configuring or checking the status of any channel, card or component is straightforward.

Another big plus for Nuera is the degree of failover redundancy supported by the gateway. Besides power supplies and fans, Nuera's ORCA GX-8 multislot chassis supports a fully redundant, failover control module. You can even upgrade the software of the redundant control module, and then switchover between active and hot-standby modules without dropping calls or having to reset or reboot the gateway. We note, too, that Nuera's Unix-based call agent can also be configured with a redundant hot-standby platform.

On the Mier Scale of Hardness, in which 1 is the village idiot and 10 is a rocket scientist, we feel a Level 5 can handle the setup and management of a Nuera voice-over-IP network, given the gateway, call agent and software tools offered.

Cisco knows voice over IP

According to recent market studies, a significant portion of the current voice-over-IP installed base is running on Cisco AS 5300s, the same voice-over-IP gateway that Cisco submitted for this testing. It's not hard to see why: The system is reliable, full-featured and straightforward to manage.

The Cisco package is currently based on H.323, which is the most widely embraced voice-over-IP protocol. Cisco says it has added MGCP and SIP to its current IOS operating code, but stops short of proclaiming that MGCP or SIP are fully deployable at this time.

The AS 5300 is a Cisco router running a recent version of IOS -- ours ran a version of 12.1. There are special modules required for voice over IP, however, called voice feature cards. The H.323 gatekeeper in Cisco's voice-over-IP package is also a Cisco router. Ours was a 3620, also running the same version of IOS. The gatekeeper cannot function as an MGCP call agent; Cisco says you'd need a third-party call agent today if you want to run MGCP.

From a performance perspective, the Cisco voice-over-IP package fared as well as Nuera's ORCA GX-8. There were no statistically significant differences between Nuera's and Cisco's voice quality, interactive call quality, call setup time or any other metric we measured.

Compared to Nuera's management applications, we thought that Cisco Voice Manager (CVM), a Java application that is included as part of CiscoWorks 2000, was not quite as complete or useful. For example, throughout the CVM software you have to manually refresh a particular screen, data element or table to see the latest updated value. We thought this severely impacted the software's use for real-time monitoring of voice-over-IP activity.

For configuration, however, CVM is more than adequate, and its on-screen help is excellent. It seems to us that by using other pieces of CiscoWorks 2000 as well -- including CiscoView, for example -- you could get a better overall picture of your voice-over-IP network. CiscoView gives you a graphical image of whatever remote Cisco platform you want to manage, including Cisco's AS 5300 and 3620 voice-over-IP systems. Then by double clicking on a particular port or module, you can obtain real-time status and traffic statistics. However, this would be a piecemeal picture, involving various applications and interfaces.

A Cisco tech support person was on site during the testing, and we noted that he used the Cisco command line for virtually all configuration and management tasks. That is not at all uncommon among Cisco aficionados, although the command structure for voice-over-IP network configuration and operations is fairly elaborate and original. But what's another dozen or so commands on top of the existing Cisco command-line interface?

On the same Mier Scale of Hardness, we feel a Level 6 or 7 would be needed to set up and manage a Cisco voice-over-IP network.

There is one other area where we Cisco scored slightly behind Nuera, and that was in configuration. We note that Nuera's gateway, while conspicuously larger than either Clarent's or Cisco's, offers the most options for bullet-proofing the system, and those options make the system much more bullet-proof than the competitors' products. Cisco's AS 5300 does not support nearly the same degree of optional failover redundancy, which we think is key if voice over IP is to take on the existing POTS/public switched telephone network with any hope of supplanting it.

Clarent: One step behind

Clarent offers an H.323-based gatekeeper, but the vendor submitted its Command Center running the vendor's proprietary call-control protocol for this testing. The NT-based Command Center is functionally equivalent to an H.323 gatekeeper or an MGCP call agent. According to the vendor, performance is better running the proprietary protocol, as long as all your gateways are from Clarent.

The Clarent Gateway 400 is also an NT-based system. But don't think the system suffered any inherent stability problem as a result. The Clarent package exhibited reliability just as good as Nuera's or Cisco's, with the same 99.9% successful call-completion rate. The system also handled being slammed with 48 call setups per second and didn't miss a beat.

A couple aspects of Clarent's performance were less than optimum. Clarent's interactive call quality was notably lower than Cisco's or Nuera's (see Figure 2). The reason for this was latency. One-way, end-to-end latency for Clarent voice-over-IP calls was on the order of 125 msec, compared to well less than 100 msec for Nuera and Cisco. Clarent's score in this test was 3.0 to 3.66, below the 4.0 threshold that is regarded as toll quality. The additional 25 msec in one-way, end-to-end latency can readily be detected in our interactive tests. Remember that the net result in interactive conversations is an extra 50 msec round-trip delay, and most listeners would consider that delay an annoyance.

In our testing of interactive call quality we added 10 msec to the actual latency, to simulate propagation delay and additional router hops that real voice-over-IP callers would experience over a 1,000-mile distance. In the case of Clarent, the resulting cumulative delay equated to a minor, but noticeable, annoyance in our interactive tests.

Another small issue was call set-up time. Where Cisco's and Nuera's systems would set up voice-over-IP calls in less than 1 second, call setup via the Clarent voice-over-IP network was consistently longer than 4 seconds.

Clarent fared well in general, however, earning a 7.9 score overall. The lowest scores it received were in the management and administration category, and in installation and ease-of-use ratings. Clarent's management consists of five different Windows applications, which in our estimation, was too many. We concluded that while the applications were all fill-in-the-blank type tools, they were more than a little disjointed and considerably less than intuitive to navigate.

The Clarent systems, both the NT-based gateway and the NT-based Command Center gatekeeper, were the most tedious and difficult of those tested to get up and running. Considerable editing of the NT Registry was also required. On the same Mier Scale of Hardness, we think a Level 8, and perhaps a Level 9, would be needed to correctly set up and run a Clarent voice-over-IP network.

The vendors whose gateways we exercised in this test series are among the marketplace leaders. All three performed well, and were reliable and effective products for rolling out an enterprise voice-over-IP network. Even so, it's clear that in addition to finding the best available voice-over-IP gateways, enterprise networkers face a sharp learning curve in understanding the subtle intricacies of voice over IP and what's involved in making voice-over-IP products work well.

---

Mier is president and founder of Mier Communications, a network consultancy and product test center in Princeton Junction, N.J. Hommer is manager of lab testing at MierComms. They can be reached at ed@mier.com or mhommer@mier.com.

How we did it
Testing methods

Interactive scorecard and NetResults

Nuera Communications

Cisco

Clarent

Product info: Gateway 400
Info supplied by Clarent