A team of 20 iLabs engineers spent eight days running more than 1,100 SIP interoperability tests, and the conclusion is that having multi-vendor VoIP devices work together is by no means a given.

Last year , the VoIP using SIP iLabs team saw a high degree of basic interoperability between different SIP phones and proxies. When we expanded the testing this year to include enterprise VoIP features, standard security parameters and video (see story ), we encountered significant failure rates. Out of 1,113 interoperability tests 18% were outright failures.

All told, we tested 75 products from 25 vendors, including IP hard and softphones, videophones, SIP proxies, firewalls, and wired and wireless switches. Our test of enterprise features (see How we did it and test plan stories) focused on those required for large-scale VoIP deployments including call forwarding, call waiting, message waiting indicator, blind and attended call transfer, hold/resume with music, and dual-tone multifrequency testing. We chose features to test SIP interoperability, and didn't look at features that don't require interoperability such as last number redial.

We ran into some major interoperability problems early on in the testing. For example, 3Com's 655 Series SIP phones proved to be so recalcitrant in their SIP implementation that 3Com engineers pulled them from the test bed.

We locked up one of Grandstream's ATA-HandyTone 486 devices, and iptel.org's SIP Express Router running released software had serious problems completing calls properly that were only solved by upgrading the box to an unreleased version of the software.

While it's important to note that we had failures across the board, the enterprise feature that requires the most SIP interoperability work is call transfer. With call transfer, among the phones that supported both blind and attended call transfer, only 43% passed the test. Call transfer requires that many SIP messages get passed around to send the call between phones without dropping the call or the audio. In the case of an attended call transfer across a single SIP proxy, there is a minimum of 46 SIP messages that must align for the transfer to work properly.

Even with the less-than-optimal test results across all features, we had strangely anomalous behavior. For example, we conducted a significant amount of the testing via speakerphone. However, when we re-tested a Grandstream phone using the handset instead of the built-in speakerphone, it failed the same test it had previously passed.