VoIP phone systems still lag significantly behind the traditional phone network for reliability and quality, and there are major differences in quality among brands of VoIP service, according to a company that evaluated six providers in the U.S.

Keynote Systems tested the services for reliability and quality by making domestic long-distance and local calls from VoIP services to the PSTN in May and June this year. Call quality was evaluated by software and used an industry standard for listening quality, according to Dharmesh Thakker, senior product manager at Keynote. The study involved more than 150,000 automated VoIP-to-PSTN calls. Keynote evaluated these kinds of calls because they are the most common type made by VoIP users, he said. The company tests Web application performance and is branching out to VoIP.

The survey evaluated VoIP services accessed via DSL and cable modem, which are geared mostly to consumers. It gives valuable new data on the state of such services, which are exploding in popularity, said IDC analyst Will Stofega, who is familiar with the results. IDC expects 3 million U.S. consumers to be using VoIP by the end of this year and 28 million by 2009, he said.

Vonage's service came in first place for reliability, a measure that included how often the service was available, the length of any outages, the average number of call attempts before a call went through, and the number of dropped calls, Keynote said. Vonage scored 80 points out of a possible 100. There was a noticeable gap in reliability between Vonage and the other providers, which included AT&T 's AT&T CallVantage, 8x8's Packet8 Broadband Phone Service, Primus' Lingo, Skype's SkypeOut and Verizon VoiceWing, Keynote said.

AT&T CallVantage ranked first for audio clarity, which included audio delay, listening quality, and the consistency of both of those factors over call time and distance. It earned 78 points out of a possible 100, and CallVantage's quality score was noticeably better than those of other providers. The listening quality rating was based on the ITU-T P.862 standard of the International Telecommunication Union and was determined using software instead of subjective human evaluation.

Though they outscored other services, even Vonage and CallVantage have some room to improve, according to Keynote.

The IDG News Service is a Network World affiliate.