Growing pains test Qwest

By David Rohde
Network World, 11/08/99

DENVER - Cracks are beginning to show in the shiny armor of America's newest national carrier.

Users and consultants charge that Qwest Communications, which recently completed its 18,500-mile national network, is missing installation dates on private lines, frame relay nodes and Web hosting facilities. Some customers are finding it hard to get through to support lines to verify orders, and a few have suffered spot outages on already-installed lines.

Qwest officials last week confirmed the company recently suffered a frame relay and ATM outage in Atlanta and a failure of some ISDN circuits in Los Angeles. They also confirmed that Qwest's Web hosting facility in Sunnyvale, Calif., ran out of hosting space less than two weeks after it opened over the summer and the carrier has since been unable to fulfill new orders there.

But Qwest executives insist they're on top of the situation. "I acknowledge that there may be customers that have been disappointed in our performance, but we don't have a systematic maintenance and provisioning crisis in this company," Qwest President Afshin Mohebbi says.

Yet Mohebbi confirmed industry whispers that Qwest ran out of optical backbone equipment from its principal supplier, Nortel Networks, during the second quarter. Qwest then turned to rival Lucent - via an emergency call from Qwest CEO Joe Nacchio to Lucent CEO Rich McGinn - to obtain equipment to "light" additional fibers in the Qwest network.

"Have we been surprised with a huge amount of de-mand for our products? Yes," Mohebbi remarks. Qwest has now increased its capital budget $200 million and is increasing orders with Nortel and Lucent, according to Mohebbi, who adds: "We're not going to cut any corners."

But some users say Qwest is barely in a position to take new orders, much less fill them correctly. Dan Healy, senior vice president of Rockefeller Group Telecom-munications Services in New York, says he placed an order with Qwest in May for two T-1 lines to Europe and is still waiting. The problem hasn't even been the installation but rather establishing a valid account with Qwest customer service. "We finally got the account set up when I sent them a termination letter," Healy says. Qwest simply has "too much business" right now, he adds.

Some analysts say they've seen chronic delays with Qwest. "I've got a [Qwest] customer who is working on a 90-site frame relay network, and it's taken them most of this year to get it installed, and they're beyond frustrated," says Lisa Pierce, a telecom analyst with Giga Information Group. Other users have ordered Qwest frame relay circuits as backup following recent major carrier outages, Pierce says, yet "the service is not there when they need to use it."

In Atlanta, where Qwest recently won several big contracts from companies including Delta Airlines, a frame relay/ATM switch recently failed for part of one day after some cards from Lucent went bad, according to Qwest. For Delta, that exacerbated problems that began last spring after it signed contracts with Qwest and MCI WorldCom for fully redundant T-3 ATM links into 26 airports nationwide.

"[Qwest] got off to a real rocky start. They didn't realize the magnitude of this effort," says Paul Millard, Delta's vice president of engineering. The carrier missed installation dates because of problems connecting with local carriers, Millard says. Qwest also couldn't light a network segment from Atlanta to south Florida because of the optical manufacturing shortage, so it had to lease additional capacity. And support out of its network operations centers in Denver and Arlington, Va., "was less than satisfactory," though it has now improved, and 24 of the 26 T-3 links are in, Millard adds.

In northern California, Qwest officials say they misjudged demand for Web hosting in Sunnyvale and almost immediately ran out of space they expected to last for 12 months. The company is now building an additional 1.6 million square feet of hosting space nationwide. And in southern California, a user consultant who asked not to be identified charged that Qwest has had periods in which no ports were available on its two Nortel DMS 250 telephony switches.

Qwest officials deny the Los Angeles switches are maxed out but confirm that 18 customers' ISDN Primary Rate Interface circuits recently went down for three days. The problem occurred because a technician failed to "fill out a data field" in a software load, says Mack Greene, Qwest's vice president of voice and data product management.

Qwest officials also say they are suffering delays of their own getting Pacific Bell to pony up local loops, but outsiders say that explanation only goes partway. "It's taking four months to get a T-1 installed," says the local consultant. "Their network is way oversold."

Indeed, Qwest sells through direct and indirect channels, and has already recruited 7,000 Microsoft-certified value-added resellers to bang the drum for Qwest services. Last year, Qwest gave its sales force tools that enable salespeople to place mass-market orders directly over an extranet, Greene says.

But even some high-profile custom orders seem to be falling through. For example, Qwest is the carrier for the Abilene Project - a high-speed research network linking dozens of universities that was announced by Vice President Al Gore last year. Qwest has had "execution and delivery" problems getting many of the schools their promised OC-3 links, with some that have placed orders as long ago as February still waiting, says Jeff Crowder, project director for communications network services at Virginia Tech University.

A Qwest spokesman says the Abilene orders are taking an average of three months but concedes some have fallen behind because of internal provisioning issues or delays from local carriers.

Qwest's challenges are coming at a critical time, because ever since MCI WorldCom announced it would acquire Sprint, Qwest has openly touted itself as Sprint's potential replacement as a comprehensive No. 3 carrier. In fact, last year Qwest acquired its own established long-distance carrier, LCI International, and began selling traditional circuit-switched services along with the high-capacity IP bandwidth it touts publicly.

That helped Qwest increase revenue (see graphic), but some users feel the buy stretched Qwest thin - not to mention its pending acquisition of US West.

And even though Qwest's physical construction is complete, officials admit that only a few of the 48 available fiber pairs at any place of the network may be lit with the proper electronics, forcing Qwest to continually raise order forecasts to meet demand. "The key point is the manufacturing issue," Mohebbi says. "Anything that we can get our hands on, we get our hands on."

Users are rooting for Qwest to fully pull itself together. Crowder says Qwest is offering T-3 Internet access links for $10,000 per month, compared to up to $35,000 from other carriers. That heartens Qwest officials, who are only too happy to take a swipe at MCI WorldCom and Sprint when asked why Qwest will continue to sell hard.

"Dumb and Dumber getting together is one of the best reasons," Mohebbi says. "People want to buy our stuff."

Reaction: Here's what some Fusion users are saying about this issue: What do you think? Add your comments to the thread