You pay a provider for a service, and that service provider delivers, right?
That's what David Taylor-Kraus, president of Atlanta Web design firm Digital Positions, thought until his Verio DSL Internet access went dead because of a line cut.
To get the problem fixed, his company would have to call Verio, and Verio would call NorthPoint, the DSL wholesaler that provided the DSL service to Verio. NorthPoint, in turn, would call BellSouth, which owned the line that had been cut. Each step through this multilayered service would take 12 to 24 hours, Taylor-Kraus feared.
In the past you may have had to deal with one or two providers - a local carrier and a long-distance carrier - for your network needs. But now the complex network services you buy can rely on four or five or more providers lurking in the background. And as companies put more of their business online and outsource support to application service providers (ASP) and Web hosters, the situation becomes even more complex, says Laurie McCabe, vice president and service director at Summit Strategies.
Keeping tabs a challenge
"It's hard for one company to have all the expertise from Great Plains [business applications] back down to the DSL connection. That's a lot of ground to cover," she says.
The days of dealing directly with all your providers are numbered. Users must thoroughly investigate these services before they buy and aggressively pursue fixes when problems arise, experts say.
If Taylor-Kraus had it to do over again, he'd buy from a provider that owned all of the elements that made up the offering. "I would absolutely rather buy a service from the primary provider," he says. "They can control the process."
In the end, Digital Positions called the third-party installer that put in the original DSL link. That third party coordinated the other providers to get the line back up and running within a day. If Digital Positions didn't have a close relationship with the installer, though, the outage would have lasted far longer, Taylor-Kraus says.
That was a year ago and was the only service interruption until last month, when the line went dead for good because NorthPoint went belly up, again demonstrating the vulnerability of such interdependent service offerings.
Paul Scheusler, network manager at Advanced Energy Systems, tried to avoid a problem like the one Digital Positions faced, but he was hit anyway.
He needed remote access DSL lines, so he went directly to Rhythms NetConnections, a national DSL provider. Scheusler figured that since Rhythms would own the service, the provider would be able to fix any problems directly. But Rhythms shuttled him off to an ISP called Intap that resold him the service set up by Rhythms.
Last month, when Scheusler's DSL-connected users could not connect to the Advanced Energy Systems network, he discovered Rhythms had pulled out of the region and taken its IP addresses with it. In the meantime, he has received new IP addresses from Intap and service has been restored.
"But I still don't know who my upstream service provider is now," Scheusler says.
Even so, having a multitiered vendor relationship isn't necessarily bad, some users say. "My take on it is, as long as it's working, I don't care how many people there are behind it," says Jeremy Thomas, a technical specialist with Macayo Restaurants in Arizona. While Macayo's 12 DSL lines are supplied by Rhythms, Thomas deals only with his ISP.
"I don't need to worry about Rhythms at all," Thomas says. "If a circuit goes down, I just call my ISP and everything's fine." And he likes the idea of one point of contact and one service-level agreement (SLA) that covers the multiple providers, as does Andrew Feldman, a vice president at Riverstone Networks of Santa Clara.
Riverstone hosts applications and storage systems with Exodus. Feldman bought a metropolitan Ethernet link from Telseon to connect to Exodus' data center, and later Exodus packaged wholesale Ethernet access with its hosting and storage services. Feldman says he would rather have bought the access and service bundle because of the single point of contact and the single SLA, provided that the Exodus SLA withstood his scrutiny.
"There is a long history in the telco market of wholesaling and retailing what other people wholesale. So we wouldn't be concerned at all," Feldman says. "Now if Exodus showed itself unable to effectively deliver a solution, clearly that would be a problem."
Such problems are bad for business, and service providers are sensitive to them. Greg Falconer, director of dial and DSL products at Verio, says the NorthPoint situation is one of the most challenging he has faced, a "catastrophe." Within four hours of hearing about NorthPoint's impending demise, Verio notified customers and the next week urged them to switch to dial-up accounts that had been set up for them, T-1s or EarthLink DSL.
Looking beyond legal obligations
Legally, Verio was off the hook for the SLAs it offered to customers on the DSL service backed by NorthPoint, because of a contract clause that voided the SLA if the DSL provider went out of business, bankrupt or terminated services. "We do recognize and understand the relationship we developed with these customers and did our best to provide services that we could control and guarantee," Falconer says.
Similarly, ACT.world, an ISP in Montgomery, Ala., was left in the lurch by DSL provider BlueStar, says Sabrina Norrell, a network administrator with ACT.world. When BlueStar announced it was pulling out of Montgomery, ACT.world arranged for another provider, ConnectSouth, to serve its DSL customers. But then ConnectSouth picked up and left before that migration was complete.
"When ConnectSouth left, of course our customers looked to us and asked us what we were doing," Norrell says. "Some of them were very irate. They look to the people who meet them face-to-face to follow up on their problems, and that is us. It's sometimes hard to explain we've done everything we can."
Michael Caspar, CTO at SirenServ, an online content provider, buys Web hosting services from Verio. SirenServ buys access to those services from SBC Communications, which in turn relies on UUNET for part of the access link. Sometimes when problems arise, SBC and UUNET point fingers at each other. He says in one instance, "after screaming and hollering," he got through to a UUNET engineer who finally moved SirenServ's service off a broken router.
Caspar says he's happy with Verio's service, but he notes its data center and network operations are separate, even though they are part of the same company. That can slow problem resolution, he says.
Through all this complexity, Caspar stands behind the service he provides to his content customers. And when his providers drop the ball, he finds himself pointing fingers to explain why SirenServ's service has failed.
"The fact of the matter is that our site was down, and that's how they look at it. They just want the site back up. It's our reputation that ends up getting bruised in the eyes of our 300,000 users and 250 paying customers when something like that happens," Caspar says.
Try to keep things simple
He recommends picking providers carefully. "It's a matter of doing your homework and trying to keep as few moving parts as possible," he says.
When times are good, providers prefer to keep customers unaware of the layers behind the service. E-Tunnels, which wholesales managed VPN services to ISPs, trains the help desks of the ISPs it serves, says Derek Ferguson, e-Tunnels' COO. "If they can't deal with a question, they put the customer on hold and the ISP calls our technical assistance center," Ferguson says.
But the goal is to give the service a single face. "Customers don't need to be aware of this back-end connection," he says.
But end users should be aware of what goes on behind the curtain, says Chris Richter, director of product marketing at Exodus. "They need to understand what comprises a service-level agreement and really understand how service is delivered, and not take things at face value," he says.
