Top tips for skating through the telecom meltdown

"In skating over thin ice, our safety is in our speed," wrote Ralph Waldo Emerson in his 1841 essay "Prudence." Such prudence certainly applies to network executives traversing the thin ice of the telecom industry. While bankruptcies, financial shortfalls, boardroom scandals and the like do not cause immediate, rampant service failures, they are treacherous conditions that require quick movement toward contingency plans.

As sure as heat erodes an ice sheet's strength, a provider's massive layoffs, indictments of high-level executives, uncertain reorganization plans, worthless employee stock options and decline in new technology investments eventually will affect its service - perhaps even fatally. Signs of degrading service include missed installation deadlines, longer trouble-ticket resolution or increased finger-pointing, extreme difficulty in resolving billing mistakes and generally sloppy customer service from embittered, beleaguered employees.

Naturally, network executives are playing it cautious. They are seeking alternatives to guarantee that looming service problems do not disrupt networks. The central trade-off is between cost savings from volume discounts using one provider and the increased reliability of contracting with two or more carriers, they say.

"Using multiple [service providers] can provide a completely stable network, but it will increase my costs 25% a year - at a minimum that equates to $100,000 a year," says Bernie Lubitz, director of telecommunication technology and services for Martin Memorial Health Systems (MMHS), a hospital system with 10 locations in southern Florida. That's especially harsh during this slow economy when managed healthcare is being pressured to provide better care at lower costs.

MMHS traditionally has gone the cost-savings route with a single provider, most recently troubled competitive local exchange carrier (CLEC) Adelphia Business Solutions. While Lubitz and his legal staff did due diligence on Adelphia's finances before contracting with the carrier two summers ago, they could not have realized the buried accounting improprieties that would lead to a Chapter 11 bankruptcy filing in March of this year. The CLEC could be the poster child for the telecom industry's woes. Adelphia has crumbled under a massive debt load and leadership practices so far afield that, in July, federal prosecutors filed an indictment for alleged corporate pilfering against four Adelphia executives, including founder John Regas and his two sons.

Even though Adelphia has obtained a cash infusion that bodes well for continued operations and it has delivered uninterrupted, quality service to MMHS throughout this period, the CLEC's troubles have given Lubitz pause. He says he will not simply renew his expired Adelphia contract, but is investigating all options for leasing critical WAN fiber. "MMHS can't cost-justify laying its own fiber, so is evaluating options from BellSouth and CLECs, including Adelphia. But we've learned that dealing with CLECs can be extremely painful," Lubitz says.

Contract wrangling

If you, too, are feeling the pain, understanding your options can bring relief. Leave the details of your services contracts to legal counsel, but get a grasp of the basics. First, know that you can't terminate your contract with a service provider simply because it files for bankruptcy - even if your contract explicitly states that you can, says Hank Levine, a partner with Levine, Blaszak, Block & Boothby, a Washington law firm that specializes in user telecom contracts. Such statements, called ipso facto clauses, "are not enforceable in the United States, period," he says.

As soon as a carrier declares bankruptcy, financial decisions rest with the overseeing bankruptcy court, which will immediately freeze contracts into place, says Colleen Boothby, another partner with the firm. Ending a contract requires petitioning the bankruptcy court.

You won't find the court sympathetic. A bankruptcy court's goal is to protect creditors - not customers. Most service providers in bankruptcy protection have neither turned off their networks nor even missed uptime requirements or other significant technical service metrics. They are not in breach of their service-level agreements (SLA). Rather, service tends to deteriorate first in what Levine calls the "soft aspects," such as responding quickly to your phone calls or fixing a billing error. In the unlikely circumstance that your SLAs spell out expected performance in soft aspects, your lawyers may be able to use failures there to convince a bankruptcy court that your carrier breached the contract. But even with evidence that the provider has missed some of its hard-and-fast technical SLAs, the court will not likely terminate a source of revenue like your monthly checks.

A carrier's bankruptcy can have two positives for customers, Levine says. For one, operations replace stock price as the company's primary focus. Now is a good time for you to press your account team into operational-related promises, perhaps shoring up their commitments to soft services (even if such promises don't hold the legal weight of a formal SLA).

Secondly, when a company sheds its debt, it can price services aggressively - and is motivated to win new business or renegotiate existing contracts. This is an excellent time to negotiate extremely favorable prices, maybe even helping you fund the expense of a second, back-up carrier.

Check your assignment

Bankruptcy proceedings typically take a year to 18 months to complete, after which anything can happen - the company might emerge and continue to function, or it might be sold in entirety or in pieces.

Watch for news of a potential sale. Such a sale could affect you because your contract could be among the assets sold in a process called assignment. While your contract might include a clause forbidding assignment, it, too, is unenforceable, Levine says. But, should your contract be assigned to a carrier unable to provide all specified services, such as datacom and long-distance, you might have legal recourse, he adds.

If you are aware of a potential sale and are concerned that your contract might wind up in the hands of a carrier you don't want, you'll need to get your lawyers hopping right away. They might be able to get your opinions heard by the bankruptcy court, perhaps by linking up with other, similarly concerned users.

When it comes to working with bankrupt carriers, size matters, too. As the largest corporate bankruptcy to date - and of a dominant carrier - contract issues with WorldCom are far more complex than they are with small CLECs, Boothby says. For instance, even if you intend to stand by WorldCom, you might not have that option. If WorldCom emerges from bankruptcy and continues operations, per typical practice, it would get to accept or reject existing contracts, no matter their terms. No one knows how much time customers would have to move should their contracts be rejected. Several large WorldCom customers have banded together to lobby the bankruptcy court for a ban on contract rejections and other issues. Consider pressing your legal team to join them. But even if your company remains apart, the involvement of these users could affect how the courts treat any request you submit, so have your legal colleagues keep you abreast of such goings-on. WorldCom also might want to insert a rejection-rights clause into any new agreement you make at this juncture. Creditors often push for this so that the company can preserve maximum flexibility, Levine says. Make sure your lawyers watch for such a clause. If WorldCom won't remove it, you could be better off with another provider that can offer competitive rates at a guaranteed term.

Owners or lessors

If your contract expires, or you have the legal wherewithal to end it, your next challenge is finding a better replacement. This obviously won't be easy given the number of troubled providers these days. One consideration is whether to align with a facilities-based provider that owns its own network infrastructure or a services-based provider that leases capacity from multiple carriers. The classic belief is that financially stable facilities-based providers such as AT&T are the safest bet for a long-term contract. But contracting with a services-based provider for WAN links is often equally safe, and in some instances safer, says Steven Taylor, president of Distributed Networking Associates and Network World's "Packet Evangelist" columnist. In the WAN, a services-based provider can tap into many more paths, owned by multiple carriers, than a facilities-based provider locked into its own infrastructure. Taylor offers as an example Virtela Communications, one of Network World's 2002 10 start-ups to watch. Virtela buys Internet bandwidth from multiple sources and moves traffic from one connection to another if performance on a particular link wanes (see story, "Wares extraordinaire").

As long as the services-based carrier can offer you evidence that it is financially well-heeled, this might be a way to get the performance benefits of splitting traffic among multiple providers while obtaining the financial benefits of a volume discount with one carrier. Of course, many services-based providers are start-ups that don't have the income or the cash reserves that the nation's large facilities-based carriers do, so future instability is a concern. Should you take this route, don't contract for more than two years of service, and be sure to spell out many soft-aspect SLAs.

Splitting the traffic

The safest bet remains dividing services among two or more providers. But that's not an automatic safety net. Certain other precautions are wise, says Barry Nance, a consultant, a Network World Test Alliance member and author of books such as Introduction to Networking.

"For maximum reliability, ask both telecom vendors to give you a detailed description of the physical data path each link uses. If both paths happen to occasionally share physical space or connections, such as between your site and the local [central office], evaluate your risk for something calamitous happening to both links," Nance says.

Because cost is almost everyone's issue these days, determine which links are critically important, and contract for a back-up provider for only those links. Then, because you are paying for both links anyway, "activating the second link as additional bandwidth and splitting the workload across both links is a natural thing to do. If both links are continually active, you don't have to take any steps to cause failover from one link to the other when problems occur," he adds.

If last mile or remote areas are your most worrisome vulnerabilities, consider fixed-broadband wireless such as microwave or satellite, suggests John Hatton, a telecom director at Burlington Northern and Santa Fe Railway Co. His experience using fixed-broadband wireless spans 15 years. In that time, he's seen microwave and satellite technologies improve to the point of high reliability.

"Depending on the level of redundancy you want, satellite technology [not really good 15 years ago] and . . . microwave are certainly worth another look, especially for remote areas," he says.

MMHS' Lubitz agrees. A microwave carrier is among five providers the hospital short-listed as it researches telecom options. It is considering the carrier for secondary service now that such providers claim "five-nines" reliability, Lubitz says.

Any of the various flavors of wireless links, from satellites to line-of-sight antennas, can provide the same kind of network interface enterprise users are accustomed to for T-1 or frame relay, Nance says. So the mechanics of setting up such connections won't require special equipment or training. But don't try to use such links only when failures occur, but load-balance between them so "when a problem happens in just one link, you'll merely suffer a lessening of bandwidth, not an outage," he advises.

One final tip: Be creative. Think about working with peers - perhaps other members of a user group or an industry-specific consortium - to negotiate prices with a carrier, users say. This can be particularly helpful in off-setting a loss of a volume discount when splitting your business among providers. Likewise, Boothby recommends, consider joining a telecom regulatory watchdog group such as the Ad Hoc Telecommunications Users Committee, which helps see that users are fairly treated by regulatory agencies over pricing and other issues. By moving quickly but prudently, your network surely will weather this telecom meltdown.