Major carriers score high on network performance, but billing woes and installation delays open the door for second-tier providers.Customers think highly of the reliability and performance of\u00a0carrier networks, but aren't so thrilled with the level of customer service, especially when it comes to billing and\u00a0management\u00a0tools. That leaves the door open for second-tier service providers such as Broadwing, Masergy Communications, Savvis Communications and\u00a0XO Communications, according to a Nemertes Research survey of about 100 major U.S. and global companies.While companies are trying to reduce the number of carriers they deal with because managing those relationships is time-consuming, 26% of respondents say they still use between 10 and 20 carriers. In some cases, they use different carriers for different services to leverage better pricing or features; in others, they must use multiple carriers to cover all their locations. IXCs vex LECs (chart)AT&T\u00a0is the most-frequently used carrier, followed by\u00a0MCI,\u00a0Sprint\u00a0and\u00a0Qwest, according to survey respondents. Many organizations are sticking with these carriers in large part because of concerns about the financial viability and geographical coverage of the newer providers.But other customers are becoming increasingly interested in the second-tier providers who offer higher bandwidth for lower costs, and have built their networks with the latest technology, so they don't have the operational issues larger providers have.What's more, the newer players go the extra mile in customer service, some IT executives say. "I'm seeing second-tier carriers succeed by providing better customer service than their Tier-1 counterparts," says John Fulmer, senior technical director of DigitalNet, a consulting firm that uses XO and and uses services from the major carriers. "I don't think customers fear using a Tier 2 as much as they used to."The survey data backs up the notion that customers are not pleased with the customer service they're getting from their incumbent carriers.Respondents rated the carriers on the following factors: network reliability, network performance, technology, cost, customer service, ease of use, management tools, service-level agreements (SLA), installation time and billing.AT&T tops the fieldRespondents were asked to name their primary carriers.Carrier %AT&T58MCI43Sprint33Qwest30SBC25BellSouth20Verizon13Equant10Internap10Level38UUNET5Infonet5XO3BT3Masergy3Broadwing3On a 1-to-5 scale, where 1 is unacceptable, 2 is poor, 3 is fair, 4 is good and 5 is excellent, the overall scores for reliability and network performance were high. Sprint received the top score for reliability at 4.38; followed closely by MCI at 4.18;\u00a0Verizon,\u00a0BellSouth\u00a0and Qwest at 4; AT&T at 3.94; and\u00a0SBC\u00a0at 3.75.Sprint also was tops in network performance at 4.22, followed by MCI at 4.09, Verizon at 4 and AT&T at 3.90.But when it came to a customer service feature like billing, the scores were far lower across the board. Sprint again was first at 3.38, followed by SBC at 3.25, BellSouth and MCI at 3, AT&T at 2.78, Qwest at 2.3 and Verizon at 1.5.Billing has long been a problem for the big carriers. This is the area where just about everyone has a story and a complaint. The only place we found some positive feedback about billing was with the second-tier carriers, in particular Masergy.The problems range from readability to inaccuracy to volume. The bill "comes every single month, and literally, a guy comes off the back of a truck with a dolly of boxes. It takes four full-time people to go through them," says the CTO of a global financial services company that uses AT&T and MCI who asked to remain anonymous. "It's a mess. They have to come up with a whole new billing scheme. These guys don't believe in consolidation. Every node is a different bill and page. I don't know how to describe it."Companies consistently say they spend 10 to 40 hours per month assessing and correcting phone bills. That's led many IT executives to demand better online billing systems - which, so far, they haven't received.Installation deprivationInstallation time is also an area where the carriers have room for improvement. Qwest scored the best at 3.7, followed by Sprint at 3.5.The amount of time companies wait for circuit installations varies depending on the geographical location and the provider. The range for a new T-1 line is four to 12 weeks, with the average at about six weeks. T-3 lines take 12 to 16 weeks. And faster, more complex optical services or diverse lines can take up to 24 weeks. As one university CIO points out, these time frames are after the paperwork is completed, which can take up to two weeks.One would think that the size of a company should dictate the response time for installations. But we didn't find this to be true. "The speed of installation? They all stink at that. There are always a million and one excuses," says a communications director for a government agency who asked not to be named. "Circuit installation is terrible. It takes 45 days for a T-1. T-3 is a complete mystery - 120 days would be good; 180 is average."The lowest overall rankings are for management tools. The only carrier to achieve higher than a "fair" rating is AT&T, with 3.17. The others were sub-3.0, and Verizon was all the way down to 1.75.The tools are unsophisticated, IT executives say. Simply finding out basic information, such as discard eligibility, is difficult from some carriers, respondents say. Complaints about MCI in this area were particularly prevalent. MCI has merged so many companies together that it's become difficult to offer management tools over various operations support systems and networks.Before the dot-com meltdown, the carriers focused heavily on Web-based management tools. But now, IT executives say that their efforts have all but vanished in this area, and they'd like to see some functionality - that's somewhat affordable, too.Carriers are becoming more focused on managed services, but they need to offer more than an all-or-nothing approach, says David Moy, technical services manager for G. James Australia, which is a privately owned group of glass and aluminum companies. "We're looking for a more complete service from carriers, but now, you either buy bandwidth or they run the whole network. There's no in-between," he says. "There are points where you don't want to manage the network, but you don't want them to do everything. We run 24-7, but we don't like to wake someone up when we have a problem at 2 a.m."Many companies say that managed services work well for them, but some said the services didn't meet expectations. One manufacturer tried to use AT&T's managed services for 18 months, "but it flat out didn't work," says a network manager who asked not to be named. The company switched to Sprint and pays\u00a0Visual Networks\u00a0to monitor the network using Visual's DSU\/CSUs at remote sites.MCI focuses on customer serviceWhen it comes to customer service, MCI scored the best, followed by Sprint. Verizon had the lowest score. Customers complained that basic areas, such as getting responses to questions, take way too long. The carriers simply aren't as organized as they need to be, customers add.MCI's high ratings reflect growing efforts to make up in customer service what the company has lost in reputation because of its financial scandal. Repeatedly, IT executives said they have seen a significant improvement in MCI's response time, trouble-shooting and proactive notification of problems.Of course, the customer experience depends heavily on the quality of the account team. Companies with experienced, stable account representatives who understand their business are happy with their provider. The survey found that there are pockets of good and bad service. One of the carrier's biggest challenges - if not the biggest challenge - is to provide consistent, solid services.Optimization gapWhere all the carriers fall short is helping customers with network optimization. Rarely do the carrier sales reps call customers to let them know when a circuit is reaching capacity, for example. "You'd think they'd want to tell me when to order a new circuit - it's more business for them," says John Wade, CIO of St. Luke's Healthcare in Kansas City, Mo. "But they don't. And it makes no sense."Even though costs aren't dropping the way they used to, participants rated carrier costs relatively favorably. Sprint received the best ratings for cost with a 4.1. Verizon scored the lowest, with a 3.0. That said, 41.7% of respondents cited "reduced costs" on their vendor\/carrier wish list. Basically, they're saying that costs are generally fair to good, but they could be better.The same holds true for SLAs. The general view is that they're needed to keep the carriers on their toes, but companies must take the time and spend the money to negotiate more stringent SLAs than the standard ones that carriers offer. Generally, the SLAs cover uptime, mean-time-to-repair or respond, latency and packet loss. However, some companies negotiate very tough financial penalties - tied to business loss - to guarantee top performance and attention of top executives.Sprint and Qwest score the highest in this area, with a 3.5 and a 4.0, respectively. The lowest ratings are for AT&T, 3.13, and Verizon, 3.0.The bottom line is that companies are looking for more innovation, better management tools and improved customer service from carriers. And they're willing to give second-tier carriers a shot.A total of 46 IT execs were surveyed and they each rated multiple carriers.