BOSTON - Reduced telecom costs, easier network administration and the promise of less expensive, more responsive disaster-recovery operations convinced Fidelity Investments it could wean itself from traditional Centrex services and move to IP PBXs.The financial services giant is one of the larger Centrex users in the country, with more than 11,000 phone lines, but it has begun a large-scale voice-over-IP (VoIP) implementation and now uses approximately 600 Cisco IP phones. This year Fidelity expects to add 700 more IP phones and 3,500 softphones, which were built in-house.Fidelity is but one of many large corporations making the move from Centrex to VoIP. Carriers have lost 1.8 million lines and more than $1 billion in Centrex service revenue since 2000, when the IP PBX market emerged, according to telecom research firm RHK.The plan at Fidelity is to move all voice traffic in its Boston headquarters to the company's Cisco-based Gigabit Ethernet metropolitan-area network (MAN), which links four buildings with gigabit-speed connections. Redundancy built into the Cisco IP-based MAN will provide faster disaster recovery at a lower price than Centrex. The only additional cost would be for the extra 50M to 70M bit\/sec of additional IP voice data on the MAN, said David Morgan, vice president of architecture and planning for Fidelity, who outlined the company's VoIP plans at a recent IP telephony conference.One of the most important facets of Fidelity's move is supporting crucial disaster-recovery operations. "Right now we have fairly expensive and complicated disaster recovery," Morgan said. "We can provide better disaster recovery with VoIP."Centrex lets Fidelity provide disaster-recovery phone service to thousands of high-level employees, such as mutual-fund managers, by rerouting phone numbers from AT&T to on-site PBXs, which provide dial-tone service if Centrex services are interrupted.Ultimately, Morgan sees IP telephony as a way to eliminate costs in terms of moves, adds and changes - occurrences that the company's Centrex carrier would charge for. The costs will be eliminated when Cisco CallManager IP PBXs are tied into the firm's Active Directory and Lightweight Directory Access Protocol directories. The ability of softphones and IP phones to reset themselves when moved also will help ease administration.Despite its perceived cost reductions, selling the ROI on IP telephony is difficult because there is no specific cost advantage to physically replacing Centrex with an IP hardware phone, Morgan says.Proving ROI with hard numbers for the technology can get fuzzy, he says. And implementation missteps can be expensive setbacks, as Fidelity has learned.When Fidelity installed one of its first IP phone pilots to about 400 desks, it invested in new switches that provided electric power over Ethernet to the phones."About eight months later," Morgan said, the company "took everyone off that floor and scattered them around the building and other parts of Boston. And now that [power-over-Ethernet] infrastructure is sitting there, and a new group has moved in using standard Centrex.The lesson there, he said, is "you [can't] put IP phones in one place and upgrade your power in just one place . . . you've got to look at upgrading power supplies everywhere."Centrex also is on the way out at the University of Pennsylvania, where voice and data are being converged with gear used normally for carrier-grade VoIP.UPenn leases approximately 28,000 Centrex phone lines from Verizon, with the contract for the service ending in two years. When Centrex's time is up, the school plans to be converged completely with IP telephony.The core of the school's VoIP network will be two softswitches from BroadSoft, a maker of gear for supporting carrier VoIP services. The Session Initiation Protocol (SIP)-based softswitches perform call-processing and gateway functions, and run on two redundant Sun Solaris boxes. UPenn chose the carrier-grade gear because its voice needs are equal to those of a city served by a regional Bell operating company, says Mike Paladino, associate vice president of networking at the university."One of the strongest drivers for this effort is that we will have a lot more control over the features we have," with the IP softswitches, compared with the Centrex service, he says. Centrex offers a minimal number of features that are hard to add or change, Paladino says. But with a converged, SIP-based network, the IT staff can offer different levels of features and applications to different groups.For instance, students could receive applications that integrate popular programs such as e-mail and instant messaging with SIP-based voice. Meanwhile, more-traditional desktop phone services could be deployed to faculty and staff offices. Requests for new features, such as conferencing or new dial plans, also would be easier to provide and manage with the softswitches, Paladino adds.SIP is an important aspect of UPenn's plans to offer expanded features to different departments. Paladino says he is encouraged with SIP's ability to let different clients from multiple vendors interoperate - such as IP phones from Cisco or Pingtel talking to software clients, such as Microsoft Windows Messenger."We're interested in what SIP has to offer and the direction SIP is taking," Paladino says. "Though it's too early to tell definitively, we're confident this is where the market will go."As UPenn rolls out its IP telephony project, the softswitches will run in parallel with the Centrex system, and all the public switched telephone network (PSTN) connectivity will go through Verizon trunks. Paladino says the goal is to hook the softswitches to the outside world over IP trunks, eliminating most traditional PSTN connections into the university. "I like the idea of being able to buy IP services from different carriers, and then put voice on top of that," Paladino says. He adds that SIP also will be a key technology because the vendors that are preparing for converged IP services - such as\u00a0WorldCom\u00a0and\u00a0Verizon, are implementing SIP infrastructures for carrying voice.Many organizations used Centrex because their telecom staffs and\/or budgets were too small to handle an onsite PBX. But with Centrex, customers sacrificed control over their voice features and onsite management capabilities. These users say IP PBXs let them have more control over their phone networks than Centrex.Delaware State University in Dover installed IP PBX technology from Siemens and shut off its Centrex phone service earlier this year. The move reduced the school's telecom service costs about 50% by cutting its Centrex lines for approximately 2,500 faculty, students and staff, at a cost of about $13 to $15 per month per line.The campus now runs a mix of IP phones in its dorms, and digital phones to its staff and administrative offices using a HiPath 3000 IP PBX and HiCom digital phone extension gear from Siemens. Besides reducing Centrex costs, the in-house managed phone system lets the staff change its service more quickly than previously, according to Charles Fletcher, assistant provost for technology and information systems."The IP phones give us great flexibility," Fletcher says. "For any office that has data service, we can put a phone in there without having to call Verizon. Those kinds of changes can be done in seconds."On the West Coast, the IT staff at El Monte (Calif.) Union High School District near Los Angeles decided it had had enough with Pacific Bell's Centrex service."We were dealing with a lot of problems with getting our carrier to respond in a timely fashion," says Garett McKay, director of technology for the city of El Monte. Similar to Delaware State's situation, moves and changes to the phone system involved calling the carrier, then waiting for it to move phone extensions or add and delete numbers.Last year, when the city went online with a Gigabit Ethernet MAN service from Pacific Bell - paid for by the federal E-Rate program - McKay saw an opportunity to get the entire district on a unified voice and data network. A Mitel IP PBX now processes calls for 500 IP phones at 10 schools across the city. And McKay's staff can make all the moves they want from a single PC."We were a little nervous about taking on our own voice management at first," McKay says, "but so far, it's been pretty smooth."In or out? Users find pros and cons to outsourced telephony and in-house VoIP. \u00a0Technology \u00a0Pros \u00a0Cons IP PBXs \u2022 Converged applications. \u2022 Added costs of infrastructureupgrades (switches, power). \u2022 Simplified management. \u2022 ROI can be hard to prove. Centrex \u2022 Outsourced service can allow for smaller or minimal telecom staff. \u2022 Expensive monthly charges for services, including simple moves, adds and changes. \u2022 Redundancy of hosted carrier- grade phone switches. \u2022 New features are hard to implement.