Are DNS problems dragging down the performance of your network? That’s what happened to the state of Maine, which was suffering from constant outages caused by a flaky DNS system. Here’s how Maine’s IT officials solved the problem.
The State of Maine’s network supports 15,000 users. It’s an all-IP network that is 100% routed. It supports all state agencies as well as many county agencies and town offices.
Maine’s network supports many critical applications including e-mail and Web services that need to be available to citizens 24/7.
"We are delivering DMV services and public safety services that are mission-critical," says John T. Scott, enterprise DNS and DHCP administrator for the Office of Information Technology for the State of Maine. "The Federal Emergency Management Agency and the Maine Emergency Management Agency rely on our network in disaster-type scenarios."
Maine has redundant systems and uses two carriers – Verizon and Oxford Networks - to support its WAN. Despite its solid network architecture, the State of Maine was suffering from regular outages.
The network problems plaguing Maine were the result of its DNS systems. Since 2001, Maine had relied upon Nortel’s NetID software running on a Microsoft server for its DNS services. The software, which was purchased in 2001, uses Oracle’s database as a back end.
"NetID never could perform reliably enough to provide service anywhere near the five nines of reliability that we were shooting for," Scott said. "We had a lot of issues related to the Microsoft server. One Windows update could break the application interface. That kind of thing happened regularly."
Instead of achieving the 99.999% network availability that was Maine’s goal, network availability hovered around 80%.
"The problem was large," Scott said. "We spent 15% to 20% of our time troubleshooting DNS issues. We became aware that NetID was failing at an exponential rate. With each revision of the software, the situation got worse."
As a result of the outages, Maine’s DNS data was not consistent. Every time NetID went down, the system would populate an outdated version of the DNS data.
"The whole reason we liked NetID is that it was a central database. But the problem was that when NetID couldn’t communicate with the database, it would provide the last version of the data. So we ended up with different versions of the data depending on what server it was talking to," Scott explained.
After two years of constant DNS glitches, the State of Maine decided to replace NetID with a new DNS system. The state’s IT department preferred a hardened DNS appliance, which wouldn’t require so much time and energy to operate.
The State of Maine established a committee of representatives from the IT departments from all branches of government. The committee considered DNS appliances from Infoblox, BlueCat Networks and Men & Mice.
"We looked at the feature sets. We invited the vendors to do dog and pony shows," Scott said. "Infoblox floated to the top of the list fairly quickly."
A year ago, Maine purchased six DNS appliances from Infoblox and began deploying them. Later this year, the state’s Office of Information Technology hopes to purchase four more appliances.
The Infoblox devices are deployed in a grid, with the grid master and a pair of production devices in Augusta, Maine, where the statewide network has its central hub. A second high availability pair of appliances is at the state’s hot site for disaster recovery, which is 12 miles outside of Augusta. Another Infoblox appliance is located in Portland, Maine.
"We’ve been very happy with all aspects of the Infoblox system," Scott says. "They’ve performed as well or better than expected. We’ve been very pleased with the support we’ve gotten from Infoblox."
Converting from NetID to the Infoblox appliances wasn’t easy because much of the state’s DNS data was corrupted. Once the DNS data was cleaned up, network performance improved dramatically.
"We’re at least at the five nines of availability," Scott says. "The system has not been out of service. Even when we had an issue, the backup nodes kicked in. So from the client perspective, we’ve hit five nines."
Scott says it was important for the State of Maine to fix its DNS problems because its DNS traffic is growing so rapidly.
"We see an average of 20 to 30 DNS requests per second," he says. "That’s well within the load that the Infoblox appliances can handle. That’s comforting for me. If three years from now, we have 30,000 state employees, either the grid could handle it or it could be beefed up to handle the traffic."
With DNS running smoothly, the State of Maine’s network operations shop is turning its attention to rolling out VoIP. The state chose VoIP equipment from Avaya, which integrates well with the Infoblox appliances.
Scott loves the security of the DNS appliances from Infoblox.
"Before, security was a constant source of worry for me," Scott says. "I needed to keep my DNS servers updated with security patches all the time, and any one of those patches would trash the application...I wanted a secure, hardened appliance with built-in encryption. Infoblox met all of those requirements."
Ultimately, the State of Maine wants to buy 10 Infoblox appliances, for a total investment of $200,000.
"The biggest return on that investment is in simple manpower savings," Scott said. "We’re no longer spending resources troubleshooting and maintaining our old DNS system. The Infoblox appliances really do run themselves."
Scott said the State of Maine also is getting better productivity because its network is not going down all the time.
"We had a lot of lost revenue because our systems were going down," Scott said. "We constantly had offices or whole regions that couldn’t connect to the network and that would affect hundreds or thousands of employees. We’re getting a huge return on investment in pure uptime."