Are DNS problems dragging down the performance of your network? That’s what happened to the state of Maine, which was suffering from constant outages caused by a flaky DNS system. Here’s how Maine’s IT officials solved the problem.

The State of Maine’s network supports 15,000 users. It’s an all-IP network that is 100% routed. It supports all state agencies as well as many county agencies and town offices.

Maine’s network supports many critical applications including e-mail and Web services that need to be available to citizens 24/7.

"We are delivering DMV services and public safety services that are mission-critical," says John T. Scott, enterprise DNS and DHCP administrator for the Office of Information Technology for the State of Maine. "The Federal Emergency Management Agency and the Maine Emergency Management Agency rely on our network in disaster-type scenarios."

Maine has redundant systems and uses two carriers – Verizon and Oxford Networks - to support its WAN. Despite its solid network architecture, the State of Maine was suffering from regular outages.

The network problems plaguing Maine were the result of its DNS systems. Since 2001, Maine had relied upon Nortel’s NetID software running on a Microsoft server for its DNS services. The software, which was purchased in 2001, uses Oracle’s database as a back end.

"NetID never could perform reliably enough to provide service anywhere near the five nines of reliability that we were shooting for," Scott said. "We had a lot of issues related to the Microsoft server. One Windows update could break the application interface. That kind of thing happened regularly."

Instead of achieving the 99.999% network availability that was Maine’s goal, network availability hovered around 80%.

"The problem was large," Scott said. "We spent 15% to 20% of our time troubleshooting DNS issues. We became aware that NetID was failing at an exponential rate. With each revision of the software, the situation got worse."

As a result of the outages, Maine’s DNS data was not consistent. Every time NetID went down, the system would populate an outdated version of the DNS data.

"The whole reason we liked NetID is that it was a central database. But the problem was that when NetID couldn’t communicate with the database, it would provide the last version of the data. So we ended up with different versions of the data depending on what server it was talking to," Scott explained.