Please describe Fannie Mae's network infrastructure.

We have five regional offices. We have our headquarters office here in Washington, D.C., where we have five buildings. We also have two buildings in Herndon, Va. So a large part of our network is managing the communications between our 12 locations. We have a frame-relay network. We have diverse routes between every building.

We have two major external users of our network. We have a lot of [dedicated] communication links with Wall Street trading firms and with the [Federal Reserve Bank.] That's for the portfolio part of our business.

At the front end, we have our MorNet Plus Network. This is the network that we use to communicate with our lenders and brokers that provide us with the [mortgage] loans. A number of the large lenders have dedicated lines. We also have developed a Web-based front end so lenders also use an ISP and come in through that. That's where most of our volume is in terms of network traffic.

On the back end with the portfolio [business], the transactions are fewer but the dollar values are a lot larger. The focus is on reliability and security, but not necessarily high volumes. On the front end, the dynamics have changed and we're looking at much higher volumes. We're looking at the ability to scale rapidly. We're looking at high availability.

What are the key applications running on the network?

The key applications on the front end are the Desktop Originator and Desktop Underwriter. These are utilities that companies can use to originate or underwrite loans. They can choose to sell these loans to us or keep the loans for themselves.

These applications were developed internally. With the latest release, we are now offering them as Internet applications. Although we still allow people to come in through the back end and bypass the [Internet] front end.

What volume of transactions are you handling with these applications?

We are getting about 140,000 submissions a day. The earlier part of the year during the [first refinancing surge], we were hitting about 100,000 submissions a day. So this second [refinancing] wave that occurred with the last rate cut has actually generated more volume than the earlier one did. These are our peak volumes of all time. Our volume over the last year has tripled. It's put a tremendous amount of pressure on the [network] infrastructure and has really stressed our scaling capabilities. We had not planned on scaling up this fast.

With the volume of transactions tripling, what did you need to add to your network?

We added a significant number of servers. We also upgraded our back-up data center to where it could handle production. We moved to having parallel production environments so we could do maintenance on one and so we would have redundancy. One [data center] is in this building and the other is in Herndon, Va. We operate this environment 24-7. It's totally redundant. Our objective is to have 120% of capacity at each site. With the volumes increasing the way they are, we tend to stretch that. The basic platform is Sun on the back end, with Windows NT on the front end.

What upgrades do you have planned for the network infrastructure in the year ahead?

The biggest problem we have with the network infrastructure, frankly, is reliability. We are continually looking for ways to add redundancy into our [external] network while trying to maintain costs. In terms of our internal network, the issue is capacity. We have started migrating toward voice over IP. We're installing that first in our regional offices, as they have a number of phone systems that are old and need to be replaced. We [also] have started to pilot desktop video. Both of those [applications] require a lot more bandwidth to the desktop than we currently have. So we're in the process of upgrading our internal capacity.

Fannie Mae is one of the few companies in the U.S. doing well this year. How are you taking advantage of that in terms of maintaining your IT budget and making new investments?

Fannie Mae has always been a very prudent company in terms of investing in technology. We don't invest in technology for technology's sake. We're not interested in being on the bleeding edge. We are very much driven by the business. This year, our largest technology challenge has been facing the amount of refinancing activity and scaling up to provide [that level of] performance. That has been a huge stretch for us and for our industry. In meeting that challenge, we have made a number of technology improvements. For example, we've upgraded our data centers. We've improved our ability to migrate. The other area where we moved ahead in terms of technology was voice over IP. That [investment] was made for business reasons not because we wanted to be first with voice over IP. Nobody's come and thrown a lot more money at me.

But, obviously, you had to increase your budget just to build up your server capacity. What kind of increases have you needed?

We probably [increased our IT budget] 8% to handle traffic that's tripled since last year. We are not going to be at these volumes for the rest of our lives. So what you don't want to do is build in the bulletproof capability to handle three times the volume and next year you're back to a third. One of the things we were very careful of is not to overbuild. A lot of what we did was accelerate purchases. I don't have any problem buying additional network and server capacity if I know it's the same equipment that is going to be used next year or the year after by other applications as they increase demand. Our basic demand for computing resources - independent of the refinancing surge - is growing 25% a year.

Your two main applications - Desktop Originator and Desktop Underwriter - were recently redesigned to take advantage of an Internet front end. In what other ways is Fannie Mae migrating toward Internet technologies?

Those applications started out under a value-added network that we constructed and charged people access [for]. Then they migrated onto the Web, and they've been re-engineered significantly. Those applications have been Internet-enabled for about a year and a half.

Across the board with our applications, we're finding that we need to provide customers with more access to them. Most of our applications were built on a two-tier, client/server basis. And they were really built for internal customers. The ways we interacted with external customers were either through batch files or very simple transaction-based environments. They gave us data, and we gave them printed reports back.

[Now] customers are demanding more and more information. They want to look at the data we have and use our tools to analyze it. That's creating a closer partnership with them, so there's tremendous value on both sides to be able to do that. The issue is that we have to take these inherently two-tier, fat-client applications and turn them inside out. What that means, of course, is re-engineering them so they can be available over the Web.

The problem is that the complexity of that infrastructure just explodes. Whereas before we'd have a fat client talking to a database server over a simple network, now we have Web servers and proxy servers and application servers and database servers. Most of our applications are not simply fill in the form or take an order. Most of them involve threading all the way through from the front end - like the Desktop Originator/Desktop Underwriter - back into some large database that has a lot of applications connected to it on the back end. The application architecture has become significantly more complex and also has become significantly more dependent upon the network infrastructure.

What lessons can corporate network executives learn from your experiences with Internet applications?

Lesson No. 1 is you can't overemphasize the amount of explaining and education you have to do to the business about the Internet technologies. There are two totally conflicting messages about the Internet. On the one hand, this stuff is very simple. It's easy to use. On the other hand, if you really want to build something that can scale and perform and be running 24-7, the cost of building and supporting that is significantly more than that old two-tier, client/server [application]. You make a big mistake if you don't get that understanding up-front.

The second lesson is that the issues of reliability and security get taken to a new level [with Internet applications]. When customers start complaining or when viruses get introduced or when the threat of bad people into your network starts to become real, the stakes for the business go up astronomically. It's a lot worse than having a few people complain because e-mail is down.

What do you see as the most promising network technologies on the horizon?

I'm very excited about voice over IP. We think we can achieve cost savings just on the elimination of the physical moves, adds and changes in moving people around to various buildings. The elimination of long-distance telecommunications charges - that's all going to be gravy. [But] we're not going to see that benefit until we get it into the Washington office.

But in terms of the network, my No. 1 priority is increased reliability. I would trade off more reliability for new features. The telecom providers are having a hard time with quality. They're having a hard time staying up. They're having a hard time telling us when they're down. And I don't really see an end to that. We see the most problems with the dedicated lines. Service agreements are not all that useful in remedying economic issues. If you have irate customers and the customers can't get through, economic remedies with your long-distance providers don't solve that problem for you. The only way we can remedy that is escalation within those organizations. The other way we're trying to remedy that is with redundant networks and using multiple providers.

How is your department organized? How many people do you have?

The larger technology organization is Enterprise Systems and Operations (ESO). We're divided into two parts. We have a number of organizational units that are closely aligned with the business. Those groups develop the applications that support the business. And they also do the support of those applications. My organization is responsible for the infrastructure that all those applications sit on. I support all the office automation, and included within that is the servers, operating systems, desktops, networks, phones, pagers, etc. That's Enterprise Systems Management. I report to the CTO. I have about 600 employees. In all of ESO, there are about 1,400 people.

What are you doing to beef up network security at Fannie Mae?

A. We're implementing PKI for a lot of our communications. We want to use it for basic e-mail transmissions, internal [and] external. We have a lot of attorney/client communications, and they have a huge need for it. We also have a need for people doing business with us over the Internet to [verify that they] are who they say they are. And we have the need for opening these applications and having the lender applications authenticating transactions. So we see a lot of different uses for PKI. The challenge with PKI is getting people to use it. It works. But it's an extra step that either developers have to take or users have to take, and a lot of times they're just unwilling to take it.

The other area of security that we're looking at is isolating our networks more. If you look at the Fannie Mae user community, there are three separate groups. You have one group that is office automation users. And you have another group that is the developer community. Then you have our core production systems. What we need to do is move to more of a separation between those environments. For example, the office automation users need Internet access. There's also increased likelihood that that's going to introduce viruses. In terms of my development environment, they need the ability to bring in new tools and try new releases. But in my production environment, I want to restrict change. One of the things we're looking at is how can we begin to isolate those three environments to lock one down very tight, leave one more user-controlled for the developers and have one that provides much more open access. We're starting out [trying] to separate them logically because obviously that's the cheapest thing to do. How far we can go into three separate pipes is unclear. We are piloting a separate development environment to see exactly how that is going to work and how much it costs.

The PKI project: Does that have a timetable?

We did a pilot this year, and we introduced it to 1,000 internal users primarily for e-mail. The technology works, but the issue has been getting people to use it. Next year we will do a pilot with some of the applications in development. There's a shared security component we're rolling out that applications can develop to and begin to take advantage of it.

How was your network infrastructure impacted by the Sept. 11 terrorist attacks?

When the third World Trade Center building went down, that cut off all our communications with New York. So we scrambled for several days to come up with alternative communications. At no point did we ever stop doing business with the Fed or with Wall Street, but there was a lot of scrambling that had to be done to achieve that. The other possible impact is that we saw a lot of increase in virus activity right after that.

How has your disaster-recovery planning changed since Sept. 11?

We are looking at many more scenarios. We're looking at each one of our individual systems and finding out what the level of reliability, availability and contingency are. Each year we do a complete disaster recovery exercise where we go through a drill on a weekend that the data center here is down and everything has to be brought up and working out in Herndon. But the expectation is that it's a short-term problem and that you're not going to operate for a long period of time at that other site. So what you're really testing is your ability to come up out there and function out there. But what if you had to operate for six months that way? That's where we're looking at the ability to have dual production centers. I have capacity at both sites. I could shut one down forever and still run. Or I have the communications capability so the people can be anywhere and they can still get to the applications and the applications can get to the lenders. And I've got enough diversity in my network so the physical location doesn't matter. Before we were looking at more fixed disaster recovery scenarios.

What was your first PC?

My first computer was a TRS 80 from Tandy back in the early 1980s.

When was your first experience on the Internet?

I was into bulletin boards before the Internet was used. My first Internet experience was with CompuServe in the early '90s.

What kind of computer network do you have in your home?

I don't have one at home. I have a stand-alone Pentium-class PC, and I use AOL for Internet access.

Related Links

Apply for your free subscription to Network World. Click here. Or get Network World delivered in PDF each week.

Request a reprint or permission to use this article.