By design, Internet security solutions slow down and add oversight to the flow of data. This sometimes puts them in the crosshairs of users wanting faster response time, and management wanting higher productivity. Firewalls, for example, were originally used almost exclusively to protect a computer network from a public network over a relatively low-speed connection. The advances in network communications performance and the adaptations of firewall technology into new uses have in some cases caused the firewall to be a performance bottleneck. While some firewalls have companion hardware accelerator solutions (and ultimately firewall policies and routing decisions will become integrated into the silicon), we can expect firewall technology to play catch up with network bandwidth enhancements and user demands for now. Because of this, there are key areas you must review when assessing the performance of your firewalls.
I. Network design
It is critically important to have an accurate network map and to comprehend the underlying design when analyzing a firewall. You need to understand the "speeds & feeds" of the firewall: that is how many network interface cards are in the firewall and what is the rated speed of each interface. It used to be rare to see a firewall that needed to do any more than keep up with a T-1 connection to the Internet. Telcos now offer services such as 10Base-T, T-3, 100Base-T and ATM. We are also seeing an abundance of companies using a firewall in an intranet or extranet configuration, connected to multiple 100base-T local segments.
In addition to understanding "speeds & feeds," it is important to understand the assumptions about how the network is to be used and what traffic patterns are to be expected. Are there peak load times to be aware of? How different are they from the norm? Does traffic tend to remain isolated to departmental workgroups, with occasional trips to a central host, or is the situation reversed? You need to know and understand the underlying design considerations that went into the network configuration.
II. Firewall configuration
The configuration may depend upon where the performance assessment request came from and what resources are available in determining the best place to start. At some point you will need to validate the configuration of the firewall itself.
You should first check the firewall for basic configuration issues:
TCP/IP settings - Are IP addresses, subnet masks, DNS settings, default gateways, etc. accurate?
Route table - Are all route table entries accurate and optimized? If you have any cases of multiple routes for a single network, you should verify that the lowest cost route is the fastest.
System logs - Do you see any entries that indicate a problem with the hardware, software drivers or any running application?
Operating system version - Is it a known good and stable version of the operating system?
After answering these questions you should look at firewall configuration issues:
Firewall software version - Are you using a known good version of the firewall software?
Rules - Do you have a reasonable number of rules for what you are trying to do and for what the capabilities of the firewall are? Sometimes firewall administrators get in the habit of copying an existing rule and making slight changes to it rather than looking for opportunities to consolidate and simplify rules. Do your rules make sense? Do you have rules that conflict with one another, causing the firewall to do extra processing before releasing the packet? It is important to understand how your firewall determines which rule takes priority and how strict rule ordering is.
Logging - How extensive and verbose is your firewall's logging functions? Are you logging every packet and doing reverse lookups on the fly to write host names rather than IP addresses to the logfile? Some firewall administrators like to be able to see what host names and domains are active on the firewall rather than seeing IP addresses that have little meaning. If logging is causing a performance issue, it may be best to turn off reverse lookups. You can run the logfile through an analyzer program offline from another system to get the same information, rather than making the firewall do all that work.
Licensing - Do you have any license restrictions on the firewall that might limit the number of concurrent IP address connections the firewall will allow? It may be necessary to upgrade the licenses on the firewall if users are waiting for a connection to be freed up. While in some cases the IS department may configure a proxy server to get around this problem, you should verify that this is not in violation of the firewall's license agreement.
VPN - Is the firewall performing any VPN functions? If the firewall is performing encryption tasks, you need to be aware that this is causing extra processing locally. You need to confirm what key size you are using and if the firewall is properly sized to do the work. Sometimes the solution is a box for traditional firewall functionality and a box for VPN activity.
Other applications - Many times the firewall is doing other things, such as performing virus checking, content filtering and URL blocking. Some firewalls are even acting as Web servers, which is almost uniformly a bad idea in terms of security and performance.
Although these types of configuration review and redesign activities often solve any performance problem at the firewall, it may be that the actual architecture of the firewall code is a problem. How do the proxies work, how does it allocate memory, how does it take advantage of hardware and operating system capabilities?
III. Firewall performance metrics
Alongside viewing the firewall from a configuration perspective, you should also develop performance metrics and track them over time.
CPU utilization. How busy is the system processor, and which processes are taking the most CPU cycles? Always be aware that an inordinately busy processor may not mean that the CPU itself is the bottleneck but that a process is taking too much time because of an improperly tuned or faulty resource, such as a network interface.
Memory utilization. If the firewall is on an operating system that uses virtual memory, verify that the firewall is not constantly paging (writing memory pages to disk). Paging is very inefficient. If the system uses only real memory, get statistics on used and available memory and make sure there is a buffer.
Disk utilization. Make sure the firewall is not running out of disk space. Sometimes firewall administrators forget to archive logfiles and let them accumulate over time.
Network interface utilization. Reviewing statistics associated with interface adapters and performing network traces with a protocol analyzer can be a very informative troubleshooting technique. Be careful about renting a protocol analyzer for an afternoon and putting too much validity into its results. It is sometimes difficult to know what to make of the data coming from a spot analysis of a production network, because the results are often skewed by the specific activities of all users and servers at that given point in time. In order to establish a trusted baseline, it is necessary to perform these tests in the absence of any users or continually run tests over the course of several weeks. The things to try to look for:
1. General health statistics of each network segment attached to the firewall. Media errors, collisions, broadcasts, maximum throughput.
2. Round-trip latency of traffic going through the firewall. Using the protocol analyzer, trace client-to-host transmissions without going through the firewall as a controlled test. Then, put the client behind the firewall and run the same set of tests. Is throughput similar? Do you notice a much larger Interframe Gap in the packets you captured going through the firewall?
3. Firewall statistics. If your firewall has good LAN statistics, you can check for several things: Maximum packets/bytes sent and received per time interval, packets being queued for firewall processing, packets being queued for transmission, transmission errors, etc. If the firewall has two primary network interfaces that traffic flows on, you should see the maximum packets received/sec number on one interface correlate with the maximum packets sent/sec number on the other interface at an almost instantaneous point in time. An observable delay between peak receiving and sending of packets is an indication that the firewall is adding latency. How much depends upon the granularity of the statistics. Does your firewall lack these statistical capabilities? Put a protocol analyzer on each segment.
The future trend in firewall technology points to applying much of the existing functionality into a hardware-applied solution, to provide higher performance. While nearly all firewalls can support multiple 10base-T interfaces without being a considerable bottleneck, you need to carefully monitor firewall performance and vendor specifications when dealing with T-3, 100Base-T and faster technologies. Careful analysis of the firewall software, operating system and performance metrics can help you to optimize performance. Above all, never do anything to enhance firewall performance at the cost of degrading overall systems security. That is what it's there for.
Radware bolsters firewall load balancer
Network World, 08/16/99
Portable firewall circumvention
Network World, 07/26/99
Network World, 07/19/99
Network World Security Alert will keep you up to date on the latest security holes and patches, with daily updates from key vendors, security organizations and Network World reporters. See the latest dispatches from the security here.