Gigabit Ethernet hits second gear
Faster chips, wider buses, Windows 2000 contribute to increased throughput, but hard drives stand as the bottleneck to getting Gigabit to the desktop.
 
|
|||
 | |||
|
By JEFFREY N. FRITZ
Well, nine months later there is good news. Second-generation Gigabit Ethernet products certainly do promise enhanced performance all the way to the desktop.
Server CPUs now clock speeds up to 550 MHz as the norm and new 64-bit/66-MHz bus architectures relieve the bus I/O bottleneck. Second-generation Gigabit Ethernet NICs have onboard intelligence that allow them to off-load functions like checksum calculations from the CPU and operating system for a marked performance increase. And, finally, Microsoft claims to have made significant strides in improving operating system network support in Windows 2000.
What do all these second-generation improvements buy us? To find out, West Virginia University's Advanced Network Applications Lab repeated our Gigabit Ethernet performance testing against these new and improved conditions.
In our last test we did not use Jumbo Frames because they were not typically used in Gigabit Ethernet networks at the time. However, Jumbo Frames have become a hot topic among Gigabit Ethernet users. Since the bandwidth of Gigabit Ethernet is so much higher than Fast Ethernet, a larger frame size is more efficient because it lowers the overhead.
The standard Ethernet frame is 1615 bytes per frame. When we increased this to 9014 bytes per frame (See Table 3), our throughput jumped to 806 Mbps with memory transfers. At nearly 81 percent utilization, we were finally seeing decent Gigabit Ethernet bandwidth utilization. The problem is that all Gigabit Ethernet vendors do not yet support Jumbo Frames. For example, neither of our switches supported Jumbo frames.
Bringing the hard drive back into the picture destroyed any performance gains we achieved with Jumbo Frames. Using file transfers, even with Jumbo Frames, our throughput dropped back to around 137 Mbps - virtually the same throughput we measured without Jumbo Frames. This is yet another indication that the hard drive has become a major bottleneck to network performance.
In our previous round of testing, the best Gigabit Ethernet performance we saw was 29 Mbps measured in a file transfer between a Windows NT 4.0 server and Windows 95 client. This amounted to a meager 3% bandwidth utilization over Gigabit Ethernet.
Using second-generation Gigabit Ethernet products and Windows 2000, the best real-world throughput was 158 Mbps, which sets bandwidth utilization at 16 percent. On average, we measured performance ranging from 137 to 145 Mbps. (See Table 1). These results bode well for connecting server and high performance workstations directly to Gigabit Ethernet - something we couldn't recommend previously.
This prompts us to ask two questions: First, what accounted for the performance improvement? Second, why didn't we see even better performance and higher bandwidth utilization?
Our network topology did not change between rounds of testing. We used the same fiber connected Allied Telesyn 9108 Gigabit Ethernet switch that we did last time. Each PC had a SysKonnect Gigabit Ethernet NIC directly connected to the switch.
What did change between test runs was that we used Dell Precision 610 workstations with 500-MHz Xeon Pentium III CPUs. These are much faster than the Dell Optiplex 200 MHz Pentium II PCs used in our last test. We installed a SysKonnect SK-9844 SK-NET GE-SX dual link second-generation Gigabit Ethernet NIC in each PC. And, finally, we used production versions of Windows 2000 Professional on the client machines and Windows 2000 Server on the servers.
The most significant improvement comes from using PCs with more processing power. We proved this by installing Windows NT 4 SP 5 and the SysKonnect NICs on the Optiplex 200-MHz PCs. In our previous Gigabit Ethernet testing, it was typical to see throughput rates hanging around 21 to 29 Mbps. We re-ran these tests and achieved results consistently around 23 Mbps.
We then installed Windows 2000 Professional and Server on the Optiplex machines. With the new operating system installed, the average throughput inched up to the 28 to 31 Mbps range.
In our first round of tests with the Dell Optiplex 200 MHz PCs, the average performance for Fast Ethernet running Windows NT 4 was about 19 Mbps. Switching to Gigabit Ethernet in that instance increased performance to about 23 Mbps. Under Windows 2000 on these same machines, we measured an average throughput of 29 Mbps over Fast Ethernet. Switching to Gigabit Ethernet, the average throughput did not change. It was virtually the same as Fast Ethernet!
Obviously, the slower machines were bumping up against internal PC constraints due to hard drive, bus, and processor speed limitations. These PCs simply couldn't process the packets as fast as the network could deliver them even with Fast Ethernet.
If you are running older servers or workstations, don't waste your money plugging Gigabit Ethernet directly into them. Our test results indicate that unless you have modern PCs running as servers or workstations, you can forget about gaining any significant network performance advantages with Gigabit Ethernet to the desktop.
We then installed NT 4.0 SP 5 on the new Dell Precision 610 PCs with the SysKonnect Gigabit Ethernet NICs and reran our performance tests. Under these conditions, we reached throughputs in the 100 to 107 Mbps range. This was a significant improvement over our previous testing. However, the utilization on a 1,000 Mbps network was still low at around 10 percent.
The good news is that with newer PCs and second-generation NICs we can now overrun Fast Ethernet with a single client/server stream. This justifies Gigabit to the desktop, particularly for server connections, but also for high-performance workstation needs such as multimedia and CAD/CAM.
Next, we tested what changes Windows 2000 would make on the more powerful workstations. We were now seeing throughputs ranging from 138 to 145 Mbps. That's impressive performance and a definite improvement over Windows NT on the same PCs, but we were still troubled by the poor network utilization. Even at 145 Mbps, we were only utilizing less than 15 percent of the available Gigabit Ethernet bandwidth.
To find out just which internal components were slowing down Gigabit Ethernet utilization in our tests, we used direct memory transfers (See Table 2). While these transfers don't reflect a real world situation, they did help us pinpoint the bottleneck. Memory testing removes the hard drives completely from the picture while leaving everything else (CPU, bus, NIC, etc.) in place. We felt that it was very likely that the hard drive, more than any other PC component, was restricting our overall throughput.
Our suspicions were correct. Using memory to memory tests, our throughput increased to 360 Mbps. This is nearly a 250 percent increase achieved by bypassing the hard drive. Mind you, these PCs did not come with inexpensive ISA hard drives. They used Quantum 9 GB Ultra2/Wide SCSI (10,000 rpm) hard drives. Even then, these drives simply could not keep up with the write requests coming in from the network.
This test indicates that a significant part of the transfer limitations over Gigabit Ethernet is due to the time it takes today's hard drives to read and write information. While users can improve hard drive performance by adding additional or faster disks, drive vendors can also contribute to network performance by designing hard drives that contain larger on-disk cache that are optimized for random I/O.
In our first article, we were critical of the way Windows machines handled network processing. Every packet had to be handled by the operating system, meaning that the CPU, bus and memory of the PC had to be involved in packet processing. This results in hampered network performance.
Windows 2000 is Microsoft's first attempt to move packet processing where it belongs, back to the NIC. In addition, there have been some major changes made in how Windows 2000 handles the IP stack that has resulted in enhanced performance.
In our previous installment, we deliberately did not tune IP parameters for our testing. The idea was not to tweak or optimize the way the operating system handled packets, but instead use the default IP parameters. We took the same approach in testing second-generation Gigabit Ethernet. However, this time there were a couple of differences.
For one thing, Windows 2000's TCP/IP implementation is largely self-tuning. For example, Windows 2000 implements the Network Driver Interface Specification (NDIS) 5.0 standard. NDIS 5.0 brings a number of features to the table, such as offloading tasks to the NIC by allowing it to perform TCP hardware checksum and IP Security. By default, NDIS 5.0 allows TCP/IP to query the NIC driver to find out what Maximum Transmission Unit (MTU) is supported. This helps the operating system use the largest possible MTU and that in turn increases network efficiency. There is also some manual tuning that you can do, such as large packet segmentation and using jumbo frames that can improve in a Gigabit Ethernet environment (See Jumbo Frames sidebar, page XX).
We used our memory tests to measure the difference between hardware and software checksum. TCP uses a checksum on both the headers and data of each segment. This reduces the chance that network corruption will result in a damaged packet making its way into the network application. Performing checksums is a mathematically intensive process that in previous Windows versions was handled by the CPU and the operating system. Since checksum has to be performed for every individual packet, this can take a heavy toll not only on network performance, but also on CPU usage.
Windows 2000 changes this by allowing the NIC to perform TCP checksum calculations. Obviously, the NIC driver must provide support for this function, but if it does, the performance increase can be significant. In our tests, offloading the checksum calculations to hardware made the throughput jump to 504 Mbps. This represents over a 50 percent improvement in network utilization.
Our testing shows that Gigabit Ethernet network performance clearly increases dramatically by using second-generation NICs, operating systems designed for high performance networks and faster, more powerful PCs. With this combination in hand, and some sensible TCP tuning, you can get closer to the kind of network performance that Gigabit Ethernet promises.
From a network point of view, not much changed from our last Gigabit Ethernet test. We used the same Allied Telesyn AT 9108 Gigabit Switch. We added a Cisco 4912G Gigabit Ethernet switch to the network because it allowed us to swap Gigabit Ethernet switches to ensure that the network switch itself did not provide any performance issues. Both switches performed equally well.
We replaced the older Dell Optiplex 200-MHz PCs with Dell Precision 610 workstations. Each Precision 610 had a Quantum 9 GB Ultra2/Wide SCSI (10,000 rpm) hard drive, a 500-MHz Pentium Xeon III processor and 128M bytes of memory. We installed Windows 2000 Professional on two of the Dell Precision 610s and used them as clients. We installed Windows 2000 Server on the third Precision 610.
We first performed real-world testing by transferring a noncompressible 378,741,855 byte file between the various machines. We performed server-to-client transfer and client-to-client transfers. Each time we conducted a transfer we renamed the source file so that Windows 2000 would not have the ability to cache the file in memory. We then measured the transfer time on several runs to and from each of the machines and averaged the results. For the file transfer test there was no attempt to tune Windows 2000.
We then tested the maximum performance we could get out of Windows 2000 over Gigabit Ethernet by doing memory to memory testing while tuning Windows 2000. The tuning we applied in our testing came from the Microsoft's "Windows 2000 Performance Tuning Guide".
The tool we used for our memory tests was the Windows NT version of Test TCP or TTCP (called NTTTCP). TTCP is a benchmark that has been used by networking industry to isolate the TCP/IP stack and network drivers. The NT version utilizes some of the advanced asynchronous I/O features of Windows 2000.
The author would like to express his appreciation to the West Virginia University Network Services team consisting of Matthew Glotfelty, Floyd Roberts and John Bird Vilseck. The author also received generous support from Allied Telesyn, Cisco Systems, and SysKonnect.
Jeffrey Fritz serves as the Principal Network Engineer for West Virginia University's Network Services department. He can be reached at jfritz@wvu.edu.
"Real World" file transfer test
Optipex 200 MHz file transfer test
The copper alternative
Gigabit choke points
Gigabit Ethernet research page
Network World, 03/20/00
We restricted our testing to Windows 2000 because to date it is the only operating system available to us that includes improvements specifically geared toward beefing up Gigabit Ethernet performance.
Jumbo frames Where's The Beef?
One thing is certain when it comes to bottlenecks - they tend to move around. For a long time, the network was the bottleneck. However, with newer network topologies such as ATM and Gigabit Ethernet, that doesn't have to be the case anymore. Today's bottlenecks to network performance can be found in the NIC, the computer bus and even the components within a server or workstation, such as the CPU. How we tested second-generation Gigabit Ethernet
Excel spreadsheet showing our numbers.
Excel spreadsheet showing our numbers.
New switch interfaces allow you to connect devices at Gigabit Ethernet speeds using ordinary Cat 5 cable. Network World, 3/20/00.
Our testing last year showed problems. Network World, 7/5/99.
with links to primers, reviews and Network World articles.
