Expecting Gigabit Ethernet to vanquish your LAN bandwidth problems?
Keep your excitement in check. While carefully implementing Gigabit Ethernet can enhance overall network performance, the technology may not be the panacea you expect because your clients and servers probably aren't powerful enough to take full advantage of its capabilities.
Gigabit Ethernet doesn't have the contention problems of conventional Ethernet because it is usually deployed as a switched technology like 10Base-T and 100Base-T. But other factors conspire to lower throughput.
Just how much lower was what we at West Virginia University's (WVU) Advanced Networking Lab set out to determine. We intentionally side-stepped the issue of Jumbo Frames - use of which speeds Gigabit Ethernet by reducing overhead - because we were curious to see what kind of performance we could achieve with plain vanilla Gigabit Ethernet right out of the box (for more on the Jumbo Frames, see "In defense of Jumbo Frames," NW, Aug. 8, 1998).
At the same time WVU was conducting its performance tests, Brian Norris, a senior hardware engineer then working at Wandel & Goltermann, was independently testing bandwidth utilization and throughput on Gigabit Ethernet and Fast Ethernet links. When the two organizations met at the Gigabit Ethernet Conference in San Jose, we learned of each other's testing and compared notes. Our results were surprisingly similar and not good news for organizations considering making the gigabit leap.
Real-world Fast Ethernet
To give us a baseline to compare our Gigabit Ethernet results against, we began by seeing how much throughput we could achieve with Fast Ethernet. Transferring a noncompressible 36M-byte file using a Novell server, we obtained an average bi-directional throughput of 18M bit/sec when communicating with a Windows 95 client and 22M bit/sec when talking to a Windows NT client. The same test was performed using a NT server. This time, the Windows 95 client saw 16M bit/sec average bidirectional throughput while the NT client yielded an average throughput of 25M bit/sec.
When Norris transferred a 40M-byte file between two Windows 95 machines, the File Transfer Protocol tool and the Wandel & Goltermann DominoFE tester he used both reported throughput of 16M bit/sec.
Remember, all this was done on a 100M bit/sec full-duplex link dedicated to this test.
Curious to discover why network performance was so feeble, Norris used Domino's Examine function to inspect the traffic in detail. It was obvious that there was a long delay time between TCP transfers and TCP acknowledgments. IPX delays were just as sluggish. The delays caused a significant drop in performance for both protocols. Norris saved a capture file containing a large chunk of the transfer and then moved on to the Gigabit Ethernet test.
Given the lackluster performance observed in the Fast Ethernet test, neither our team nor Norris expected a huge performance increase from Gigabit Ethernet. We were right. Norris' test revealed a disappointing 20M bit/sec throughput - a mere 20M bit/sec out of a possible 1,000M bit/sec!
Once again, Norris ran detailed analysis on the captured gigabit traffic using Domino Examine. Although the acknowledgement delays were shorter than with Fast Ethernet, they were still apparent.
We fared only slightly better, with the average bidirectional performance with the Novell server using NT and Windows 95 clients hovering around 25M bit/sec. The NT server's performance with the Windows 95 client was 21M bit/sec.
The NT server and NT client combination performed somewhat better at 29M bit/sec.
In short, network bandwidth utilization was poor on Fast Ethernet, never exceeding 25% of network bandwidth, and was downright lousy on Gigabit Ethernet, never exceeding 3% of available bandwidth. Utilization figures such as this should give pause to anyone expecting a dramatic boost from an upgrade to Gigabit Ethernet.
Network performance is affected by a host of PC factors, including bus speeds, interrupts, hard disk read/ write speed and protocol stack processing. While most of these are not significant for 10Base-T, they begin to become factors with Fast Ethernet and really become issues with Gigabit Ethernet. Even though we were using reasonably fast PCs, CPU and memory usage were frequently maxed out during our tests. It appears that most of the system's resources were devoted to processing the packets and reassembling the file.
But there are a host of other potential choke points besides the CPU and memory. Network interface card (NIC) hardware and drivers, for example, are often a weak link in the chain. A poorly performing NIC can easily become a network bottleneck, though even the fastest, most efficient NIC does not guarantee top-notch network performance.
Often the problem is the protocol stack. When a machine needs to send a file across an IP network, the file is broken up into small pieces by TCP.
A TCP checksum is calculated for the fragment, and then it is passed on to the IP layer. IP encapsulates the fragment, calculates another checksum, and passes the newly formed packet down the line.
On the receiving end, the IP layer recalculates the checksum to ensure the integrity of the packet, strips off the IP encapsulation and passes the fragment to the TCP layer. The TCP layer recalculates the TCP checksum to ensure integrity of the fragment and awaits the arrival of the rest of the fragments so the frame can be reassembled.
It is easy to see that a significant number of calculations are involved in packet processing.
In many workstations and servers, checksum calculations and packet processing are largely conducted in software. Administrators first noticed the effect of software packet processing when Fast Ethernet arrived on the scene. For the first time, the pipe between the machines was large enough to unmask a stack-processing problem, where before the network was thought to be the only cause for poor application performance.
The reality is that software stack processing is inefficient. As such, it becomes a constant delay in network performance, regardless of how much you increase the bandwidth between machines.
Need further proof that processing packets in software is inefficient? Simply look inside any high-performance network switch. You will notice that packet processing is conducted in specialized hardware called Application Specific Integrated Circuits (ASIC). Processing packets in on-board silicon is always faster than processing packets with software.
Now look inside your Windows workstation or server. Except for ASICs that are just now beginning to appear on select NICs, you'll find that packet processing is done entirely in software. That's why switches can process packets at Gigabit Ethernet wire speeds while PCs can not.
Because many operating systems force protocol stacks to be run completely in software, the operating system plays a significant role in network performance. Without improving protocol stack performance, any operating system will experience mediocre network performance regardless of how much network bandwidth is available.
For example, when Windows NT Workstation is communicating across the net, the operating system is involved in nearly every network-related action. As a result, Windows NT Workstation sets no speed records when it comes to networking.
Other operating systems, such as Mac OS and Windows 9X, are no better. Because much of a Unix workstation's packet processing is performed in hardware, Unix is often the preferred operating system for many network devices. Unfortunately, the operating system is not the only limitation to network performance in today's PC. Issues such as CPU utilization and bus speed come into play, as well.
Let's do the math. If performance equaled bandwidth, PCs would receive packets from Fast Ethernet at 100M bit/sec. One MHz equals roughly 1M bit/sec. In the latest generation of PCs, the system bus runs at 100 MHz, but older PCs have a 33-MHz or 66-MHz bus. A 33-MHz bus simply will not cut it even for Fast Ethernet speeds because the 100M bit/sec network is feeding into a bus operating at approximately 33M bit/sec.
Now let's increase the throughput demands by placing the PC on a Gigabit Ethernet connection. Gigabit Networks can deliver packets at 1,000M bit/sec, far more than the average PC bus can accommodate. In fact, with Gigabit Ethernet we can easily overwhelm the bus and CPU with network traffic. Not a pretty picture.
Improvements on the way
The upshot, then, is that some network managers may be busily upgrading Fast Ethernet to Gigabit Ethernet when they are not even close to achieving the potential of the Fast Ethernet they currently use.
This does not imply that Gigabit Ethernet is a poor technology. Rather, what we have is a situation in which the PC client simply needs to catch up with the network. By migrating to faster network technologies such as Gigabit Ethernet, we have succeeded in moving the bottleneck from the network to the PC.
Given this, how can we gain better Gigabit Ethernet performance? We can start with improved NICs. Newer NICs incorporate dedicated processors and Media Access Controllers (MAC) that are IP-aware, meaning they can process IP packets and perform IP checksum calculations using hardware. In addition, large First In First Out (FIFO) NIC memory buffers will increase performance by providing a buffer big enough to store a number of complete packets. All this boils down to reduced transaction time (often referred to as turnaround time) which decreases the constant delay in the network performance equation.
The next step toward better performance is to convince operating system vendors to let go of packet processing. This may be a tough sell, but it is essential to get the operating system out of the packet processing business if we are to realize optimum network performance.
Finally, bus speeds inside PCs need to ratchet upwards. Even on systems with a 100-MHz PCI bus, the bus interface between the NIC and the rest of the system is most likely 33MHz and only 32 bits wide. Faster, wider system and NIC buses are needed to take full advantage of Gigabit Ethernet performance.
Happily, some high-end servers are appearing with 64 bit/33MHz and 64 bit/66MHz buses. The wider bandwidth and increased speed should enable these new machines to do a better job of keeping up with Fast Ethernet, though they don't approach the level needed for true gigabit performance.
With so much of the packet processing still tied to the operating system, these fast buses will be underutilized in PCs and servers. Operations such as interrupt processing frequently divert the CPU's attention and, as long as the CPU and the bus must become directly involved in handling individual packets, that will take a toll on network performance. Distracted by interrupts and other data processing requirements, a PC with the fastest modern bus cannot possibly process packets at wire speeds.
Until all of these improvements come together, the promise of end-to-end Gigabit Ethernet will remain much greater than the reality.
The author expresses his appreciation to Brian Norris, senior hardware engineer at TTC Corp., for his assistance in the preparation of this article.
Reaction: Here's what some Fusion users are saying about this article: What do you think? Add your comments to the thread
Gigabit Ethernet Net Resources
Primers and tutorials.
When 1-Gig just won't do
Network World Fusion Focus on High Speed LANs, 05/17/99.
How far can Gigabit Ethernet go?
Network World Fusion Focus on High Speed LANs, 05/10/99.
Gigabit copper pedal to metal
Vendors ready Gigabit Ethernet over copper cards. Network World, 05/10/99.