Throughput is meaningful, but only when a network is heavily loaded. Another metric, delay, is meaningful for all traffic, all the time, on all networks.
Editor’s note: This is a corrected version of a column originally posted March 24.
Let me be the first to admit test labs have gotten something very basic very wrong: We put too much emphasis on throughput because it’s simple and sexy. What could be cooler than seeing how fast the latest network widget runs?
Plenty, as it turns out. In our recent test of 10G Ethernet switches, there were big differences in throughput. The paradox is that the faster the network, the less important throughput becomes.
Throughput is meaningful, but only when a network is heavily loaded. Another metric, delay, is meaningful for all traffic, all the time, on all networks.
Devices that add high delay slow your network regardless of whether it runs at 1% utilization or at 100%. What’s more, applications may start conking out well before delays creep into the dozens of milliseconds.
Delay with Gigabit or 10G Ethernet interfaces is usually in the dozens to hundreds of microseconds. When I write about results like this, I usually add boilerplate text saying, “applications don’t suffer until delay reaches into the dozens of milliseconds.”
Sorry, but I was wrong. Even delays in the microseconds can degrade TCP performance dramatically on Gigabit and 10G Ethernet networks. The reason has to do with the way TCP works.
Normally, we do throughput tests using connectionless traffic like plain IP or UDP/IP packets. With connectionless traffic, delay doesn’t have any practical impact on throughput (provided delay is constant, which is a whole other issue).
But even constant delay can have a huge impact on connection-oriented protocols like TCP. Considering that at least 80% of Internet traffic uses TCP, this is a rather large oversight in the context of throughput testing (see http://ipmon.sprint.com for recent Internet backbone observations).
In TCP, a transmitter can send only a limited amount of data, called a window, before the receiver must send an acknowledgement. A window may include multiple packets, but if the transmitter doesn’t get acknowledgements for any of the data, it can’t send anything further.
Here’s where delay comes into the picture. Let’s say we’re using a 10-gigabit switch to receive 1,518-byte Ethernet frames. Let’s further assume network utilization is light. At 10% utilization, we would receive 81,274 frames per second, or one frame every 12 microsec.
If the sending TCP window size is 16K bytes, no more than 11 frames’ worth of data can be unacknowledged at any one time. For 11 frames at 12 microsec each, any delay of 132 microsec or more will prevent the transmitter from sending more data until any of the outstanding data is acknowledged.
Of course, the same is true for delays of less than 132 microsec too, but it’s delays greater than this level that can have an adverse impact on throughput.
Two switches in our 10G Ethernet test – from Avaya and Force10 – delayed each 1,518-byte frame by more than 40 microsec. Add up 11 delays like that, and we’re well beyond the point where throughput can be degraded.
It’s not possible to draw generalizations stating exactly how much throughput would suffer in all cases. In most cases, at least one acknowledgment will come back before a whole window has been transmitted, partially mitigating the high delay. TCP also has various mechanisms to optimize the delivery of acknowledgments, such as delayed and selective acknowledgments, and these too may offset the effects of high delay.
Still, as a rule it’s safe to say that when it comes to TCP, high delay trumps high throughput every time.
Force10 recently retested its own switch with new software. Force10 says delay for large frames is now 23 microsec, roughly half the value we measured. Though this is a big improvement, it’s still high enough where it could degrade TCP performance at 10-Gbit/s rates. Besides, it’s important to bear in mind that window sizes change dynamically; the larger the window the greater the impact of even a little delay.
It’s possible to determine the effect of TCP windowing on any network. All you need are three values: frame length, TCP window size and network utilization. What does delay look like on your network?
