Most network people understand the terrible affect delay can have on TCP sessions. TCP must pause data transfers while waiting for ACKs. The more network delay there is, the longer this pause will be. However, many people, including most other IT professionals, do not understand this problem. They assume everything is connected at 1Gbps. Plus, TCP window sizes - how much data a host will send/receive before waiting/sending an ACK - can also significantly affect the transfer rate.
Sadly, most TCP sessions never even get close to 1Gbps, particularly over a WAN. I had a unique situation to prove this to user a while back. We have an OC-3 (155Mbps) between our two main offices on opposite US coasts. The round-trip delay is 75ms.
This user opened a ticket complaining of slow file transfers between the two offices. He was only getting 196 KBps transfer rate (1.568 Mbps). He was sure there was a problem with the network since he was sure he should be able to transfer faster. So, I asked for source and destination IP addresses and did a test from my own PC while doing a packet capture.
First, I downloaded a file to my PC with Ethereal running from the remote server. My PC (Windows XP) started the TCP session by advertising a large TCP window size (64,512), but the server returns a small window of 9,280:
4176 > http [SYN] Seq=0 Ack=0 Win=64512
Len=0 MSS=1160
http > 4176 [SYN, ACK] Seq=0 Ack=1 Win=9280 Len=0 MSS=1460
4176 > http [ACK] Seq=1 Ack=1 Win=64512 Len=0
Thus, the server will only send 9,280 bytes before it waits for an ACK from my PC.
Now, a few packets later when the HTTP 200 reply comes from the server the TCP windows grows to 16,384, but then it never got any larger:
Protocol Info
HTTP HTTP/1.1 200 OK (application/zip)
Source port: http (80)
Destination port: 4176 (4176)
Sequence number: 1 (relative sequence number)
Next sequence number: 1161 (relative sequence number)
Acknowledgement number: 428 (relative ack number)
Header length: 20 bytes
Flags: 0x0010 (ACK)
Window size: 16384
Checksum: 0x7183 [correct]
Hypertext Transfer Protocol
Media Type: application/zip (762 bytes)
After a few seconds when TCP stabilizes it creates a situation where the server sends 14 packets then waits for the ACK from my PC. Since the round-trip-time is about 75ms between the two offices, the data transfer pauses while this ACK is in flight. Once the ACK was received, the data transfer starts again. I could see this again and again in the trace.
So, let's do some math. The server is sending packets of 1,214 bytes each. It takes about 85 ms total to receive the 14 packets and send an ACK. So:
14 packets * 1,214 bytes = 16,996 bytes (there's a full TCP window)
So, in 85ms, the server sends 16,996 bytes. Now do a proportion to find out how much is sent in 1 second (1000 ms):
16,996 X
--------- = ---------
85 1,000
85X = 1,699,6000
X = 199,952.94
now convert to kilobytes:
199,952.94/1,024 = 195.26 KBps
Look familiar? That's the exact value my user was reporting as a problem. The network, and TCP, is working perfectly.
If you are able to get both ends to use the maximum TCP window size (65,536) with an 85 ms RTT, your theoretical maximum transfer rate is:
65,536 X
--------- = ---------
85 1,000
85X = 65,536,000
X = 771,011.76
now convert to kilobytes:
771,011.76/1,024 = 752.94 KBps
As a network guy I convert all things to bits, so the theoretical maximum rate over the WAN is:
752.94 KBps * 8 = 6023.52 Kbps = 6Mbps
When you work through the packet trace this turns out to be a very simple math problem. It can be a powerful way to show users that the network is fine. Unfortunately, you won't be able to increase the speed of light for your users.
More >From the Field blog entries:
Ok, the Economy is Bad, Some Cost Savings Tips
Want to Make Your Network Faster?? Use Google Chrome
Is Comcast's Usage Cap Really That Bad?
Cisco's Arrogance Bugs Me Again
It's One of Those Opinionated Days (Again)
CCIE Wireless
Go to Cisco Subnet for more Cisco news, blogs, discussion forums, security alerts, book giveaways, and more.
that's W/d
You've very clearly set forth the W/d equation (window/delay). Keep in mind that TCP looks at the MSS (i.e. TCP payload). What’s usually difficult to see in a trace is how many packets are sent before waiting for an ACK since TCP does a good job of interleaving. You end up having to look at which ACK goes with which payloads to see how much is unacknowledged.
There’s another interesting one: bandwidth-delay product (Bd), which works from the other side of the question. Instead of answering why things are slow, it seeks to answer what the window size would have to be in order to fill the pipe. So, if you have an OC-3, B = 155*1024*1024 = 162,529,280 bps. If your delay is 75 ms (0.075 sec), the formula is 162,529,280/8*0.075 = Window size of 1,523,712 bytes. Since you can’t enter a window size of more than 65,535, you’ll have to enable windows scaling (RFC1323) to get there.
Also note that when determining the transmission rate, TCP will consider the smallest of three windows: send, receive, and congestion. You can usually change the receive window by changing the OS settings (often referred to as “the” TCP window). The congestion window is a calculated value and is affected by packet drops, so it cannot be changed. And the send window is usually controlled by application settings. Of the three only the receive window can be observed in packet decodes. The other two can only be inferred from the data. I saw an issue where the client had a huge TCP receive window, but the server had an 8k send window; when latency was high, throughput was very low.
Dr TCP is an easy Windows GUI for enabling RFC1323. For Unix, you’ll have to hit the man pages.
alternative to FTP
We had this exact problem on our OC-3 link. Our FTP transfers were horribly slow. We didn't get anywhere near the throughput we expected. We actually found a product called FileCatalyst (http://www.filecatalyst.com) that uses UDP rather than TCP to transfer files. It is commercial but considering how much money we were wasting by not fixing the issue it was a small price to pay. That and it was actually really easy to use.
re: alternative to FTP
FTP + UDP = TFTP. There are several free applications that provide this functionality.
packet lose
thanks for the post, can you post one about packet lost in the network?
Post new comment