Most network people understand the terrible affect delay can have on TCP sessions. TCP must pause data transfers while waiting for ACKs. The more network delay there is, the longer this pause will be. However, many people, including most other IT professionals, do not understand this problem. They assume everything is connected at 1Gbps. Plus, TCP window sizes - how much data a host will send/receive before waiting/sending an ACK - can also significantly affect the transfer rate.
Sadly, most TCP sessions never even get close to 1Gbps, particularly over a WAN. I had a unique situation to prove this to user a while back. We have an OC-3 (155Mbps) between our two main offices on opposite US coasts. The round-trip delay is 75ms.
This user opened a ticket complaining of slow file transfers between the two offices. He was only getting 196 KBps transfer rate (1.568 Mbps). He was sure there was a problem with the network since he was sure he should be able to transfer faster. So, I asked for source and destination IP addresses and did a test from my own PC while doing a packet capture.
First, I downloaded a file to my PC with Ethereal running from the remote server. My PC (Windows XP) started the TCP session by advertising a large TCP window size (64,512), but the server returns a small window of 9,280:
4176 > http [SYN] Seq=0 Ack=0 Win=64512
Len=0 MSS=1160
http > 4176 [SYN, ACK] Seq=0 Ack=1 Win=9280 Len=0 MSS=1460
4176 > http [ACK] Seq=1 Ack=1 Win=64512 Len=0
Thus, the server will only send 9,280 bytes before it waits for an ACK from my PC.
Now, a few packets later when the HTTP 200 reply comes from the server the TCP windows grows to 16,384, but then it never got any larger:
Protocol Info
HTTP HTTP/1.1 200 OK (application/zip)
Source port: http (80)
Destination port: 4176 (4176)
Sequence number: 1 (relative sequence number)
Next sequence number: 1161 (relative sequence number)
Acknowledgement number: 428 (relative ack number)
Header length: 20 bytes
Flags: 0x0010 (ACK)
Window size: 16384
Checksum: 0x7183 [correct]
Hypertext Transfer Protocol
Media Type: application/zip (762 bytes)
After a few seconds when TCP stabilizes it creates a situation where the server sends 14 packets then waits for the ACK from my PC. Since the round-trip-time is about 75ms between the two offices, the data transfer pauses while this ACK is in flight. Once the ACK was received, the data transfer starts again. I could see this again and again in the trace.
So, let's do some math. The server is sending packets of 1,214 bytes each. It takes about 85 ms total to receive the 14 packets and send an ACK. So:
14 packets * 1,214 bytes = 16,996 bytes (there's a full TCP window)
So, in 85ms, the server sends 16,996 bytes. Now do a proportion to find out how much is sent in 1 second (1000 ms):
16,996 X
--------- = ---------
85 1,000
85X = 1,699,6000
X = 199,952.94
now convert to kilobytes:
199,952.94/1,024 = 195.26 KBps
Look familiar? That's the exact value my user was reporting as a problem. The network, and TCP, is working perfectly.
If you are able to get both ends to use the maximum TCP window size (65,536) with an 85 ms RTT, your theoretical maximum transfer rate is:
65,536 X
--------- = ---------
85 1,000
85X = 65,536,000
X = 771,011.76
now convert to kilobytes:
771,011.76/1,024 = 752.94 KBps
As a network guy I convert all things to bits, so the theoretical maximum rate over the WAN is:
752.94 KBps * 8 = 6023.52 Kbps = 6Mbps
When you work through the packet trace this turns out to be a very simple math problem. It can be a powerful way to show users that the network is fine. Unfortunately, you won't be able to increase the speed of light for your users.
More >From the Field blog entries:
Ok, the Economy is Bad, Some Cost Savings Tips
Want to Make Your Network Faster?? Use Google Chrome
Is Comcast's Usage Cap Really That Bad?
Cisco's Arrogance Bugs Me Again
It's One of Those Opinionated Days (Again)
CCIE Wireless
Go to Cisco Subnet for more Cisco news, blogs, discussion forums, security alerts, book giveaways, and more.
that's W/d
You've very clearly set forth the W/d equation (window/delay). Keep in mind that TCP looks at the MSS (i.e. TCP payload). What’s usually difficult to see in a trace is how many packets are sent before waiting for an ACK since TCP does a good job of interleaving. You end up having to look at which ACK goes with which payloads to see how much is unacknowledged.
There’s another interesting one: bandwidth-delay product (Bd), which works from the other side of the question. Instead of answering why things are slow, it seeks to answer what the window size would have to be in order to fill the pipe. So, if you have an OC-3, B = 155*1024*1024 = 162,529,280 bps. If your delay is 75 ms (0.075 sec), the formula is 162,529,280/8*0.075 = Window size of 1,523,712 bytes. Since you can’t enter a window size of more than 65,535, you’ll have to enable windows scaling (RFC1323) to get there.
Also note that when determining the transmission rate, TCP will consider the smallest of three windows: send, receive, and congestion. You can usually change the receive window by changing the OS settings (often referred to as “the” TCP window). The congestion window is a calculated value and is affected by packet drops, so it cannot be changed. And the send window is usually controlled by application settings. Of the three only the receive window can be observed in packet decodes. The other two can only be inferred from the data. I saw an issue where the client had a huge TCP receive window, but the server had an 8k send window; when latency was high, throughput was very low.
Dr TCP is an easy Windows GUI for enabling RFC1323. For Unix, you’ll have to hit the man pages.
alternative to FTP
We had this exact problem on our OC-3 link. Our FTP transfers were horribly slow. We didn't get anywhere near the throughput we expected. We actually found a product called FileCatalyst (http://www.filecatalyst.com) that uses UDP rather than TCP to transfer files. It is commercial but considering how much money we were wasting by not fixing the issue it was a small price to pay. That and it was actually really easy to use.
re: alternative to FTP
FTP + UDP = TFTP. There are several free applications that provide this functionality.
packet lose
thanks for the post, can you post one about packet lost in the network?
how tot determine packet size
Hi, can you please advise how you confirmed that the server is sending the 1214 byte packets.
"The server is sending packets of 1,214 bytes each".
additional details on your example
I agree with Anon, your packet trace example doesn't provide any supporting info on 1) the 14 packets you see and 2) each packet is 1,214 bytes. I'm assuming that is the MSS you saw under TCP options (noted in a packet capture as LEN)?
You added 10ms to delay to the current RTT of 75ms. That was strange because I thought the RTT includes the ACK delivery. So why was additional 10ms added to the RTT you provided of 75ms?
"14 packets * 1,214 bytes = 16,996 bytes (there's a full TCP window)" - I'm assuming if I look into one of packet traces for RWIN I would see the window size of something like 16,996 bytes? Hence, all I really need to know is the RWIN size and not the # of packets or the MSS. I can know the estimated number of packets if I do the math, WINDOW SIZE/MSS = # PACKETS?
I'm trying to replicate your example and those are the questions I have including finding similar examples on the Internet.
FYI - it would also be super nice if there was a simple GUI APP (that can be installed on a client and server of coarse) that would show what is the RTT, Throughput, and Window Size for the connection between them. Heck maybe also the packet loss, etc. The numbers would be variable, but that would be a useful tool for troubleshooting and understanding the performance conditions. I don't think there is anything simplified like that out there.
Thank you very much for the posting!
cp
Reply to comment
It is good to move as chance bring new things in life, paves the way for advancement, etc. But it is well known to everyone that moving to new location with bulk of goods is not an easy task to move or shift from one place to other place because I have experienced about that because I face the problem of mobile phones There I go to village near to my city faced that problem there.
Post new comment