Most network people understand the terrible affect delay can have on TCP sessions. TCP must pause data transfers while waiting for ACKs. The more network delay there is, the longer this pause will be. However, many people, including most other IT professionals, do not understand this problem. They assume everything is connected at 1Gbps. Plus, TCP window sizes - how much data a host will send/receive before waiting/sending an ACK - can also significantly affect the transfer rate.
Sadly, most TCP sessions never even get close to 1Gbps, particularly over a WAN. I had a unique situation to prove this to user a while back. We have an OC-3 (155Mbps) between our two main offices on opposite US coasts. The round-trip delay is 75ms.
This user opened a ticket complaining of slow file transfers between the two offices. He was only getting 196 KBps transfer rate (1.568 Mbps). He was sure there was a problem with the network since he was sure he should be able to transfer faster. So, I asked for source and destination IP addresses and did a test from my own PC while doing a packet capture.
First, I downloaded a file to my PC with Ethereal running from the remote server. My PC (Windows XP) started the TCP session by advertising a large TCP window size (64,512), but the server returns a small window of 9,280:
4176 > http [SYN] Seq=0 Ack=0 Win=64512 Len=0 MSS=1160 http > 4176 [SYN, ACK] Seq=0 Ack=1 Win=9280 Len=0 MSS=1460 4176 > http [ACK] Seq=1 Ack=1 Win=64512 Len=0
Thus, the server will only send 9,280 bytes before it waits for an ACK from my PC.
Now, a few packets later when the HTTP 200 reply comes from the server the TCP windows grows to 16,384, but then it never got any larger:
Protocol Info HTTP HTTP/1.1 200 OK (application/zip) Source port: http (80) Destination port: 4176 (4176) Sequence number: 1 (relative sequence number) Next sequence number: 1161 (relative sequence number) Acknowledgement number: 428 (relative ack number) Header length: 20 bytes Flags: 0x0010 (ACK) Window size: 16384 Checksum: 0x7183 [correct] Hypertext Transfer Protocol Media Type: application/zip (762 bytes)
After a few seconds when TCP stabilizes it creates a situation where the server sends 14 packets then waits for the ACK from my PC. Since the round-trip-time is about 75ms between the two offices, the data transfer pauses while this ACK is in flight. Once the ACK was received, the data transfer starts again. I could see this again and again in the trace.
So, let's do some math. The server is sending packets of 1,214 bytes each. It takes about 85 ms total to receive the 14 packets and send an ACK. So:
14 packets * 1,214 bytes = 16,996 bytes (there's a full TCP window)
So, in 85ms, the server sends 16,996 bytes. Now do a proportion to find out how much is sent in 1 second (1000 ms):
16,996 X --------- = --------- 85 1,000 85X = 1,699,6000 X = 199,952.94 now convert to kilobytes: 199,952.94/1,024 = 195.26 KBps
Look familiar? That's the exact value my user was reporting as a problem. The network, and TCP, is working perfectly.
If you are able to get both ends to use the maximum TCP window size (65,536) with an 85 ms RTT, your theoretical maximum transfer rate is:
65,536 X --------- = --------- 85 1,000 85X = 65,536,000 X = 771,011.76 now convert to kilobytes: 771,011.76/1,024 = 752.94 KBps
As a network guy I convert all things to bits, so the theoretical maximum rate over the WAN is:
752.94 KBps * 8 = 6023.52 Kbps = 6Mbps
When you work through the packet trace this turns out to be a very simple math problem. It can be a powerful way to show users that the network is fine. Unfortunately, you won't be able to increase the speed of light for your users.
More >From the Field blog entries:
Go to Cisco Subnet for more Cisco news, blogs, discussion forums, security alerts, book giveaways, and more.
Michael Morris is a communications engineering manager at a $3-billion high-tech company. His background is in enterprise WANs working with telcos and developing large-scale routing designs. He has worked on networks at government and corporate organizations, including networks at two Fortune 10 companies. In his current role, he leads a team of 10 engineers responsible for large-scale IT networking projects and architectural standards for data networks, storage area networks, IP telephony, contact centers, and security. Michael is CCIE #11733 and recently became one of the first three Cisco Certified Design Experts (CCDE) ever (#20080002). He has 11 years experience in networking and communications, including four years as a paratrooper in the U.S. Army. He has a bachelor's degree in MIS from the University at Buffalo and is working on his MBA from NC State University. In 2008, he was awarded the Network Professional Association (NPA) Professional Excellence and Innovation Award for his work on network architecture, templates and enterprise MPLS design.
Michael Morris's From the Field blog is also featured on the Cisco Learning Network. See it there, along with the blogs of other Cisco Experts.