Out-of-order Packets Trash Voice and Video

If you plan to combine multiple T1s or E1s to construct a higher bandwidth logical link, you should know that combining links can lead to out-of-order packets that can hurt voice and video stream quality. There are several ways to fix this problem, and the right solution depends on your situation.

Our colleague John Bartlett encountered this problem working with a company that has a large central office and many small offices scattered around the country. The company is migrating to an MPLS VPN-based WAN and wants to move its videoconferencing from ISDN onto the new IP-based infrastructure. Currently the remote offices are served by a single T1 line, with a second T1 line carrying voice along with ISDN carrying video. The plan is to convert the second T1 to IP traffic and combine it with the first link to provide about 3 Mbps of bandwidth to each office.

The two T1 lines could be combined to look like a single link using either Cisco Express Forwarding (CEF) or Multilink Point-to-point Protocol (MLPPP).

CEF gives the user several ways to balance the traffic load across the two links. One way assigns packets to a link on a round-robin basis. The first packet goes into link A, the second into link B, the third into link A and so on. The receiving router then reassembles the packets into a single stream and forwards them on. The problem is that with CEF the packets remain in the order received, and if one link has more delay than the other, it is possible (even probable) that packets will be received in the wrong order. TCP stacks automatically reorder packets upon arrival so they are delivered to the application in the right order. TCP can handle out of order packets but even a few begin to trigger window behavior that is similar to packet loss which degrades throughput.

Voice and video endpoints are very sensitive to packet reordering and may run out of buffering or the necessary CPU power to reorder packets that consistently arrive out of order. Voice and video applications thus become the canary in the coal mine showing signs of a network performance problem ahead of other applications.

Another CEF technique assigns a flow to a specific T1 link. This works adequately for voice streams. A call between phone X and phone Y always uses T1 link B, and so all its packets traverse the same link and remain in the right order. Phone calls use relatively low bandwidth and many calls will end up fairly evenly distributed across the two T1 links to balance the load.

For video, however, this causes problems because it uses much more bandwidth per stream. Suppose you want to support two 384Kbps video conferencing calls from a remote office. A 384K call actually uses 460Kbps at the network level. To support two 460K streams you need to assign the video class of service (2 * 460K / 3 Mbps) = 31% of the link bandwidth. If this is split evenly across the two links, there will be sufficient bandwidth if and only if the two simultaneous video calls are assigned to different links. CEF does not guarantee this as it does not distinguish between a video call and any other IP flow. If three flows come up in a row (video, web, video) then the two video calls will be assigned to the same T1 link.

So what can you do to fix this? You can use MLPPP, which assigns a sequence number to each packet crossing the virtual link, and reorders packets on the receiving side to ensure they are forwarded in the same sequence as sent. This is what John recommends. But he points out that there is no such thing as a free lunch. The processing overhead required to re-sequence packets on the receiving end is usually done by the router CPU and can slow things down. To address that, you may need to upgrade the router at the end of the MLPPP link to ensure it can handle the extra work.

In short, if you have two or more T1 or E1 links between offices and you want to combine them to support voice and video, examine the tradeoffs and choose the right algorithm and the right hardware to make it work for your situation.