Jim participated in a recent Network World chat on the topic of application performance management and WAN acceleration. This is the third in a series of newsletters that will expand upon the questions raised during that chat. Today's topic - WAN deduplication.

WAN deduplication has the same goal as compression and caching - to send less data over the WAN. The way WAN deduplication works is that the first time that a file is sent across the WAN, deduplication algorithms identify data patterns in the file and store these patterns in WAN optimization appliances on each end of the WAN link. On subsequent passes, the deduplication algorithms in the appliances identify these patterns and replace large pieces of data with notably smaller pieces of data.

However, the way that WAN deduplication is implemented impacts how effective the solution will be. One of the implementation options is the layer of the OSI stack where the deduplication is applied. For example, a WAN deduplication solution that works at the TCP layer (often referred to as Layer 4 or the transport layer) optimizes only TCP traffic. Any applications that use UDP, proprietary or encapsulated protocols (i.e., GRE, and IPv6 tunneled inside of IPv4) can only be optimized by a deduplication solution that works at the IP layer.

Another option is whether the solution is disk or RAM-based. Disk-based systems typically can store as much as 1,000 times the volume of patterns in their dictionaries as compared with RAM-based systems, and those dictionaries can persist across power failures. The data, however, is slower to access than it would be with the typical RAM-based implementations, although the performance gains of a disk-based system are likely to more than compensate for this extra delay. While disks are more cost-effective than a RAM-based solution on a per byte basis, given the size of these systems they do add to the overall cost and introduce additional points of failure to a solution. Standard techniques such as RAID can mitigate the risk associated with these points of failure.

Yet another option is that some optimization solutions implement a token-based approach to deduplication while other solutions implement an instruction-based approach. A token-based approach relies on the use of a token to represent chunks of data. In contrast, an instruction-based approach uses specific start-stop instructions to indicate where duplicate data can be found and retrieved.

Steve Taylor is president of Distributed Networking Associates and publisher/editor-in-chief of Webtorials. Jim Metzler is vice president of Ashton, Metzler & Associates.