'WAN Optimization': the key WAN technology advance of the past decade

With combination of disk-based deduplication and fixing Microsoft CIFS for the WAN, Riverbed led in enabling data center consolidation

In our last several columns, we've been exploring the history of the Enterprise WAN and the consequences of that history.

The combination of reliable but very expensive private WAN Frame Relay and now MPLS services, and the need to address key factors affecting application performance over the WAN as well as the "chattiness" issues of certain applications has driven technological innovation in WAN Optimization and WAN Virtualization technology. And what most of the technological innovations have in common is that they take advantage of Moore's Law price/performance advances in cheap CPU and memory (and sometimes disk), and that they are all stateful.

Here we continue our exploration of the consequences of WAN Service history, and focus specifically on the two-ended WAN Optimization appliance market over the last several years, and in particular at how and why Riverbed Technology has led and shaped that market.

Where Peribit was essentially the VisiCalc of the WAN optimization industry – the first player with a complete solution – Riverbed quickly became the Lotus 1-2-3, the industry leader with a generally superior solution for the largest part of the market, a position they have retained to this day.

As we started to get into two columns back, Riverbed's two-ended appliance-based solution for WAN Optimization, in addition to taking advantage of CPU cycles and large DRAM, also introduced the use of a hard disk to be able to deliver "compression" ratios of 2.5x - 4x on average, better than the 2 - 3x more typical of DRAM-based compression implementations. At least as importantly, after the initial transfer of any file – or other large object – the data is "cached" (although for historical marketing reasons of not wanting to be compared to a Web cache, Riverbed and most of the WAN Optimization players dislike and go to great lengths to avoid the term "cache," preferring terms like "data deduplication"). Consequently, the second and successive access to the data from the same location can deliver "LAN-like" performance, with an effective compression ratio, and total acceleration factor, of up to 100x.

Just as for the DRAM-based compression technology, this "caching" / data deduplication ability to have bits stored on a local hard disk works not just for file access, but for any TCP-based application. Email most importantly, where large files are frequently sent to multiple users at the same location. This contrasts with WAFS (Wide Area File Services)-based solutions such as the one Cisco introduced. In a WAN environment where 128 Kbps to 2 Mbps were (and largely continue to be) the typical bandwidth at most locations, such a WAN Optimization solution deservedly beat WAFS-type solutions, even ones which could preposition files to the branch and thus potentially offer acceleration for the initial access to large files. [We will most assuredly return to this topic in future columns, as this equation can change when most locations have far higher available bandwidth.]

Beyond disk-based data deduplication, the other major innovation Riverbed introduced was an application-specific approach to Microsoft's CIFS protocol for file service. As we covered previously, CIFS in Windows XP and earlier versions was especially chatty, very much designed for LAN environments, and so problematic over the WAN, performing poorly on WANs with even moderate latency, or in the face of any packet loss at all. The SMB 2.0 file service protocol in Windows 7 and Windows Server 2008 and later versions are less chatty and work reasonably well with WAN latencies if there is no packet loss, but still suffer severe performance problems on the WAN in the face of loss.

By essentially doing local CIFS termination, Riverbed's solution provided much faster access to files on a remotely located file server even for the first access, and when combined with the previously noted data deduplication, delivered that LAN-like, up-to-100x acceleration factor for multiple accesses to the same data.

The trend towards data center consolidation began even before WAN Optimization was introduced, but the specific capability of Riverbed's WAN Optimization solution to speed up remote file access so dramatically for Windows-based file servers both accelerated the trend of data center consolidation, and that in turn was the single biggest reason propelling the enormous market uptake of WAN optimization.

Riverbed's roots were in the application and caching space, and so it is not surprising that they took an application-specific approach to speeding up the performance of applications on the WAN. Despite the fact that the market is labeled "WAN Optimization," in fact most of what is involved is optimizing individual applications to work better over the existing, fixed WAN (usually MPLS), rather than a solution to make the WAN itself work better for applications, as WAN Virtualization from companies like Talari or Ipanema, or cloud-based WAN Optimization solutions like those from Aryaka do.

After Riverbed became the market leader, most other vendors focused on keeping up with them, even as Riverbed themselves went back and filled in other areas (monitoring/visibility, support for UDP-based applications, QoS) where they had been behind. Silver Peak found an early niche as the high-performance leader, in terms of aggregate bandwidth supported, and did well in the market for data center backup and disaster recovery. But most players now find themselves competing in all parts of the market.

CIFS was the one protocol / application that truly was broken on the WAN. While "application-specific" optimizations can indeed improve performance for HTTP and HTTPs-based applications, for many other applications, there is relatively little that application-specific improvements offer for other applications and protocols, yet each application-specific optimization increases complexity – for both the network administrator and the vendor – and also solution fragility. So while there is no question that application-specific improvements for CIFS/SMB, HTTP and HTTPS bring clear added value, there are diminishing returns to most other application-specific optimizations in practice.

The market for WAN Optimization grew very rapidly from its infancy to over $1B in 2008, but has slowed more recently; based on analyst estimates, 2012 will likely be less than a $2B market. While 16% - 19% annual growth is nothing to sneeze at in IT these days, and growth from virtually nothing to $2B annually in just about 11 years is extremely impressive, the WAN Optimization market no longer has the mega-growth it used to have. Part of this is the beginning of market saturation, part relates to a platform transition from market-leader Riverbed, and part relates to the overall slowdown in IT spending. I believe a less spoken of part, however, is because of the difficulty of cost justifying the combination of the capital expense of appliance deployment (virtual or not), as well as the ongoing cost of management and operations, especially for smaller locations.

WAN Optimization will continue to be an important technology in enterprise WANs going forward; it is one of the technologies of the Next-generation Enterprise WAN (NEW) architecture. However, WAN Virtualization, the use of colocation facilities, distributed/synchronized file services and perhaps cloud-based WAN Optimization-as-a-Service are each more likely sources of innovation and market growth going forward.

In future columns, we'll look further at these other technologies comprising the NEW architecture that will lead WAN advances over the coming decade.

A twenty-five year data networking veteran, Andy founded Talari Networks, a pioneer in WAN Virtualization technology, and served as its first CEO. Andy is the author of an upcoming book on Next-generation Enterprise WANs.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2012 IDG Communications, Inc.

IT Salary Survey: The results are in