Layer 4 switch summit

When it comes to testing Layer 2 and Layer 3 switches, there's widespread agreement that throughput is the key benchmark. But simply counting

packets per second doesn't cut it when it comes to the new breed of Layer 4 switches.

These products, known as Web switches, e-commerce switches or content-aware switches, are typically targeted at a specific function, such as load balancing. What separates Layer 4/7 switches from Layer 2/3 switches is the ability to look inside packets and make sophisticated forwarding decisions based on information contained in the HTTP header.

Currently, there is no standard benchmark or agreed-upon set of metrics to help network executives get a handle on the performance of these products. And there is no single test tool that can simulate the real-world Internet traffic flows needed to run these switches through their paces.

In an effort to rectify the situation, The Tolly Group and Network World recently invited switch vendors and test tool makers to a two-day conclave on the New Jersey shore.

Attendees included representatives from switch/ load balancing/cache vendors Alteon WebSystems, ArrowPoint Communications, Cabletron, F5Labs, Foundry Networks, Lucent, Radware, Holontech, InfoLibria and iPivot/Intel. The testing community was represented by Ganymede Software, Netcom Systems and the National Laboratory for Applied Network Resources. NLANR's Web Polygraph tool is widely accepted as the benchmark for testing cache products, but it doesn't yet have a test tool for Layer 4 load-balancing switches.

The vendors checked their competitive instincts at the door - and during sessions led by Kevin Tolly, president and CEO of The Tolly Group, and Network World Editorial Director John Gallant - came to agreement on a number of key issues.

Probably the most significant agreement was that the baseline benchmark for Layer 4 switches should not be packets per second, but transactions per second, with a single transaction consisting of a request and a response.

"Testing a Web switch requires a new approach," said Ervin Johnson, director of technical and product marketing at ArrowPoint. The important metric is the time it takes for a client to make a request and for a Web server to respond to that request.

The group agreed on these other key metrics:

* The number of HTTP connections per second the switch is able to set up.

* The number of concurrent connections the switch can handle.

* The error rate, or the rate of transaction requests that aren't completed.

The group also agreed that because the true test of a switch is the ability to recognize and then properly forward different types of traffic, a proper test needs to send a workload through the switch that closely resembles real-world traffic. And that's not easy when it comes to Web traffic thanks to a number of different variables.

The group decided to create a baseline Web site scenario that looks like this:

* 100% of the traffic is HTTP Version 1.1.

* The mean file size is 10K bytes.

* The median file size is 3.5K bytes.

* The maximum number of persistent connections is 64.

* The maximum number of simultaneous connections per client is four.

* The number of client IP addresses per hour is 64,000.

* The number of virtual IP addresses is 32.

* The number of real IP addresses equals the number of server ports.

* The percentage of malicious traffic is zero.

* The percentage of encrypted traffic is 5%.

The idea was that these would represent the defaults on "dials" that could be turned up or down depending on the requirements of a particular test.

The group also divided the Layer 4 switch market into three categories and determined which metrics were most important in each. The categories were:

* Web site hosts and ISPs.

* Individual e-commerce sites.

* Enterprise networks.

The enterprise would typically have a single router link to the Internet, limited numbers of source IP addresses and multiple types of traffic, representing various corporate applications.

A Web site hosting company or an ISP would have multiple router uplinks, large numbers of virtual IP and source IP addresses; would require denial-of-service protection; and would need significant connection capacity.

An e-commerce site would be concerned with redundancy, encryption, server-to-server transactions and ensuring that the load-balancing switches created persistent connections with potential customers.

While the vendors accomplished a lot during the meeting, there is still work to be done. "The true measure of the meeting's success will be if any uniform benchmark gets implemented," Tolly says. "The onus is now on vendors and test tool companies to coordinate any such uniform benchmarking effort."

For network executives, this means it might be appropriate to ask a Layer 4 switch vendor how its switch stacks up against these key metrics and to ask the vendor what role it is playing in the benchmarking effort. i