Crowning our new switch leaders

By CHARLES BRUNO AND KEVIN TOLLY
Network World, 05/10/99

The spin doctors at network switch companies spend countless hours devising stories that they hope will position their switches as cost-per-port leaders, throughput champions and standard bearers. These folks are good at what they do, making it all that much tougher for you to weed through their claims and make informed buying decisions.

To give you at least one independent point of comparison when evaluating switches, The Tolly Group and Network World devised a benchmark that makes it easy to see which vendors have the best price/performance ratio in the switching business.

In the inaugural 1999 SwitchMetric, we benchmarked 15 products - seven Gigabit Ethernet switches, six Fast Ethernet switches and a pair of token-ring switches - from nine vendors. Using a state-of-the-art, $1.35 million test bed featuring Netcom Systems' SmartBits Advanced Multiport Performance Tester/Analyzer/Simulators [More on our methodology]. The Tolly Group identified each switch's best aggregate throughput. We then divided that figure by the manufacturer's suggested retail price, as provided by the vendor, for the switch configuration tested.

The result is a cost per gigabit of throughput, a new metric intended to help you determine the price you pay for each gigabit per second of throughput a switch provides. Think of it as analogous to the cost-per-MIPS metric used in benchmarking computer systems.

Gigabit Ethernet, Layer 2 leaders

Switch Cost/Gigabit

3Com Corp.
SuperStack II Switch 9300
$1,325

3Com Corp.
CoreBuilder 9400
$1,764

Foundry Networks Inc.
BigIron 4000
$2,280

Foundry Networks Inc.
BigIron 8000
$2,413

HP
ProCurve Routing Switch 9308M
$3,515

Click for complete SwitchMetric results

The 1999 SwitchMetric establishes a set of baseline results that we will build on over time. As we bring more switches into the lab and subject them to our benchmark, we'll continually add the results to our database. We will keep an online leaderboard up to date on Network World Fusion [Click for the leaderboard], and the full results database will be available from The Tolly Group at switchcentral.tolly.com. Results will be published in print at least a couple of times each year, as well.

For this initial round, Network World and The Tolly Group invited every switch maker to the benchmarking party. Nine accepted and were able to deliver switches in time for this initial report. Ten more agreed in principle and promised to participate in later tests. Only Cisco outright declined.

Cisco executives argued that users are interested in more than simple price/performance numbers. We couldn't agree more, but we also know that price/performance is still an important consideration - albeit one of many. Our door is still open should Cisco change its mind.

The inclusion or absence of vendors, switch topologies and protocols tested is indicative of the state of the network market. Olicom, for instance, was the only token-ring vendor to test in the initial round, underscoring the popularity of Ethernet-based switching. (You'll find the Olicom results in our online charts. Based on its numbers, it appears token-ring switch vendors will have a tough time making the price/performance case vs. their Ethernet brethren.)

The results: Gigabit switches

For each type of switch, we gave vendors the option to participate in any or all of three basic tests - Layer 2, Layer 3 IP and Layer 3 IPX - depending on which markets they believe their products are best suited to serve. For each protocol, we conducted separate tests with frame sizes of 64, 512 and 1,518 bytes. The cost per gigabit of throughput results offered are derived from tests with 1,518-byte frames, which offer the least processing overhead of all frame sizes tested and thus should result in the highest throughput.

On the Gigabit Ethernet switch side, all switches offered between 12 and 64 ports and were configured in a fully meshed network design. Nearly all of the switches tested achieved wire-speed throughput at each frame size, although there were some exceptions, all of which are detailed in our online charts.

3Com's SuperStack II Switch 9300 and CoreBuilder 9400 proved to be the best value for Layer 2 Gigabit Ethernet switches, offering a cost per gigabit of throughput of $1,325 and $1,764, respectively, in the 1,518-byte test.

This is especially interesting in light of the fact that the 3Com switches were the only Gigabit Ethernet products that failed to deliver wire-speed throughput in 1,518-byte tests, with the 9300 at 94% of wire speed and the 9400 topping out at 85%. On the other hand, both devices did achieve wire-speed rates and exceptionally low cost-per-gigabit ratings in the 64- and 512-byte frame tests.

3Com officials point to an Ethernet clock tolerance issue as a possible reason why its switches failed to perform at wire speed for the 1,518-byte tests. In short, the clocks that drive a switch and the SmartBits traffic generator tool operate independently; if the switch clock is behind the SmartBits' clock time, the test tool could transmit enough data to overwhelm a switch's buffers, causing packets to drop.

Foundry Networks' BigIron 4000, a 32-port Gigabit Ethernet campus switch, delivered the next best value in the Layer 2 tests at $2,280 per gigabit of throughput using 1,518-byte frames. Foundry also delivered the best cost per gigabit of throughput for 64-port Gigabit Ethernet switches, with a figure of $2,413 for its BigIron 8000.

In the Layer 3 tests, however, Foundry bested the competition by a margin of more than $1,000 for IP and IPX traffic. The BigIron 4000 and BigIron 8000 delivered cost per gigabit of throughput figures below $2,500 in each test, whereas the closest competitor, the Hewlett-Packard ProCurve Routing Switch 9308M, came in at just over $3,500 for IP and IPX.

The test results support the idea that many gigabit switches can unify subnetworks and perform as the routing point in the center of an enterprise network backbone. The simple fact that the majority of these devices offered wire-speed throughput indicates they are well-positioned at Layer 2 and Layer 3 to push routers further out to the network's edge. The ability to manage routing and switching from a single device should prove a cost-effective alternative to purpose-specific routers.

Fast Ethernet findings

On the Fast Ethernet side, The Tolly Group tested six devices with port densities ranging from eight to 84 ports. All were primarily workgroup-oriented switches, some were hybrid models with Gigabit Ethernet uplinks, some were pure Fast Ethernet, and others mixed Fast Ethernet and 10M bit/sec Ethernet. We derived a cost-per-gigabit figure by adding the total capacity and expressing it in a gigabit-per-second rate.

Among pure Fast Ethernet switches, HP's ProCurve Routing Switch 4000M delivered the lowest cost per gigabit of throughput at just $956 for 1,518-byte frames in our Layer 2 test. And HP's ProCurve Switch 2424M, a 24-port Fast Ethernet switch with one Gigabit Ethernet uplink, delivered a cost per gigabit of throughput of $1,195 - the lowest cost for any hybrid Fast Ethernet/ Gigabit Ethernet switch.

Note, though, that both HP switches stopped short of offering full wire-speed throughput. The 4000M delivered 94% of its theoretical maximum 4G bit/sec throughput, and the 2424M delivered just under 91% of its rated 3.4G bit/sec throughput.

Another interesting note is that the price/performance spread between Layer 2 and Layer 3 functionality on Cabletron's SmartSwitch Router 2000 is nil, indicating that some switch makers may no longer be charging a premium for Layer 3 functionality in Fast Ethernet switches. Aside from Cabletron, no other vendor offered Layer 3 functionality in a Fast Ethernet switch, an indication that vendors that participated in this round of testing are committed to Fast Ethernet in the flattened edges of enterprise networks.

Going forward

While this first round of testing for the 1999 SwitchMetric provides a good deal of insight into the cost implications of Gigabit Ethernet and Fast Ethernet switches, the findings will become more valuable throughout the year as we test products from additional vendors and add those results to the mix. At an average cost per gigabit of throughput of $2,840, Gigabit Ethernet switch vendors appear to have some work ahead to catch up with 3Com's $1,325 minimum cost per gigabit of throughput at Layer 2.

Likewise, on the Fast Ethernet switch side, HP has set the price to beat with a cost per gigabit of throughput of $956. Testing also demonstrated that the best cost per gigabit of throughput deals may not necessarily deliver wire-speed throughput, leaving users with an interesting choice between cost and performance.

The gauntlet has been thrown down. We invite any and all vendors to come into the lab and try to best 3Com, Foundry and HP, which we expect will be back to try to beat their best.

Crowning our new switch leaders

The results: Gigabit switches

Fast Ethernet findings

Going forward

Related Links