How we tested the switches

We assessed switches with 10 sets of tests covering L2 and L3 unicast performance; IGMP group multicast capacity; L2 and L3 multicast performance; NAC/802.1X; storm control; power consumption; switch manageability, security, and usability; and switch features. A detailed, complete methodology is available here.

In the L2 unicast performance tests, we configured each switch with a single VLAN encompassing all ports. We attached a Spirent TestCenter generator/analyzer to all 48 gigabit Ethernet and two 10-Gigabit ports on the switch and ran three sets of tests: all ports, gigabit ports only, and 10-gigabit ports only. We offered traffic to the gigabit ports in a fully meshed pattern and to the 10-gigabit ports in a meshed pattern. For each test, we conducted separate 60-second runs with 64-, 256- and 1,518-byte frames, and measured throughput, average latency and maximum latency for each frame length.

The L3 unicast performance tests were similar to the L2 unicast tests, except in this case we configured each switch port to use a different VLAN and IP subnet.

In the IGMP group capacity tests, we reverted to an L2 configuration, enabled IGMP snooping, and set the switch to act as an IGMP querier. In this test, 47 of 48 TestCenter Gigabit Ethernet ports joined some number of IGMPv3 groups; the 48th TestCenter port acted as a monitor to detect flooding.

After sending group membership (join) messages and waiting at least twice the switch's IGMP query interval, TestCenter's ScriptMaster software then offered multicast traffic to the switch's first 10-gigabit port, destined for all multicast groups. Per RFC 3918, if all groups received at least one frame, the test iteration was considered a pass. If loss or flooding occurred, the iteration was considered a failure. Using either a step or binary search algorithm, we repeated this procedure to determine multicast group capacity.

In the L2 multicast performance tests, we configured all switch ports to join a single VLAN, to use IGMP snooping, and to act as an IGMP querier. Then TestCenter's 48 gigabit ports joined 500 IGMPv3 groups (or fewer, depending on results from the group capacity test). After waiting at least twice the switch's IGMP query interval, TestCenter's ScriptMaster software then offered multicast traffic to the switch's first 10-gigabit port, destined for all multicast groups. Using a binary search algorithm, TestCenter determined the throughput rate. In a separate test, TestCenter measured average and maximum latency at the throughput rate.

In the L3 multicast throughput and latency tests, we configured each switch port to use a separate VLAN and IP subnet, enabled protocol independent multicast-sparse mode routing on each port, and set the switch to act as a PIM rendezvous point. The test setup and traffic pattern was similar to the L2 multicast test. We again determined the throughput rate and measured average and maximum latency at that rate.

To assess 802.1X/NAC support, we developed six scenarios that describe roles a switch might play as part of the NAC infrastructure. In this case we attached the switch to a Windows 2003 server running Juniper Steel-Belted Radius Enterprise Edition 6.1 (SBR). The SBR configuration used Windows Active Directory credentials to authenticate users.

In the first scenario, the switch places an authenticated client (in all cases, a PC running Windows XP Professional and Juniper Odyssey client software) into a previously configured VLAN. The second case is like the first, but requires authentication of multiple clients attached to a single port. In the third case, the switch dynamically assigns a VLAN after authentication. In the fourth case, the switch dynamically applies an access control list after authentication. In the fifth case, the switch places a client into a guest or restricted VLAN upon authentication failure. Finally, the sixth case determines whether a switch port concurrently supports 802.1X and MAC authentication support.

To assess storm control, we initially planned to use common attack techniques such as ARP and TCP SYN flooding but time and logistical constraints prevented us from completing these tests. Instead, we assessed storm control and anti-spoofing mechanisms by reviewing documentation and vendors' responses to our features questionnaire. While we spot-checked a few configuration parameters on some switches, we did not verify that every storm control and anti-spoofing mechanism worked as advertised.

We measured power consumption using Fluke 322 and Fluke 335 clamp meters. This test involved three measurements: AC line voltage; AC amperage when idle; and AC amperage when fully loaded. We fully loaded the switch control and data planes by configuring Spirent TestCenter to offer traffic at line rate to all ports consisting of IPv4 packets with IP options set. We derived wattage by multiplying voltage and amperage.

Our tests of switch manageability, security, and usability had objective and subjective components. In the objective component, we determined which management methods the switch supported over IPv4 and IPv6, as well as its ability to conform to best security practices (for example, by disabling vulnerable services such as telnet and enabling secure service such as SSHv2). We also determined which management methods were enabled by default, and which could be enabled/disabled by users. In addition, we determined whether erasing configuration files would remove all personally identifiable information, a regulatory requirement and security best practice.

The subjective part of our assessment consisted of our judgments on ease of accomplishing these and all other tests described here.

To assess the final area, switch features, we asked vendors to complete a detailed questionnaire. We did not verify every answer to this questionnaire.

-- David Newman and Rodney Thayer

< Return to main test: 10 Gig access wwitches: Not just packet pushers anymore >