EXCLUSIVE TEST: Huawei switch: Good first effort

Chinese vendor enters crowded U.S. switch market with device that delivers excellent power consumption, decent performance

Page 2 of 2

On the other hand, there’s no true standout feature of the switch, and there were places where we wished the software were more polished. This is a decent, workmanlike effort – but for Huawei to succeed in North American networking it will need to bring some sizzle along with the steak.

How we tested the Huawei switch

We tested the Huawei switch for performance, features, usability, power consumption, and security.

To evaluate device performance, we used the Spirent TestCenter traffic generator/analyzer to run the industry-standard benchmarks described in RFCs 2544, 2889, and 3918.

For unicast traffic, we assessed the switch in terms of throughput and latency by offering traffic on all ports at line rate. We also determined unicast MAC address capacity.

In the throughput/latency tests, we used separate traffic patterns for gigabit and 10-gigabit Ethernet ports. Initially, we used a four-port fully meshed pattern for the 10-gigabit Ethernet ports, but later scaled that back to a port-pair pattern. We offered fully meshed traffic on all 48 gigabit Ethernet ports.

All tests involved the seven standard Ethernet frame lengths recommended in RFC 2544 – 64, 128, 256, 512, 1,024, 1,280, and 1,518 bytes – and the test duration was 60 seconds in all cases. Also as recommended in RFC 2544, we measured latency at the throughput rate.

To measure MAC address capacity, we used the RFC 2889 wizard in Spirent TestCenter. This wizard conducts a binary search to find the largest number of MAC addresses a switch can learn without flooding. In all test iterations, Spirent TestCenter's MAC address aging timer is set to twice that of the switch under test. We ran the RFC 2889 wizard on three ports, and then manually repeated the test on 48 switch ports. We determined MAC address capacity to the nearest 1,000 addresses.

We measured multicast performance with tests of throughput and latency, and of multicast group capacity. For throughput and latency, we followed the aggregated multicast throughput procedure described in RFC 3918. We configured the Spirent TestCenter test tool to act as an IGMP querier, and to offer multicast traffic to one of the switch’s 10-gigabit Ethernet ports. We also configured the Spirent tool to join the same 400 multicast groups on all 48 gigabit Ethernet ports, using IGMPv3 join messages. After sending the join messages, Spirent TestCenter offered traffic to all subscribers at line rate. Here again, the test duration was 60 seconds. We measured throughput and latency using the seven standard Ethernet frame lengths from RFC 2544.

To measure multicast group capacity, we used the RFC 3918 wizard in Spirent TestCenter. This wizard joins a fixed number of groups, and then attempts to forward traffic to all groups. The test instrument used a binary search to find the highest number of groups joined successfully. The test passes if the switch forwards traffic to all groups. If the switch fails to forward traffic to one or more of the groups joined, the test fails. We used the most stressful possible condition of having receivers on 48 gigabit Ethernet ports concurrently join all groups. Spirent TestCenter offered multicast traffic to one 10-gigabit Ethernet port.

We measured power consumption using Fluke 335 clamp meters. This test involved three measurements: AC line voltage; AC amperage when idle; and AC amperage when fully loaded. We fully loaded the switch control and data planes by configuring Spirent TestCenter to offer traffic at line rate to all ports. We derived wattage by multiplying voltage and amperage.

To evaluate device security, we used Spirent Mu-8000 security analyzer to run a protocol mutation test on the switch’s SSH server daemon. In a mutation test, the Mu analyzer iterates over every field in SSH headers, injecting unexpected and/or illegal values, and then assesses the response of the device under test. Taken individually and together, this test involves nearly 40,000 different mutations of possible SSH requests and responses.

We also used the SSH modules in Rapid7’s Metasploit security assessment tool against the switch.

During both the Spirent and Metasploit security tests, we concurrently ran continuous SNMP queries and offered data-plane traffic at 10 percent of line rate to 50 ports (2 10-gigabit Ethernet and 48 gigabit Ethernet). The purpose of this additional traffic was to verify availability of other control- and data-plane functions while the SSH daemon was under attack.

Thanks

Network World gratefully acknowledges the assistance of vendors that supported this project. Thanks to Spirent Communications, which supplied its Spirent TestCenter and Mu-8010 test instruments for this project. Spirent's Liang Kan and Emil Moral also provided engineering support for this project. Thanks also to iXsystems, which supplied an iX-2212 storage server and the FreeNAS storage appliance; to VMware, which supplied its vSphere 5 virtualization software; and to Fluke, which supplied a Fluke 335 clamp meter for power measurement.

Newman is a member of the Network World Lab Alliance and president of Network Test, an independent test lab and engineering services consultancy. He can be reached at dnewman@networktest.com.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:
| 1 2 Page 2
Must read: 10 new UI features coming to Windows 10