How We Did It
We used equipment from Spirent Communications and Extreme Networks to build a test bed for high-availability testing. Spirent's SmartBits traffic generators were used to generate traffic that would cause our IP Security test device to establish 400 IPSec Security Associations. The traffic flowed from the SmartBits over an Extreme Summit 48 Fast Ethernet switch to an IPSec tester. From there, we went through another Extreme Summit 48 switch to a 100M bit/sec LANtronix hub, which was the common point of connection for all nodes in the high-availability cluster, then to another hub, another Extreme switch, and finally to the SmartBits on the other end.
Once we got 400 IPSec Security Associations established, we ran tests against each VPN device pair at load levels of 1M bit/sec, 10M bit/sec, 20M bit/sec and 40M bit/sec. Each test ran for 100 seconds. At 10 seconds into the test, we caused some high-availability event to occur, such as unplugging a patch cable or turning off one of the boxes. We measured the total amount of lost traffic. If a cluster failed to recover during the 90 seconds we gave it, we considered it a failure.
In any case in which the system failed, we did a complete reboot and re-established the 400 IPSec Security Associations before trying the next test. For systems without hardware accelerators (Rainfinity, Stonesoft and Foundry), we only reported numbers at 1M bit/sec and 6M bit/sec (rather than 10M bit/sec) because the systems could not keep up at speeds higher than that.
Our failover tests fell into four categories. Link failure tests were designed to simulate the most common network failure: a bad patch cable or connection. We ran link failure tests on the inside and the outside interfaces. Power failure tests, the second category, simulated a system crash, halt or shutdown. In the case of the Solaris-based systems, we didn't dare to repeatedly pull the power plugs out of the wall, so we simulated this by simultaneously disconnecting all the patch cables from the "failing" system and then going through normal system shutdown on the isolated system. This was important for failback tests, the third category, which attempted to measure how quickly a reinitialized high-availability system would rejoin the cluster and be ready for the next event.
Connectivity failure tests comprise the fourth test category. These tests simulated a more complex kind of failure in which the systems are able to communicate with each other, but some other kind of connectivity is lost, such as swapped patch cables or a failing or suddenly misconfigured switch fabric.
After running all the tests, we averaged and reported the lost traffic percentages across each category at each bandwidth.
For tests in which the product failed outright, we did not include the infinite failure time in the average but did note the failure in our overall results.
RELATED LINKS
