Our tests focus on three server components - storage subsystem, CPU and network connection. We test each component with applications that emulate real-world conditions, because the performance of each subsystem affects the performance of the server as a whole in normal usage.
We use Bluecurve, Inc.'s Dynameasure File Professional 2.0 to test the storage subsystems, and Dynameasure SQL Professional 2.0 to stress the CPUs. We also run Mercury Interactive Corp.'s LoadRunner performance measurement tool to exercise each server's network interface and data bus systems.
We run all of our tests on each server against Microsoft Corp.'s Windows NT Server 4.0 and Novell NetWare 4.11, with one exception. Dynameasure SQL Professional is incompatible with NetWare's file addressing conventions, so our database testing results reflect the performance of only Windows NT Server.
We run each test several times, gradually increasing the number of simulated users. For the Dynameasure file and SQL tests we make eight runs, starting with six users and increasing by six users with each iteration until we reach 42 users. For the LoadRunner tests, we start with 40 users and increase by 40 with each test until we reach 280 users. We then graph each data point for each server.
In our file server tests, we use Dynameasure File Professional to copy several file types (binary, text, image, compressed binary, compressed text and compressed image) to and from the server. Dynameasure controls and compiles the data from the simulated users on each of the test clients.
Similarly, for our database tests we run Dynameasure SQL Professional to perform simulated order entry tasks employing indexed read transactions with two- and three-table joins that return a small number of rows.
To test the hardware's ability to act as a Web server, we created a LoadRunner script that simulates a user surfing a Web site, gathering HTML text and image data. The Web servers we used for these tests were Netscape Communications Corp. Enterprise Server 3.5.1 under Windows NT and the Web server bundled with NetWare 4.11.
Our price/performance graph compares the overall performance of the servers against their street price. To get the performance index, we first average the NT and NetWare results for our file and Web tests. Next, we generate a single unitless number for overall performance by factoring together the values from the file, SQL and Web tests. In that process, we weigh the file and SQL tests at 35% each, and the Web tests at 30%. Combined, they yield the overall performance value.
To generate our overall performance number, we first determine if our performance curves are linear. If they are, we average all the values; if not, we average the values in the linear portion of the curve, past the "knee." Finally, we normalize each of the resulting performance values for file, SQL and Web tests in order to make the values comparable. Specifically, we divided the average values for each of the three types of tests by the largest value we found in each category in this month's first test with our new benchmarking tools, in order to make sure that the scale of each set of results was comparable. We'll continue to use the normalizing factors from this round of tests in subsequent ones as well to make them all comparable.
RELATED LINKS
Review of six Pentium II servers
The first done with these benchmarks.
