We use Bluecurve, Inc.'s Dynameasure File Professional 2.0 to test the storage subsystems, and Dynameasure SQL Professional 2.0 to stress the CPUs. We also run Mercury Interactive Corp.'s LoadRunner performance measurement tool to exercise each server's network interface and data bus systems.
We run all of our tests on each server against Microsoft Windows NT Server
4.0 and Novell Netware 4.11, with one exception. Dynameasure SQL
Professional is incompatible with Netware's file addressing conventions,
so our database testing results reflect the performance of only Windows NT
Server.
We run each test several times, gradually increasing the number of
simulated users. For the Dynameasure file and SQL tests, we make eight
runs, starting with six users and increasing by six users with each
iteration until we reach 42 users. For the LoadRunner tests, we start with
40 users and increase by 40 with each test until we reach 280 users. We
then graph each data point for each server.
In our file server tests, we use Dynameasure File Professional to copy
several file types (binary, text, image, compressed binary, compressed
text and compressed image) to and from the server. Dynameasure controls
and compiles the data from the simulated users on each of the test
clients.
Similarly, for our database tests, we run Dynameasure SQL Professional to
perform simulated order entry tasks employing indexed read transactions
with two- and three-table joins that return small number of rows.
To test the hardware's ability to act as a Web server, we created a
LoadRunner script that simulates a user surfing a Web site, gathering HTML
text and image data. The Web servers we used for these tests were Netscape
Communications Corp. Enterprise Server 3.5.1 under Windows NT and the Web
server bundled with NetWare 4.11.
Our price/performance graph compares the overall performance of the
servers against their street price. To get the performance index, we first
average the NT and Netware results for our file and Web tests. Next, we
generate a single unitless number for overall performance by factoring
together the values from the file, SQL and Web tests. In that process, we
weight the file and SQL tests at 35% each, and the Web tests at 30%.
Combined, they yield the overall performance value.
To generate our overall performance number, we first determine if our
performance curves are linear. If they are, we average all the values; if
not, we average the values in the linear portion of the curve, past the
"knee." Finally, we normalize each of the resulting performance values for
file, SQL and Web tests in order to make the values comparable.
Specifically, we divided the average values for each of the three types of
tests by the largest value we found in each category in this month's first
test with our new benchmarking tools, in order to make sure that the scale
of each set of results was comparable. We'll continue to use the
normalizing factors from this round of tests in subsequent ones as well to
make them all comparable.
RELATED LINKS
Review of seven Xeon-based servers
Network World, 10/19/98.
