How we did it

By John Bass
Network World, 09/18/00

Our testbed comprised 15 clients with a minimum configuration of two 400 MHz Pentium II processors with 128M bytes of RAM. Each client has one 100M bit/sec Ethernet network interface card for connection to the Fast Ethernet test network.

We used Benchmark Factory test software to coordinate the test development, client load, result gathering and archiving for all the tests. We ran a series of file, network and database tests against Windows NT 4.0, Windows 2000 and Novell

Subscribe to the Product Review e-mail newsletter

NetWare 5.1.

Using Benchmark Factory software, we defined several tests to look at the performance of the file subsystem. We broke our benchmarking tests into two categories -- small file and large file transfers.

For the small file transfer tests, we used a three-dimensional test matrix of transfer direction (read/write), block size (1K byte/8K byte), and transaction type (random/sequential). This test matrix resulted in eight tests. We separated all combinations into individual tests to see how each server would react in each situation. The small file transfer tests use a file size mix of 80% 1K-byte file, 10% 10K-byte files and 10% 50K-byte files.

For the large file transfer tests, we combined the reads and writes together in the same tests. We then created a set of tests that covered all combinations of the transfer type (random/sequential), and block size (1K byte/8K byte). This resulted in four tests. The reads and writes were combined because this emulates large file service behavior for services such as FTP and home space services. The reads and writes were distributed as 90% reads and 10% writes. For each of the large file transfer tests, the file size distribution was 80% 500K-byte files and 20% 1M byte files. Ninety percent of each of the file sizes are reads and 10% are writes.

Each of the files needed for each virtual user was created at the beginning of each test. Each test ran five iterations of increasing load. The number of virtual users started on each client controlled the load. The number of virtual users for each step was found by running each of the tests against each network operating system to find where the knee of the performance curve lay. From there, we determined a standard set of load parameters to run the tests.

The CPU database test used Microsoft SQL Server 7.0 with both NT and Win 2000. We increased the number of virtual users from two to 30. The number of virtual users in no way implies a limitation the database server. Each virtual user in our test is atypical of a 'real world' user in that the load generated by these virtual users is much larger. Each virtual user calls a computationally heavy stored procedure residing on the database server thus reducing the number of network transactions per unit time to concentrate the load on the processors. The virtual users then waited 9 seconds before executing the Stored Procedure again.

We used another database test employing a much less intensive transaction to test the network I/O performance of the server. This was achieved by generating a large amount of transactions against the server.

Like the CPU database test, the number of virtual users is increased from two to 30 to obtain a characteristic performance curve.

To test this server's performance in a NetWare environment, we ran similar CPU and network test against an Oracle database running on NetWare 5.1.