NOSes /

How we did it

By Joel Snyder
Network World, 05/15/00

We built a test lab using Intel-based servers and 10 clients. The directory server was a generic dual-Pentium III/650-MHz system with 380M bytes of memory, running a standard Windows NT Version 4/Service Pack 6a installation. The disk was a Fast/Wide SCSI drive from Seagate and a standard Adaptec controller. The test clients were Cubix Density servers based on the Celeron 366-MHz chip set (www.cubix.com). All network communications were via switched 100M bit/sec full-duplex connections. Our entire setup was hooked to a Cybex Autoboot Commander 4XP that allowed us to connect dozens of PCs to a single monitor. Without that we would not have been able to control everything (www.cybex.com).

Our data set was composed of 100,000 "person " entries using the standardized InetOrgPerson schema. Each entry had 23 attributes, and the entire data set was 75M bytes in length. We figured that with a 75M-byte data set, each directory should be able to hold the entire data set in memory.

We used a mix of benchmarks, drawing from Mindcraft's Directorymark 1.1, as well as tools from Innosoft and Netscape to conduct high-volume directory queries to verify that the Mindcraft numbers were not off the mark with any one product. Because Directorymark 1.1 had certain ambiguities in the benchmark, we chose to report raw numbers for the Mindcraft tools rather than the summary "Directory Operations Per Second" value from combining all numbers.

Before each test, we ran a warm-up for 10 minutes to help preload the cache from the database. This was a major advantage to servers that did perform caching, as we could see the performance of the server change dramatically over the first few minutes of the warm-up.

Three of the tests used the query profile recommended by Mindcraft for Directorymark 1.1: messaging, addressing and modify. The messaging test used repeated queries on a single indexed field with exact matches and a single Lightweight Directory Access Protocol (LDAP) bind operation at the beginning of the run. Messaging picked a subset of 10,000 entries from the 100,000 entries for their tests. We ran the messaging test with a single client and with 10 simultaneous clients, reporting both numbers. Each test was run three times, and the average was reported.

The addressing test was similar to the messaging test, although with substantially more types of queries: LDAP bind operations after every fifth operation, exact match searches on four different fields, wildcard searches and some operations that were supposed to return "not found " results. (The exact mix is described at Mindcraft's Web site.) As with messaging, we ran the addressing test with one client and with 10 simultaneous clients.

The modify test retrieved entries from the directory and updated one or more fields. We ran the modify test with a single client until every single entry in the database had been updated exactly once.

Bulk-load testing was conducted using whatever bulk loading tool was provided by the vendor of the directory. In some cases, bulk loading was done by writing directly to the directory database. In other cases, the vendor provided no tool specifically for bulk loading, and the data was loaded "over protocol," meaning that either the LDAP protocol or, in the case of the X.500 directories, the DAP protocol, was used to load the entries into the database. In general, loading over protocol is substantially slower than using a special bulk-loading tool.

We also used tools provided by Innosoft and iPlanet to perform queries of the directory using exact match hits. We reported the Searchrate results for one and 10 simultaneous clients. In theory, these results should be identical to the Mindcraft messaging tests; in practice, they generally differed by a few percentage points. We wanted to use multiple tools to validate that there were no bugs in either test set that would produce pathological results.

Back to main story