How we tested 6 compliance products

Here's how we conducted the Clear Choice Test of six compliance products.

New Boundary and Elemental provided preconfigured servers running their management components. We installed the management components of all the other products on Windows 2003 servers with 3-GHz processors and 1GB RAM. We used VMware Workstation to support multiple products on the same server.

We first tested agent deployment and the products' ability to connect to hosts with agentless technology. We had a test bed of 10 hosts, comprising Windows XP, 2003, 2000, Linux and Solaris. We ran discovery scans to identify systems not running the necessary agent and deployed agents using the product console, if possible.

We reviewed out-of-the-box compliance policies for the regulatory standards and best-practice configuration guidelines of the Health Insurance Portability and Accountability Act, Gramm-Leach-Bliley legislation, and the National Institute of Standards and Technology. We launched scans of our test systems with some of the default policies and reviewed the results.

We then tested the products' ability to customize default templates and configure custom checks. We created checks to ensure Sophos Anti-Virus and the Windows firewall were running and Google Desktop's search-across-computers feature was not enabled. We checked Windows patch compliance, values in several registry keys, password settings and user account status on Windows 2003. For the Linux and Solaris systems, we evaluated several configuration checks and installed patches.

To assess the products' access-control features, we created several users with different permissions. We also attempted to create a user who could view only reports, ideally through a Web interface.

When issues were identified during compliance checks, we read through the resulting report to see what information was provided on the identified issue and how to correct it. We then tested the products' autoremediation functionality.

For reporting, we looked for the products' ability to export reports into multiple formats and autodeliver to a defined e-mail address after a scheduled policy check. We attempted to create a delta report showing the specific changes made to a system over a period of time. We also looked for a product audit trail that showed which users performed what actions within the system.

< Previous: Preventsys offers twist | Conclusion: Conclusion >