Keep those Webplates spinning
Performance monitoring platforms that can keep your site in balance.
Anyone managing a Web or e-commerce site can sympathize with the juggler whose stage act consists of balancing a variety of spinning plates on poles. Like that classic performer, the Web site administrator must keep a variety of systems in balance while the whole world is watching. Just like those spinning plates, the failure of one part of the Web puzzle can easily bring the entire edifice crashing down around the ears of a hapless administrator.
As Web sites become more complex, and the degree to which companies depend upon their Web infrastructure for minute-to-minute operations increases, a crop of tools has grown up to help administrators keep everything in proper motion. Together with TesCom USA, we decided to look at products that aim to solve the problem of Web site performance monitoring.
How we did it
Interactive Scorecard and NetResults
Making your Web site healthy
Web site monitoring interactive buyer's guide database
Archive of Network World reviews
Subscribe to the Product Review e-mail newsletter
We invited eight vendors to participate in the review, and four accepted. We looked at Freshwater Software's SiteScope, Platform Computing's Site Assure, Holistix Software's self-named Holistix and Agilent's Firehunter/Pro and Firehunter eCommerce combination. All four packages were operated in a Windows NT/Windows 2000 environment (see "How We Did It"). Also invited were Resonate with its Resonate Commander, ProActiveNet with eBizSuite, NetScout with nGenius Performance Monitor and Netuitive with Netuitive 5.0. Even though several of the vendors offer monitoring services as well as monitoring products, we tested only the software products. The Buyer's Guide interactive chart includes a range of products and services concerning Web site performance and site management.
|
Making assurances
Platform's Site Assure uses agents to monitor targeted systems. Its tests verify the availability of services, examine performance counters and respond to problems by taking direct corrective action or by notifying an administrator via pager or e-mail. Key to this sort of testing, especially for Web servers, is the ability to test the availability of Web pages, both individually and as a complete set of pages making up a Web transaction.
Site Assure's method for developing a Web transaction test involves stepping sequentially through the relevant pages in an abstract presentation, showing the administrator the actual HTML source content being returned at each stage. Unlike the methods of the other products, this lets you make selections based on good or bad returned content that might be hidden from an actual user's view.
Site Assure's Live Event viewer presentation gives you a good real-time display of how systems monitored by Site Assure are doing at any particular moment. Combined with the historical logs, the system can present a fairly comprehensive view of target-system activity.
Site Assure can respond to problems by alerting an administrator with e-mail messages, SNMP traps, console messages, event log entries, or by performing conditional testing, agent-centered actions (such as restarting a service on a remote system), or enabling or disabling entire sets of policies.
In our test, we emphasized e-mail as the alert method and agent actions as the corrective measure. Both were easy to configure, and the only problem we encountered in actually performing an automated restart of a service was that we were shutting down the wrong service for test purposes.
Additionally, Site Assure had the most robust multiuser security features of any of the products, allowing the administrator to define each administrative user's access to monitors and policies with explicit control over read, modify and execute access on the individual monitor or policy level. Thus it would be easy to set up a Web administrator with great control over the entire Web server but read-only access to a related SQL server, and give a database administrator mirror-image controls over the SQL server with only visibility to the Web server.
Site Assure reports are largely graphical, and can be added or customized by the administrator, pulling data from the various configured agents. These reports are also live rather than historical, updating information as the agents continue to monitor the system. More traditional reporting, in the historical sense, is accomplished with log files maintained by the system.
Installation was simple, with options to install the full manager, a remote console or local "agent" service. Agents are required on the targeted systems, and communication between the manager and agent may be encrypted, using a set of four dedicated TCP ports (defaulted to starting at 4040). This allows for easy configuration to cross firewalls if needed. The encryption may be set to none, 64-bit or 128-bit as desired. The manager and all agents must be set to the same encryption level, but may be accidentally configured to different levels, preventing communication. We also had a minor gripe in that screens that ask for passwords or encryption keys aren't explicitly clear about whether a new value is being defined or a previous value is being used.
The process of deploying the various testing agents seemed a bit cumbersome the first time through, but the process quickly became familiar. This was especially true because each agent can gather a large number of related metrics, each of which may have several optional settings such as values for "polling" and "smoothing" intervals. It only required deploying two or three agents, though, before we had a good feel and were able to deploy five or six more with relative ease.
Our testing did identify one explicit bug: The agent templates delivered did not properly handle negative values for some threshold tests. Such values must be put inside quotation marks for the tests to operate properly.
With all its strengths, we were fairly disappointed in Site Assure's documentation. The online manual is essentially just an electronic version of a printed book, with a table of contents on one side and the text on the other. Unlike a printed book, you can't hold it in your hands and flip through it for information, nor does it make up for that shortcoming by taking advantage of any good visual presentation. The material is accurate, but pedestrian.
The Holistix approach
Holistix Software's Holistix product uses "monitors" (the specific tests and metrics) grouped together into "Web applications" (the system pieces that make up a particular functional Web site or e-commerce system). "Action plans" encapsulate the various notification and corrective actions that the system can take, while "policies" define the relationship between the Web applications, their monitors and the various available action plans. The installation process asks for the default Web site to monitor and sets up a default Web application with some basic monitors based on that information, greatly easing the familiarization process.
Adding monitors to a Web application definition was easy, and we especially liked that we could explicitly test the monitors and action plans before committing them to use. This was useful in setting up e-mail notification, which by default attempts to relay messages through a server at Holistix. In our case, this relay would have presented a severe problem because our target e-mail server was not on a public network.
Like the other products, Holistix allows a monitor to be defined that tracks a series of Web access steps that make up a Web transaction. Holistix's URL Sequence monitor can be built step-by-step, with the Holistix manager accessing each page, showing the controls and links for each page, and letting the administrator choose the next step from those items marked "good" or "bad." While this method worked well, we preferred Site Assure's method of showing the entire returned HTML content, which made it easier to pick out the page content that would define a good return, indicating that the right page or answer had come back rather than just any reply.
Holistix's policy templates let us create sets of commonly used metrics and associated actions. You can also apply system-level templates, which in the case of an NT target let us identify the resources on the target via any combination of port checking, SNMP, PerfMon counters or conversation with an installed Remote Holistix Agent. We were surprised there wasn't a monitor that told us whether a particular NT service was running, something even Freshwater's completely agentless SiteScope could do. Holistix officials say such a monitor is planned in a future release.
The general status monitor gave us a good overall look at application health, while the predefined reports are broken into several categories for easy access, including: performance/availability, troubleshooting, operation and configuration. The health summary reports were interesting because they rolled up most of the other details into a high-level graphical view of the application state and underlying metrics. However, there is no capability for customizing or adding to the reports directly in the product.
Of the four products, we found Holistix's abilities to respond to problem conditions the most impressive. Holistix incorporates the concept of escalation into its response model, allowing the administrator to easily define a set of action plans, the results of which may vary depending on how long or how often an error condition exists. For example, an action plan could be developed that pages an operator when a problem first occurs, pages a manager if the problem persists for a few minutes, restarts a service if no correction occurs in 10 minutes and escalates to some sort of crisis response after that.
We installed Holistix from a provided CD, though it is also available as a download from the Holistix Web site. Holistix maintains the configuration and metrics information in a database, using the included Microsoft Data Engine (MSDE) or a SQL Server 7.0 back end. Holistix recommends using the MSDE database only for small installations. In our case, we used that rather than the SQL option to avoid placing product-specific information into our SQL server or deploying another instance of SQL.
The online help system is well laid out, and contains some good examples and entry-level explanations to get the new user started in understanding and setting up a Holistix monitor system. We referred to the help files several times during the installation and configuration process, and were usually able to find what we needed in only a few moments.
Scoping it out
Available as a download from Freshwater's Web site, our immediate observation about SiteScope was "basic is as basic does." Uncomplicated from beginning to end, the simplicity of SiteScope impressed our testers. But that simplicity masks a fairly powerful and useful tool.
SiteScope uses monitors to lay out its services, with a monitor being a single type of test performed against a single target system. For example, a URL monitor tests the availability of a particular Web page on a given system, and a database monitor connects to and retrieves information from a specific database. SiteScope provides 32 types of monitors, including CPU, Domain Name System (DNS), disk space, FTP, mail, memory, ping, performance counters, and several URL and Web server monitors. Interestingly, SiteScope provides these monitors using only the service running on the one management system. Unlike the other three products that rely on agents running on target systems, SiteScope has no such requirement.
Among the monitors available with SiteScope is an HTTP Transaction Monitor, which lets you test a series of HTTP steps that might occur in a complex Web transaction such as searching a database or completing a purchase. With our basic e-commerce Web site, constructing such a test wasn't particularly difficult, but for more complex sites the manual nature of building a transaction might be limiting.
Another feature we liked was that unlike the other three products, SiteScope never seemed to place screen elements outside the scope of an 800 by 600 resolution display - a major plus to an administrator not blessed with a large format display or the visual acuity of an eagle.
After configuring a series of monitors to look at the availability of NT, network, HTTP and SQL Server resources in our test network, we then configured an e-mail alert to let us know when a resource became unavailable. The setup process for e-mail notification includes a test feature, which includes visibility into the entire SMTP conversation - excellent for debugging the setup. A minor gripe was that once successfully configured, the conversation display went away too quickly for us to read. Alerts by pager and SNMP traps are also available, though we did not test them.
Because it uses no client-side agents, SiteScope's ability to take corrective actions relies on running applications and batch files local to the server where SiteScope is installed. To ease use of this feature, Freshwater provides several sample batch files that can be used almost out of the box to perform such tasks as restarting failed services. One problem we had was that the scripts may rely on SiteScope passing the name of the alert that called the script, yet the default alert names don't necessarily carry the correct information to identify a service. Also, we were surprised that no sample script was provided for starting a remote service. However, devising such a script (which also overcame the alert-name problem) only took about 5 minutes.
A predefined set of basic reports gave a clear picture of target system behaviors and the alert conditions. The report formats strike a fairly good balance between online and printed readability, and can be detailed enough to satisfy most needs. One report of a day's behavior (or more properly, exaggerated misbehavior) on our test system ran to 24 pages without becoming overly cluttered.
Installation required an e-mail address for the SiteScope administrator, the target directory, and stating whether we were updating from another version. After that, we installed the SiteScope NT service, the documentation and a Java run-time engine. A few more simple configuration options (entering the name for our SiteScope server and selecting a couple of security settings) and the product was ready to go.
Like three of the four products, SiteScope comes with only online documentation in a Web-based format. However, the SiteScope documents include two versions, one intended for online use and the other a single-file format provided explicitly for printing. While sending the 328 pages to a printer in a single pass might make you temporarily unpopular with co-workers, it provides a nice option if you still like a physical document. Whether online or physical, the documentation is well laid out and straightforward, covering basic concepts, terminology and explicit steps in a logical sequence that minimized the need to dance madly from place to place to accomplish a task.
Firehunter
Firehunter, from Agilent, is actually a combination of products. Firehunter is available in three versions: Firehunter/L, Firehunter and Firehunter/Pro, each of which is designed for a different scale of environment. Added to the midlevel Firehunter product in our test harness was the Firehunter eCommerce package, which adds some specific tests for commerce-based Web sites. Of the four products we tested, Firehunter eCommerce seemed to be the most promising out of the box, but instead soon proved to be the most frustrating.
The Firehunter package is qualitatively different from the other three products - its focus appears to be on business management more than systems management. Three aspects of the product highlight this difference. First, Firehunter has tools for explicitly defining the goals of and measuring compliance with service-level agreements (SLA). Second, the baseline measurements and planning reports included are meant to help ISPs and application service providers (clearly Agilent's intended market) determine what SLA goals will be achievable. Finally, tie-ins are provided to allow Firehunter to view extracted data from commerce sites and report on the current "business health" of the monitored site.
The eCommerce option adds several additional metrics, including the ability to define Web transactions and Web-related SLA metrics. The Firehunter method of building a Web transaction test was the most direct of any of the products we tested, using a two-pane display to show each Web page of the target system next to the options for recording the steps as one goes through them. Firehunter does this by placing a proxy service between the target system and the administrator's Web browser, with the Web browser configured to point to the Firehunter proxy. Unfortunately, while the browser must be set to use the proxy to record the session, it must not be set to do so when initiating the recording session - requiring the administrator to switch back and forth to build each test.
Firehunter has another interesting distinction from the competing products in our test - it is the only product that presents separate administrative and user consoles. The administrative console is where the services and associated tests are defined, while the user graphical user interface (GUI) provides a view into the test results, associated events, various reports and a help desk view showing administrative notations.
As soon as we installed it, Firehunter merrily scanned our network segment and built a default list of services that it was prepared to monitor. Whatever arguments against the idea of automatically scanning for services in a network, it is an instructive process, and we discovered several services and systems visible from our test harness that we had not realized were there. In a larger, real-world environment, such a function could be enlightening to managers who are supposed to know what's going on in their Web sites.
Understanding how to define the tests to be performed was helped by the sheer number of tests predefined during the initial scanning phase of setup. Additional tests could be added by dragging-and-dropping icons from one side of the administrative console onto the list of monitored systems on the other, and then filling in the relevant blanks. Some of the necessary information was not clearly defined, but most questions could be muddled through with some educated guessing.
The user GUI had its own set of problems, none of which were crippling but were annoying at the least. The first portion of this application, the help desk, is a view-only window into annotations about events made elsewhere. Similar to what users might see visiting the network status page of an ISP's Web site, these messages might include notes from service technicians that a particular server was down for maintenance or that a router was now fixed. These annotations could be made through the detailed status displays on other tabs. But annotations made to event records rather than to server or service records did not seem to appear. Thus a technician's note that he was working on the DNS failure might not appear on the help desk screen, if the note was appended to the record of the failure rather to the DNS service. This would seem to limit the use of that information.
The GUI's graphs view allows the user to see detailed graphic information about each metric being gathered. The view is cumulative, in that selecting a metric adds its graph to the page of graphs - without (unfortunately) an option not to do so.
The graphs display can thus rapidly become cluttered with extraneous information. Asking the system to refresh its data clears the displayed graphs - a rather unexpected result. There is a separate command to refresh the graphs, but it was not obvious.
Firehunter seems to be focused on an environment where administrative and help desk personnel are actively monitoring the systems at most times, or at least immediately available. Paging, e-mail and SNMP traps are the methods discussed most in the documentation, along with "running reports," which is actually just a batch process for extracting the product's reports to a Web site or similar target. This batch process option could be used to take other corrective actions (in much the same fashion as with Freshwater's SiteScope), but we couldn't find examples or sample scripts.
Even defining actions such as an e-mail alert was extremely cumbersome. No selection lists or drop downs are provided, requiring the administrator to handwrite notes or do a multiple window "stare and compare" to transpose long, convoluted definition strings from one portion of the system to another without the benefit of cut-and-paste. Correctly setting up an e-mail notification required three attempts, and after each, the configuration needed to be saved and the Firehunter service restarted. We found no test capability, short of waiting for the defined event to occur and seeing if a message appeared in the target mailbox.
On our first attempt, installation of the base product would not complete, but appeared to hang when trying to initialize the services. One error message, which we finally noticed hidden in a list of installation status messages during our third installation attempt, pointed us to a possible problem with reverse name resolution. However, reverse resolution appeared to be working normally, so we called Agilent's technical support. That process quickly revealed that the problem was a bad record in an outdated "hosts" file on our server, which the Agilent software was using to do reverse lookup. Eliminating the bad record allowed installation to complete.
Firehunter was the only product in our test group that came in a box, with hard-copy documentation. Included were "concepts" and "installation" books that provided good overviews of the intended uses for the product, but which often left detailed instructions to the online HTML format documentation. In general, this approach would be good, but we found that when we clicked on various help icons in the administrative tool, many of the documents were not stored where the system expected them to be, and some did not exist anywhere that we could find.
Overall, we found Agilent's Firehunter too temperamental to be effective. The overall number of irritations, from the lack of lists and drop-downs when configuring tests, to the stubborn insistence on showing a graph for everything we ever clicked on, made using the tool more work than the results merited.
Sorting it out
Of the four products we tested, three seemed to be in fairly close contention, while Agilent's Firehunter fell short in our estimation. Among the other three, Platform's Site Assure was our pick by a slim margin, largely because of the richness of its security model, an often-neglected feature in administrative tools. However, the final selection for a particular site is likely to come down to issues of personal preference or site-specific needs on the part of the administrator who will use the tool. Concerns about multiuser security definitely gives the nod to Site Assure.
For small to midsize sites, however, the ease of deployment and the flexibility with which tests can be grouped, moved and redefined may make Freshwater's SiteScope the logical choice. A need for multilayer incident responses, on the other hand, may tip the scales toward Holistix Software's Holistix. But any of the three seem able to help you keep your Web plates spinning along.
RELATED LINKS
Keep Those Webplates Spinnings
Interactive Scorecard and NetResults
Web site monitoring interactive buyer's guide database
See which tool would best suite your needs.
