Evaluation points

I remember evaluating management products four or five years ago based on their completeness. They won points by having strengths in data gathering, data store, topology, analytics, visualization and automation, when appropriate. Evaluation in terms of completeness made sense at the time, and carried with it a kind of reassurance. It meant that a management package we might recommend would be good at all the core enabling capabilities that management products should be good at.

During the last two years, however, I’ve changed my approach. No single management software package is likely to be good at all of the above.

Rather than looking for completeness, Enterprise Management Associates became much more interested in just where a given vendor was putting the bulk of its investments. Even frameworks and platforms – which are in reality made up of many management software components – tend to have certain strengths and weaknesses.

The vendor community is itself becoming more aware that, in this age of consolidation, no single software investment is an island. For each management product to boast its own superior completeness is becoming less glorious and less efficient. Plus, new technologies such as Web services hold the potential to support an environment of open and distributed services across different management components from different vendors.

Still, here are more details on the capabilities to look for. I have, with some trepidation, decided to mention some vendor examples for the purpose of clarity. To those vendors keeping score I have to stress that this list of examples is NOT COMPLETE, nor are individual vendor references intended to comprehensively reflect product functionality. EMA is considering the development of a 500-listing vendor directory – just to put the very few examples below in perspective.

* Data gathering. This is obviously the foundation for any type of effective management. Points can be given for breadth of data assimilated from SNMP and RMON2, for support for systems- and application-related information, and for synthetic and observed transactions for evaluating performance. Policy-based data gathering with some form of analytics relevant to critical management tasks also gets points. Low cost, ease of deployment and minimal burden on a network also get points. And the ability to integrate and support other systems is critical. One example of highly efficient, agent-based data gathering for OEM support that emerged into a more complete product is Network Harmoni, recently acquired by Micromuse.

* Data store. While access to different intelligent agents may seem easy and natural enough, vendors that excel in data stores are still fairly inconspicuous within enterprise management. Nevertheless, framework vendors are putting significant investments behind designing and managing data stores. Computer Associates’ Common Object Repository is a good case in point. NetScout’s Common Data Model is a good example of a strategy designed to support a broad range of performance-management analytics from multiple brands. However, recognition that a data store is a key foundation for designing a complete management strategy is more prevalent in OSS, so it’s not surprising that more OSS-related vendors, such as Quallaby, have leading-edge, open data stores.

* Topology and topological associations. These can range from network topologies, such as with Visionael, to configuration-related data, such as those from a whole host of vendors. Mapping relationships can help the management of application and system components. Topology can help support everything from problem resolution to inventory and asset management, provisioning, and service planning.

* Analytics. These are rightfully the single biggest area of investment today across the industry. Effective analytics are a critical foundation for automation. Analytics support everything from problem isolation (SMARTS, Aprisma, Magnum, Tavve, NetQoS, Panacya); to design and planning (Opnet, Compuware, Hyperformix); to service-level management, service accounting and modeling (XACCT, HP OpenView’s IUM, Opticom); to auditing and correlating actions taken in change and configuration management; to performance management.

* Visualization. This may seem like the lightweight in this list, and I suppose that, compared to analytics, that’s true. However, visualization can mean many things. Points are given for role awareness with a design supportive of real-world decision-making. Points are also given when operational insights can be automatically linked to business and service priorities. High scores are given for visualization capabilities that can integrate information from multiple sources with enhanced business and operational context (Managed Object Solutions, Edge Technologies).

* Automated actions. These can help with everything from cutting a trouble ticket effectively, as with BMC’s Remedy; to provisioning, such as with IBM’s Think Dynamics acquisition; to simply resetting a threshold; to software distribution based on policies and client needs; to quality of service, such as with Packeteer; to many other areas.

This isn’t to say that you should plan on one single dominant vendor for each of these categories. In some, such as analytics, this will likely never be the case – as the requirements are too broad. In others, such as topological development – the world is just not yet ready. But you should consider how the management products you buy today can be leveraged to support your other management investments over time.