Great topic. One of the issues I see is a desire for monitoring to show the end user experience is or the business transaction. That approach is great at identifying that a problem exists, but almost useless to solve it. The support desk is probably getting calls at the same time that provide identical information.

If you have a baseline of performance components at lower levels it is much easier to identify where something has changed. Getting a grip on latency, loss, jitter, server response times (DNS, web servers, databases), data volumes, routes, CPU utilization... means that you have information that you can take action on.

Unfortunately it is hard to match the actionable data up to the business transaction. Add in the semi-blind state because no site has perfect coverage. With some good tools many problems jump out at you and others potential issues can be ruled out.

If all else fails then blame the group that can diagnose the problem quickest (or at least escalate to this group). Since the network group has tools with the broadest visibility, they can determine whether the problem is in the network or an application and which application is the culprit.

Regards,
Ben Haley