Four steps to application nirvana

Good application performance management is about people and process.

Many companies start down the application performance management path by investing in management products and stop there. But NetForecast’s research shows that the biggest performance bang for the buck comes from investing in people who implement APM best practices.

Listen to a related podcast discussion: Curing application management headaches.

NetForecast recently completed a benchmarking survey of 300 enterprises and their APM practices and results. Here’s what we found:

•  Enterprises investing the most in good practices more than double application performance effectiveness compared with those investing the least.

• In particular, they experience a better than 300% improvement in their ability to solve problems quickly, a nearly 150% improvement in their ability to learn about problems proactively rather than through user complaints, and they are twice as likely to favorably assess response times for their important applications.

•  This year’s results show a 10% overall improvement in benchmark scores compared with last year, indicating that enterprises are doing a better job of implementing best practices.

•  In addition we learned that the ITIL ITSM framework is gaining popularity, while the FCAPS framework is on its way out.

Four steps to APM best practices and benchmarking

When all is said and done, what matters is how users experience an application — can they reach it, and once there, can they remain productive. Good application performance management optimizes application availability and response time, and best practices help make that happen.

Best practices harness human behavior, education, relationships and communication to understand, measure and communicate about application performance — as well as to link application performance to the business. These best practices must be embedded into a continuous improvement process that ensures application performance meets your business needs.

This process begins by understanding your user and application needs, gathering (and measuring) relevant performance data, and reporting (or communicating) that data in understandable form to the right people.

The reports serve as input for IT and business groups to collectively determine what measurements and thresholds best support the business (such as linking).

Here are the four best practices and some examples of what they entail:

1. Understand:•  Define what technical parameters are important.Establish which applications are mission critical. Document this information and distribute it throughout the organization.

• 

• 

2. Measure:•  Measure the important technical parameters defined above. Track those measurements over time. Set critical thresholds. Automate data gathering and correlation.

• 

• 

• 

3. Report:•  Submit relevant performance reports regularly to management. Communicate important measurement information throughout the enterprise (along with explanations about what it means and how it should be used — when needed).

• 

4. Link:•  Work with business managers to ensure that applications monitored are business critical. Establish business-relevant performance targets, and application-level service-level agreements. Meet periodically with business managers to review the above.

• 

• 

Benchmarking these best practices allows you to assess your progress. A benchmark score shows on a numerical scale how well you are implementing best practices. This allows you to compare your own best practice implementations with the industry norm and with those who are executing well and achieving best results.

Survey results: Continuous improvement

This summer’s survey of 300 enterprises is the second APM benchmarking survey NetForecast has completed — the first was in February 2007.

We analyzed each respondent’s answers to formulate an APM benchmark score for each of the four best practices using a 10-point scale, with 10 the best (highly organized and formal approach) and 0 the poorest (no attempt to perform the function). We then aggregated the four individual best practice benchmark scores into an overall best practice score for each enterprise.

This year’s median benchmark score for all surveyed enterprises is five — smack in the middle of the 10-point scale. Only three enterprises achieved exemplary benchmarks scores above 9, and only two received scores below one.

This year’s median score is a 10% improvement, compared with the results from our February 2007 survey, which showed a median of 4.

And this year’s smooth bell-shaped distribution around the median is better than last year’s curved distribution, which bulged asymmetrically toward poorer scores. This indicates that overall APM practices are getting better.

Diving into the individual best practice elements, communicating scored the highest marks, followed by understanding, linking and, finally, measuring.

Comparing this year’s scores to last year, the biggest improvements came in communicating and linking, which showed 40% improvements. The median score for understanding application performance improved 30%, while the median score for APM measurement best practices remained essentially unchanged. These results point to process improvement over the past year.

So, APM practices improved over the past year. But how does that translate into results?

Higher benchmark scores equals better performance

The survey results show extremely positive correlations between best practices benchmark scores and actual application performance delivered to the business.

On the whole, enterprises with excellent best practices deliver 100% better results to their users than those with poor practices.

Here’s where the rubber meets the road. Our survey results show that best practices exert their most dramatic effect on improving the time it takes enterprises to solve problems, with a 338% score improvement in problem-resolution time among those with best practices compared with their poorer-performing counterparts.

The bulk of those with poor benchmarking scores describe their problem resolution times as too long, whereas respondents with high benchmarking scores generally describe problem resolution times as meeting or exceeding industry norms.

Those with the top best practices scores were more than twice as likely (144%) as those with poor scores to discover problems through systems vs. learning about them from users — and they were twice as likely (93%) to favorably assess the overall response times for their important applications. Highest scoring respondents were also more likely (70%) to assess the performance of their business-critical applications as meeting their business needs.

Availability showed the least improvement in effectiveness from the worst to the best performing enterprises. This may indicate that availability is relatively consistent across the survey population. It may be that availability may be nearing as good as it can get.

ITIL is in

We asked survey participants if they have adopted or plan to adopt formal frameworks to improve application performance management. Sixty-two percent of respondents have adopted or plan to adopt the IT Service Management (ITSM) framework as defined by ITIL, while 24% have or will adopt Fault, Configuration, Accounting, Performance, and Security (FCAPS) as defined by ISO.

There was a hefty increase in the percentage of ITIL ITSM adoptees from 54% in 2007 to 62% this year, and an even heftier decrease in FCAPS adoption from 46% last year to a mere 24% this year. The ITIL ITSM framework is clearly waxing in popularity and the FCAPS framework is on the wane

Distant contenders this year are the Object Management Group’s (OMB) Common Object Request Broker Architecture (CORBA) at 5%, and the Distributed Management Task Force’s (DMTF) Common Information Model (CIM) barely on the radar screen at 3%.

Impediments to APM best practices

Finally, we asked enterprises to identify impediments to improving application performance. Insufficient cross-group collaboration, insufficient manpower, and lack of proper tools tie for the top of this year’s list of impediments with nearly 50% of respondents mentioning them.

A second grouping of impediments were mentioned by about 40% of respondents — perceived insufficient value for the effort required, inadequate process and lack of expertise.

We saw a 15% increase over last year in respondents citing insufficient cross-group collaboration as an impediment. We surmise that managing application performance requires more interdepartmental collaboration than departments have been used to, and organizations have yet to implement processes that facilitate the needed collaboration. Groups are often more accustomed to sniping at each other about who is at fault than working together to solve performance problems.

Also of interest is a 12% increase in respondents citing perceived insufficient value for the effort required as an impediment to improving application performance. This likely reflects increasing awareness that investment is needed in APM facilities and process, yet convincing management to spend is often challenging because the return on that investment is hard to quantify.

Conclusions

The NetForecast survey results show a compelling relationship between implementing best practices and better enterprise business application performance. This indicates a worthwhile return on the investment in the people and processes required to improve your best practices benchmark score. Our data reveals that the higher your benchmark score the better your business-critical performance results are likely to be.

We also find that enterprises are quickly learning the APM best practices ropes, with a 10% overall improvement in benchmark scores over last year. That improvement may continue because better application performance makes a business healthier and users happier and more productive. In short, the NetForecast study results show that implementing APM best practices is well worth the trouble.

Until now discussion about application performance management has centered on products. As important as products are, it is the people and the processes that make performance better. We encourage enterprises to approach APM holistically and put products in their proper place within a larger business and human-centered context.

Visit our App Performance View blog at for more findings from the NetForecast survey.

Sevcik is president of NetForecast and founder of the Apdex Alliance. He is a leading authority on measuring, assessing and improving the performance of networked applications. Sevcik has contributed to the design of more than 100 networks, including the Internet, and to the success of more than 25 application management products. He can be reached at peter@netforecast.com.

Wetzel is an associate of NetForecast and a veteran of the data networking industry. She works with network product vendors and service providers to develop and implement successful product and marketing strategies. She can be reached at rebecca@netforecast.com.