A recent study might put an end to a longtime debate on the most effective method to measure and optimize end-user experiences online.

NetForecast put out a comparative study, per the request of application performance management vendor Adlex, which compares products from the likes of Adlex with competitive offerings that take a different approach to performance measurement. According to NetForecast's results, real testing, while technically challenging in terms of the volumes of data collected, reveals more accurate end-user experience data than synthetic measurements.

The two approaches have often spurred discussion because of the various challenges of each approach and the validity of the results following the tests. Adlex and companies such as BMC Software, Coradiant, Computer Associates, IBM, HP and Ipsum, offer tools to measure application performance for end users by tracking real traffic. For real testing, measurement tools often use appliances or software probes to passively monitor all client and data center interactions by attaching to a mirror port on the edge router in a data center.

Synthetic Web application and site performance measurement tests from the likes of Keynote and Gomez use agents distributed throughout the Internet to determine how a Web site and the applications running on it react to peak loads and various geographies. Yet despite broad areas of coverage, NetForecast says the synthetic tests don't capture enough data for IT managers to get a clear picture of how well their sites meet service levels and how well applications respond to end-user requests.

"Our analysis shows that synthetic measurements randomly and frequently understate the actual response time seen by real users but as much as 50%, and overstate the response time of other applications by as much as 350%," states the study performed by NetForecast analysts John Bartlett and Peter Sevcik.

The report goes on to say that synthetic agents usually interact with less than 10% of all Web site URLs and less than 15% of the ISPs connecting real users to the site.

"It is dangerous to extrapolate the performance characteristics of an entire application environment and its user communities from this small sample," the report states.

Ann Bednarz is associate news editor at Network World.