To know how your website is really performing you need to put it through ALL, not just some of its paces. There are two primary website testing approaches: one uses a slimmed down subset of browser functionality packaged into a ‘website tester’, and the other uses a fully functioning browser. If your website doesn’t shake, rattle, or roll, then the pared down approach may give you a good sense of your site’s performance–but if you have a dynamic, media-rich website, the diet version may delude you into thinking site visitors are having a happier experience that they actually are. Here are the results of some research we did to compare the two approaches.
What We Found
We did some experiments to see which approach gives the most accurate website performance measurement results, and found that full browser-based testing gives more accurate results because it measures the true end-user experience by executing all of the software and loading all of the content. In contrast, the slimmed down website tester-based approach doesn’t test a complete range of website elements.
You can see the different results in the average response times for MSNBC’s home page. In our testing, the average browser-based response time was consistently about six seconds slower than the website tester-based approach.
MSNBC Home Page Results
Test results for Microsoft’s home page are intriguing because at the start of the test period the browser-based test response times were approximately double those of the website tester response times. Then, about two thirds of the way through the test period, Microsoft apparently made a change that dramatically improved the website’s performance. That change is obvious in the browser-based results, but isn’t reflected at all in the website tester-based results.
Microsoft Home Page Results
The Takeaway
Many websites now routinely contain JavaScript code, XML, AJAX, Flash, Flex, and other sophisticated content. To accurately reflect the actual end user experience, you need to exercise all of these capabilities.
Website testers only exercise a limited part of a website and so miss much of the true end user experience time. They deliver metrics such as time to first byte, time to load the base page, and time to load the content referenced on the base page–but they don’t test all aspects of the site, nor do they retrieve information as a real browser would. For example, rather than fetch page elements in the sequence that the browser would follow, they may fetch them simultaneously. As informative as the resulting metrics may be, they constitute only a part of the true load time for a media-rich page.
Capturing near real-world metrics requires a fully functioning browser executing all of the software and loading all of the content as directed by the website.
Read more about our research in the NetForecast report: “Web Performance Measurements: Full Browser vs. Website Tester” found here.
