FCC’s Opportunity to Improve ISP Speed Testing

Accurately measure what matters and develop a quality improvement plan

Good public policy requires good information, and as we pointed out last week, when crafting its national broadband policy the FCC relied on tests that measured the wrong things, and did so inaccurately. In the FCC's defense, repurposing comScore data was expeditious and better than nothing--but for public policy's sake the FCC should get better data in the future.

First off, ISP speed tests need to measure what matters--and what matters is the speed an ISP actually delivers to customers. Measuring actual circuit speed requires removing performance variables outside an ISP's control (e.g., home computer resource constraints, and home traffic/test traffic contention). None of the testing solutions the FCC is currently using (comScore, Ookla, and M-Lab) can do this because they use software-based tools that measure from the browser. Isolating an ISP's performance requires placing a special piece of measuring equipment in a customer's home.

Going forward, the FCC needs to design a test for highest accuracy, not just repurpose tests that happen to be convenient. This means hardware must be added to the mix. This approach overcomes the testing shortcomings we detailed last week. We know of at least one testing outfit, SamKnows that uses such an approach. SamKnows has developed test and measurement software running on a Cisco router placed in a subscriber's home to continuously track the performance of the broadband connection. This device serves as the home access router and also provides the platform for SamKnows-operated tests.

The critical aspect of the SamKnows approach is that it performs tests only when there is no traffic from any home device on the ISP access line. The tests measure up and downstream speeds using multi-thread test methodology. In addition the software measures other performance parameters such as latency, DNS responsiveness, and voice over IP quality.

The FCC should foster such hardware-based approaches to ISP performance measurement. This is a golden opportunity for the open source community to create test and measurement software for Linux-based routers. Customers and service providers alike will benefit from such alternative choices for accurate measurement.

No new measurement approach will be perfect, nor will it be the last word in accuracy--so the FCC must continually press to improve measurement and reporting quality. Long-term quality goals and the uses for performance information must be well defined so ISPs, testers, and the FCC can collectively raise the bar for test accuracy.

The FCC should seize this chance to lead ISP speed measurement efforts, not simply depend on whatever testing solutions happen to be currently available in the marketplace.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:

Copyright © 2010 IDG Communications, Inc.

IT Salary Survey: The results are in