Policy Implications of FCC Reliance on Faulty ISP Performance Data

ISP performance testing should be subject to scrutiny and oversight

The recently published FCC "Connecting America: The National Broadband Plan" cites comScore speed test results at least five times. As reported in our last posting, comScore's testing methodology and data is plagued with flaws, and the concern is that the FCC is crafting public policy using this faulty information. Many speed test services suffer the same errors (or possibly introduce new ones), so the solution lies not with with a new speed test, but rather requiring full transparency regarding how speed tests are performed.

It is unsafe to assume that simply by using open source test software you will achieve complete transparency. Speed test methodology and analysis factors must be publicly understood to achieve transparency. We must have answers to the following questions and many more. Under what conditions is the test software operating? What precautions were taken to assure that the test is "clean" with no degradation outside the ISP's control? The industry needs a clear framework for each element of the test methodology to be properly defined. A proper description framework would assure that these and other questions are answered.

It is also imperative to have access to the actual raw test results. It is easy to (intentionally or unintentionally) insert bias into the results through the complex data aggregation and analysis that must be performed to reach a conclusion. That process must also be described--and access to the data must be open so others can either replicate the analysis or perform their own. The FCC has put out a call for transparency in how the ISPs define service levels. The same transparency needs to apply to the organizations policing those service levels.

For NetForecast's comprehensive assessment of comScore's technical and statistical methodology, read our "ComScore ISP Speed Test Accuracy" report.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:

Copyright © 2010 IDG Communications, Inc.

IT Salary Survey: The results are in