Based on NetForecast's analysis of the test methodology underlying the FCC's Measuring Broadband America report and our experience evaluating alternative consumer broadband performance measurement methodologies, we conclude that the SamKnows consumer premises measurement boxes used produce high-quality test results. That said, however, some of the test servers introduced data inconsistencies and errors, and the data analysis was insufficient transparent to fully validate the results. (This posting is part of a series from NetForecast's newly published report FCC's "Measuring Broadband America" Report Tells Only Half the Story).
The FCC's broadband performance monitoring project involves measuring Internet connection performance at consumers' homes using on-site performance measurement 'white boxes' provided by the British firm SamKnows. At the other end of each consumer test site is an array of target test servers.
The SamKnows boxes are well suited to the FCC testing because they are minimally affected by consumers' home network, devices, and traffic and they provide:
- Better accuracy than alternative measurement solutions
- Controlled performance measurements
- Consistent and repeated tests
- Broad ISP subscriber participation
- Tests of many critical performance parameters
- Reliable customer premise devices
NetForecast's analysis revealed that the array of servers on the other end of the consumer tests are sources of data inconsistencies and even errors because their locations are not subject to due diligence, and they are managed by a hodgepodge of organizations and administrative controls.
On the data analysis front, we note insufficient transparency in explaining the data analysis methodology and we note inconsistencies between the stated data analysis plan and its execution. More descriptive information is needed to determine and validate how data calculations behind the FCC results were made.
These criticisms should be viewed as areas for future improvement. On the whole, the FCC report provides the most thorough and accurate documentation to date of the true performance delivered by US ISPs, and it is a commendable start to the FCC's ongoing ISP performance monitoring program.
Click here to get the FCC report.