The FCC recently used speed test results from comScore as an absolute indicator of specific ISPs' performance. NetForecast analyzed comScore's testing methodology and data to assess whether it accurately reflects broadband ISP performance, and to assess the appropriateness of using the data to reach general conclusions about the actual performance ISPs deliver to their subscribers. We uncovered problems on both counts.
We found that the effective service speeds comScore reports are low by a large margin because its data calculations underreport performance and place many subscribers in a higher performance tier than they purchased. Details of our analysis are available in a NetForecast report, but here is an overview of our findings.
NetForecast's investigation of comScore's ISP speed test methodology uncovered the following data gathering errors:
- Only one TCP connection is used
- Client-server delay is variable
- Participants' computers may be resource constrained
- Test traffic may conflict with home traffic
- Decimal math is incorrect
- Protocol overhead is unaccounted for
- Purchased speed tiers are incorrectly identified
All of these data gathering errors cause the actual speed delivered by an ISP on its network to be underreported, and the individual errors create a compounding effect when aggregated in an individual subscriber's speed measurement. As a result, the actual speed delivered by each ISP tested is higher than the comScore-reported speed for each result of every test.
The data interpretation errors stem from incorrect speed tier identification. These errors are important because they cause the test results to overstate the disparity between the median actual and maximum advertised speeds. The problem stems from a flaw in comScore's method for determining the advertised speed, which often incorrectly bumps subscribers into a higher speed tier than they actually purchased--essentially penalizing an ISP for surpassing its advertised speed.
ComScore calculates the "average actual speed relative to the advertised speed" by dividing the consistently-underreported speed delivered by a frequently-inflated purchased speed. The combination of an inaccurately low numerator and inaccurately high denominator leads to a ratio that incorrectly shows ISPs delivering dramatically low performance relative to what they sold to their subscribers. The actual/advertised effective performance ratio based on this methodology is incorrect to such an extent that it is of extremely limited utility.
ComScore's speed test was designed for ISPs to compare performance to one another in specific markets. Although appropriate for its original use, the test is inappropriate for the FCC to use as an absolute indicator of specific ISPs' performance.
For NetForecast's comprehensive assessment of comScore's technical and statistical methodology, read our "ComScore ISP Speed Test Accuracy" report.