ISPs and Usage Meter Accuracy

Independent validation of a Comcast usage meter

A number of broadband ISPs, including AT&T, Time Warner Cable, Charter Communications, and Comcast have announced their intent to meter Internet usage. Assuming they follow through on this intent, which is controversial in the blogosphere, it will be extremely important for consumers that the meters they use accurately and fairly track usage. For the past seven months NetForecast has tested and analyzed the accuracy of one of these meters deployed last week in a trial by Comcast in Portland, Oregon. You can find our complete report on the analysis here.

The meter we tested is based on the traffic counter of the Cisco Cable Modem Termination System (CMTS) model 10000. Our analysis validated that the accuracy of the Comcast meter for subscribers served by the Cisco 10000 CMTS is within plus-or-minus 0.5% over the month.

Over the coming weeks, we will share more of our test results and associated conclusions with our readers. This week we begin by describing our testing methodology and summarizing our results.

Usage Meter Testing Methodology: To perform the testing, we acquired regular Comcast High-Speed Internet accounts in various locations on the Comcast network. We instrumented the accounts with a test laptop PC running Windows XP and a Linux-enabled Linksys router on which we installed Tomato router firmware, and we used FTP accounts on various NetForecast servers on the Internet.

The basic NetForecast test involved a Perl script performing an FTP file transfer from one of our test servers to the laptop. The test consisted of repeatedly transferring files of several file sizes in complex patterns. These tests were also performed as uploads from the laptop to the server. The Perl script generated a log file documenting the transfer results and detailed timing information for each transfer.  The Tomato firmware provided a second independent view of the traffic transferred during each test.

We took extreme care to ensure that no traffic other than the test traffic was sent or received through the cable modem. First the laptop and router were the only devices connected to the cable modem.  Second the laptop was cleansed of all applications that could generate traffic not needed for the tests. Furthermore Windows XP and other applications were configured not to ask for or receive any software updates. Finally, remote management of the laptop was carefully scheduled not to occur during testing.

For each test, we produced our own carefully documented records of traffic sent up and down. The first record was the basic file size as recorded on a typical PC. That value was adjusted to account for FTP/TCP/IP and Ethernet overhead both upstream and downstream. This became a calculated estimate of the actual traffic sent and received on the wire. The second record came from the Tomato firmware which also recorded the traffic it processed both upstream and downstream to the cable modem.

We continuously received meter data from three key places in the Comcast meter system during testing phase. First we received IPDR records from the ARM collecting IPDR data from the CMTS involved in testing. The calculations were validated under controlled conditions at the NetForecast test lab. This was an early indicator of the CMTS traffic measurement accuracy. Next, we received hourly traffic records from the CEMP. This provided a detailed preview of the result presented on the meter portal. And finally, we logged into each of the Comcast accounts to observe the meter as displayed on the meter portal (data shown to the subscriber).  This completed the end-to-end view of meter accuracy.

Meter accuracy validation is the comparison of the three traffic reports from Comcast against the real traffic observations from the NetForecast tests. The NetForecast observations were seen by NetForecast only and were not shared with Comcast.  The Comcast data was supplied by Comcast to NetForecast without Comcast's knowledge of which tests were performed and when they were performed.

NetForecast performed these tests from May through November 2009 under a variety of conditions. It is important to note that NetForecast only saw IPDR data and meter records for cable modems that were on NetForecast test accounts.

Here are the results from NetForecast's independent meter validation study.

Usage Meter Test Results: Based on the results of our testing and analysis, we validated the Comcast meter for subscribers connected to the Cisco 10000 CMTS to be accurate within plus-or-minus 0.5% over the month. Furthermore, the reporting system maintains that accuracy throughout all the elements of the meter system--up to and including the final view as seen on the customer portal. This applies when comparing the meter with the actual traffic sent to and from the cable modem, which includes some protocol overhead.

If there is a failure in the measurement, recording, processing, or storage of the traffic data, then the overwhelming bias of such failure is to show less traffic than the subscriber actually sent. The system is designed to err towards showing less rather than more traffic than was sent by the subscriber.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:

Copyright © 2009 IDG Communications, Inc.

IT Salary Survey: The results are in