If everyone started watching "24" or "CSI" on video iPods or streamed over the Internet - instead of on TV in their living rooms - these top-rated shows would probably go the way of "Cop Rock."
That is, Jack Bauer would become the victim of, not terrorists, but bad ratings. This is because Nielsen Media Research cannot collect data about what people watch on handheld video-viewing gadgets or from PCs streaming network TV shows. While Nielsen estimates around 90% of TV viewing still happens in homes, it's this burgeoning 10% that TV networks and advertisers are desperate to delve into.
"The industry is just dehydrated for this data," says Bob Luff, Nielsen's CTO, who oversees the design of the technology used to collect and measure TV viewing habits in more than 40,000 Nielsen homes. "But how do you measure what's on a video iPod at 38,000 feet on an airplane's fold-down tray?"
The demand for this new data is changing how Nielsen collects and processes its ratings numbers, which each year drive TV advertising and programming worth $70 billion and determine what soars and bombs in show business.
Over the years, keeping up with video technology has been a challenge for Nielsen, Luff says. The company's traditional technology for tracking TV viewing involved set-top boxes - called People Meters - which recorded what channels were watched and when. This technology used physical taps and probes wired into a TV or cable box to read channel numbers or the UHF/VHF frequency of the set. Analog modems in the back of the Nielsen People Meter provided nightly phone calls to a data center where information on viewing events - every minute of TV watched, including channel changes - was deposited at 2 a.m.
The increasing complexity of home electronics, and new technologies such as time-shifted viewing via TiVo, forced Nielsen to come up with new ways to identify what people were watching, and when. Nielsen worked with TV broadcasters to invent two technologies that let its meters recognize when a show is playing, based on signals hidden in the audio. One method, called psychoacoustic encoding (see story), injects a digital time stamp and program title - or "active signature" - into the audio tracks of TV shows as they are broadcast. Another technique, called passive signatures, creates a kind of audio fingerprint for TV shows; a split-second sample of audio is digitized, creating a unique signature, which also can be recognized by metering equipment.
These two types of signatures will also allow Nielsen to tap into the viewing habits of mobile video users, or even those watching regular TV outside the home.
One gadget in the works is the Go Meter, which some Nielsen participants next year will begin to carry with them throughout the day.
"If you think [iPod] Nano, you're pretty close to what the Nielsen Go Meter will look like," Luff says. The device samples the ambient audio surrounding the user and searches for the psychoacoustic codes embedded in TV programs. Recognized signatures are logged and stored in flash memory in the Go Meter. The gadget's docking station - similar to a PDA or iPod cradle - recharges it at night and uploads the day's codes during the 2 a.m. data deluge via a modem.
A smaller monitoring device is in the works for video iPods or smart phones that can play video. The Solo Meter - also due to hit next year - is about the size of a triple-A battery, and sits between the earphone jack on an audio device, and the headphone jack. The Solo Meter intercepts audio between the iPod line-out jack and the head phones, and stores only the audio signatures that are identified in flash memory.
In measuring eyeballs glued to cell phone video screens, Nielsen is eliminating even the cradles that power up and download data from devices. Software developed by the company that runs on smart phones lets the devices call Nielsen's data center directly over the cellular network, delivering data on what the user watched while riding the bus or waiting in line at the bank that day.
"If we can't measure all video watched in a household, then we have to abandon that household," Luff says. The theory goes like this: If dad, mom and sister are watching ABC, FOX and CBS on separate TVs in a house, but brother is watching streamed NBC shows on the Web, the data for that family would not be valid.
To meter streamed shows on the Internet, Nielsen has developed a software agent that would be downloaded onto Nielsen ratings participants' PCs and would report to Nielsen the streamed network TV content watched on that PC. Luff says this could extend someday to nontraditional streamed video, such as content from YouTube.
Nielsen also is starting to tap into video on demand (VOD) servers that cable and phone providers are installing to provide on-demand programming to subscribers. Because VOD programs are not watched on a particular schedule, and the content contains no active or passive audio codes recognizable by Nielsen's gear, the company created software that runs on VOD servers on providers' networks, which send logs to Nielsen's data center containing what was watched and when.
Data center growth
Winter Corp., a research firm that surveys the largest databases in the world, usually ranks Nielsen's database as one of the 10 largest. Nielsen's headquarters in New York is the central repository for viewing information sent every night by hundreds of Nielsen devices in the field. Nielsen's main database - which contains more than 20TB of TV viewing data for the last five years - grows each evening by the approximately 40GB of data collected from Nielsen family households. "If you can remember a particular 'Star Trek' episode from five years ago, we can find out how many people who were pet owners at the time were watching," says Kim Ross, Nielsen's CIO. (Nielsen's offline archives provide TV viewing data back to the "Howdy Doody" era, he adds).
The national data is collected primarily via a bank of 1,000 analog modems in the data center. Devices in Nielsen homes across the country call in to the center at 2 a.m. in their respective time zones. The data is collected by 6 a.m., which gives the staff of analysts and programmers just a few hours to compile and make it into a deliverable report for Nielsen clients.
Besides compiling the raw data, Nielsen collects the broadcast logs of all major TV networks, as well as the logs from more than 13,000 cable TV head-ends nationwide. To complete the picture, this data must be crossed-referenced with incoming data from Nielsen monitoring devices. A staff of more than 1,300 does this every morning.
As impressive as Nielsen's database, computing infrastructure and analyst manpower may seem, Ross feels the company is on the verge of a processing and storage shortfall. The mobile metering projects being dreamed up by CTO Luff's team would generate a flood of data that soon may be pouring into Nielsen's data center.
"Today the ratings calculations run on the mainframe, and we've had no reason to move them," Ross says. "But these new projects could multiply our applications and data, and maybe drive it up 10-fold. One of the places we would run out of gas real soon, or drive up our cost quickly, is in that calculation area."
One way Ross is preparing for this is by pulling some of the blade server-based distributed processing it uses in other parts of the business into the core processing role. At the edge of its network, Nielsen uses IBM blade servers running Linux and Business Analytics software. This platform is used by advertisers and TV networks for running analysis and reporting.
The distributed Linux blades ended up being so fast, Ross says, his server staff plans to install a system of multiple blades to augment the mainframe processing. Parts of Nielsen's internal data analysis run staged as Java routines and distributed across 10 Linux blades.
"This will give us that Google-style scalability," Ross says. "If you set out to take every viewing event, just from every set-top box, this alone could drive up our processing and storage needs a thousand times," Ross says.
Got a suggestion for a Wider Net story? An offbeat network industry-related topic? A fascinating personality we should profile? Contact Bob Brown with your ideas at firstname.lastname@example.org.
Learn more about this topicPsychoacoustic coding: What Nielsen uses to track TV viewing10/09/06CEATEC - Toshiba, Canon show 55-inch SED television
10/03/06Nielsen: half of all searches used Google in August