About those polls . . .

Online opinion polls suck. They suck for one simple reason: The respondents are self-selected, so there's no way to know how the results might compare with a statistically valid survey of randomly chosen people.

Online opinion polls suck.

They suck for one simple reason: The respondents are self-selected, so there's no way to know how the results might compare with a statistically valid survey of randomly chosen people.

So the question before us today is not whether online opinion polls suck or not: They do. The question we're here to address is whether such polls are literally worthless . . . or just practically worthless.

That thin distinction matters, at least to me, because of something written here June 20. That column cited an online survey - despite the fact that they suck - to help buttress my contention that companies should be required by law to report extortion attempts against their networks. Conducted on our Web site, the survey tallied 51 votes in favor of a legal mandate and only three opposed. After making the requisite apologies for the suckiness of online surveys, I wrote: "Self-selected sample aside, that's an unmistakable consensus."

That statement prompted this reasonable complaint from a reader, Ross McKenzie: "An exceedingly small, self-selected sample from the Web is an unmistakable consensus? I would say that any and all conclusions from faulty data are faulty, notwithstanding how good a story it makes."

I did mention that online polls suck, right? But was even this 51-to-3 result literally worthless, as the reader suggests?

OK, I'm trying to wiggle off the hook here. Toward that end, I enlisted the advice of a pair of academics: Adam Berinsky, an associate professor of political science at MIT, and Jordan Ellenberg, an assistant professor of mathematics at Princeton. (It's another exceedingly small sample, but at least they didn't select themselves.)

"I'm afraid that I can't give you good news," Berinsky told me. "Without some adjustment for the nonrandom nature of your sample, you can't make any inferences about public opinion. Some companies, such as Harris Interactive and Polimetrix, are working on ways to adjust self-selected interviews, but the jury is still out."

"In your case, I would need to know more, but if you simply asked people to take an online poll, I'm afraid that your critic is correct."

Case closed . . . and the columnist can forget about getting into MIT, right?

Maybe, but Princeton's not a bad backup, and they seem to grade journalists more generously there.

"The only reason to use weak data like that is if stronger data is not available," Ellenberg first cautioned me. "It would be better to e-mail 10 people you consider trustworthy on this subject and see how many say 'yes,' than to use an online survey of 50 people. But I think that as a journalist you are OK as long as you make clear what the methodology is.

"You should . . . say that in your opinion this [51-3 result] means something. I would tend to agree that it means something - but when I say that, I am speaking just as a person, not a mathematician. Because the question really isn't a mathematical one."

That sounds like a gentleman's C . . . and I'll take it.

Here's why news organizations of all kinds sprinkle their Web sites with self-selected opinion polls, despite the fact that even journalists understand that such polls are unreliable: Readers enjoy them. They're fodder for discussion, and, as an editor here put it not long ago, "People just like to click on buttons."

Here's why I used this particular poll result: 51 to 3 is 51 to 3, maybe not to a mathematician, but to most everyone else. That's 94% to 6%.

And it turns out that even a mathematician can be tempted by 51 to 3. I asked MIT's Berinsky if he'd take into consideration the 51 to 3 data point if we were to conduct a random survey on the same question and then bet on the outcome.

"Well, if I were to bet, I might go with the 94% side, but the problem is that you can only say that 94% of the people who took the survey supported your point of view," Berinsky says. "We can't draw any inferences from self-selected polls, no matter how lopsided the split."

The mathematician wasn't budging, but we see where the man was going to put his money.

Want your own grade? The address is buzz@nww.com.

Copyright © 2005 IDG Communications, Inc.

The 10 most powerful companies in enterprise networking 2022