Data-mining Usenet

Now, no wisecracks about how data-mining the Internet's oldest public space would mean coming up with a mountain of X-rated JPEGs and make-money-fast spams.

Marc Smith, a research sociologist at Microsoft, has begun looking at ways of extracting trend data and other information from the network. His Netscan software sucks in the messages from 50,000 or so Usenet newsgroups and then analyzes them every which way, including average number of posts, the size of the posts, cross-linked newsgroups, etc., etc (the software is mounted on his site, so you can play with the info yourself).

The goal of the Netscan project is to collect base-line measures of the Usenet, its structure and dynamics so as to map of the kinds and qualities of the groups and institutions that form when people use the net to interact with one another. Netscan provides a range of measures of activity in the Usenet including the number of messages in each of the groups studied and the number of people who participate in them. This can reveal some interesting patterns when this data is analyzed over a period of hours, days, weeks or longer. Other network media like email lists, chat rooms, and proprietary discussion systems could also be studied in this way.

Those of you a little wary of anything in which "Microsoft" and "personal data" are mentioned in the same sentence might not be thrilled by this statement from the project FAQ:

The ultimate goal is to shed light on the vast invisible continent of social cyberspace and to see the crowds that are gathered there.

Because while sociology is the study of groups, it doesn't take too much imagination to figure out how something like this could be used to track specific individuals - or at least, the online names of individuals. Used properly, something like this might be helpful in law enforcement, but even there, recent news would give one pause.

Via Anil Dash.

Contact Online Executive Editor Adam Gaffin

Archives