Web site policy makers are playing favorites and Google is the big beneficiary, say Penn State researchers.
The research team created a search engine called BotSeer used to examine more than 7,500 Web sites, and found a pro-Google bias in terms of which search engine Web crawlers were or were not allowed access.
Web site administrators use robots.txt files to regulate Web crawlers, also known as spiders and bots, and to help prevent servers from getting overloaded. The researchers found about 4 in 10 sites used robots.txt files, up from 1 in 10 in 1996.
“We expected that robots.txt files would treat all search engines equally or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines’ market share,” said C. Lee Giles, the David Reese Professor of Information Sciences and Technology at Penn State who led the research team that developed BotSeer, in a statement.
“Robots.txt files are written by Web policy makers and administrators who have to intentionally specify Google as the favored search engine,” Giles said.
Yahoo and MSN were also given greater than average access to Web sites, but Google showed up in robots.txt files more than twice as often as either of these competing search sites.
The findings are described in greater detail in the paper “Determining Bias to Search Engines from Robots.txt.”
Other network-oriented Penn State research of late includes efforts to advance photo searching, push copper network links to support higher data rates and better secure databases .
The future of networking as seen through the works of university and other labs.
Our mission is to give you a peek into the future of networking by tracking "alpha" research at university and other labs and at companies based on this work. Your Alpha Doggs are Network World editors Bob Brown, Linda Leung and Neal Weinberg.
|
|
