The Search engine ranking game

Mark Gibbs' Web site tips, plus network applications news headlines

When the marketing department confides in you over a martini or two that they have spent a gazillion dollars with search engine optimization (SEO) companies and have no idea what kind of bang for the buck they have got, you can casually mention that it's easy to find out. You might also like to tell them that you'll assign one of your best people to the task.

Now you can figure out rankings the hard way: Just go to as many of search engines as you can and enter the search term that matters to your marketing people and then read the results to see how far down the results list the URL you care about comes.

For example, on Google, the term "Web applications" places the archive for this newsletter (http://www.nwfusion.com/newsletters/web/) with a ranking of 53 while on Yahoo! we're number 45.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

When the marketing department confides in you over a martini or two that they have spent a gazillion dollars with search engine optimization (SEO) companies and have no idea what kind of bang for the buck they have got, you can casually mention that it's easy to find out. You might also like to tell them that you'll assign one of your best people to the task.

Now you can figure out rankings the hard way: Just go to as many of search engines as you can and enter the search term that matters to your marketing people and then read the results to see how far down the results list the URL you care about comes.

For example, on Google, the term "Web applications" places the archive for this newsletter (http://www.nwfusion.com/newsletters/web/) with a ranking of 53 while on Yahoo! we're number 45.

Obviously, this is a task that demands automation if you plan to check for any significant number of search terms and or with any frequency. But it's not a simple task if only because the people who own the search engines don't like you doing this too often - it skews their statistics and adds extra load on the servers.

If a search engine company detects automated scanning of their services for any reason they are likely to take action (see Google's service terms). They could block the source IP address, (or do other things such as "throttle down" the data delivery rate to suspected scanners but, not surprisingly, they don't discuss things like this in public).

So to get away with automated page rank checking people have to get devious. For example, AADsoft.com (see links below) offer AgentWebRanking which goes to each search engine and behaves like a human being doing a search even though it is checking a company's ranking position for a search term. The company claims that this is the only utility of this type to behave in this way.

I'm guessing that it manages the number of connections it makes, presents a "normal" browser agent string, and interleaves requests for other content to reduce the chance of triggering a behavior pattern detection system at the search engine's end.

AgentWebRanking provides customized search engine ranking report in HTML, XML, Word and CSV formats; supports English, French, Dutch, Deutsch, Spanish, Italian, Danish, Portuguese, Norwegian, Swedish and Finnish character sets and interfaces; and includes regular updates on search engine configurations to ensure continuous operation.

All in all, it is an arms race with continuous escalation with the search engine marketing companies trying to get what they want from the search engines without getting caught.

AgentWebRanking works with major search engines and directories in the U.S., Canada, France, U.K., Germany, Ireland, Holland, Italy, Switzerland, Austria, Belgium, Spain, Portugal, Denmark, Finland, Norway, Australia, New Zealand and Sweden. The product costs $150 per year per license for the business edition and $500 per year per license for the professional edition.

P.S. Another search engine ranking resource worth checking out is Bruce Clay, LLC. On this site there's a lot of interesting and useful information on the search engine optimization world.

Read more about software in Network World's Software section.

Mark Gibbs is a consultant, author, journalist, columnist and blogger.