Making dynamic content indexable

If your Web site is to be found it must be findable which means it must be indexed. Assuming that you’ve done all the right things to ensure that search engines will find you and index you correctly (meta tags, etc.) there’s one other thing you might want to fix: your dynamic content.

The problem is that when a search engine’s spider finds a URL that includes an HTTP “get” request (that is, it has a “?” in it) it will ignore variable data – that’s everything from the question mark onwards.

For example:

http://www.mywebstore.com/product.asp?pid=20001 – The variable part of this request is the data following the question mark – that’s “pid=20001”.

For search engines this is a good strategy as it avoids the situation where a site providing dynamic content could deliver an unlimited number of pages programmatically (the same problem doesn’t exist with “post” requests because the variable request data can’t be embedded in a regular URL).

So if you want to have your dynamically generated product catalog indexed you’re going to have to be cunning. If you use ASP here’s a tool to get your site indexed: AlphaSierraPapa’s PortalPageFilter (see links below).

PortalPageFilter reformats your links so that the example above would read:

http://www.mywebstore.com/PPF/pid/20001/product.asp

That URL is now fully indexable by spiders although you should note that Google will accept only two documents per site per day, AltaVista three, while Inktomi will greedily consume up to 300.

AlphaSierraPapa says that with PortalPageFilter “All you have to do is call a function provided by us to generate the URL on the referring page, but the calling page does not need to be modified – PortalPageFilter presents a standard query string, just as it was before!”

Note that these translations aren’t done on the fly – see the link below “Automatically rewriting URLs” for the explanation why it is not practical.

An unlimited time demo version is available that is restricted to documents in a specific directory. The full version for a single Web site is $125 and the price for unlimited sites on a single server it is $350.