Vertical search engines and what Google can’t do

There is no doubt that when it comes to Web searching, Google is the 800-pound gorilla. But before you assume that Google and the other big boys are (at least at present) the only players, consider that there’s a bigger ‘Net that Google and the others can’t index and that generality isn’t the only need.

By my first remark, I am referring to what is called the “hidden” or “deep” Web, the stuff that isn’t in Google’s database. A Google press release earlier this year (see links below) stated that the service indexes some 6 billion items consisting of 4.28 billion Web pages, 880 million images, 845 million Usenet messages, and a growing collection of book-related information pages (up from 3 billion this time last year).

According to various estimates, Google’s Web page index is estimated to be indexing, at best, only about 30% of the Internet’s total HTML content, while the other estimates put that nearer to 10%. For the other types of content Google may be only indexing a fraction of what is available.

As for the generality issue, Google is good enough for everyday purposes but as your search interests become narrower and more niche-oriented, Google is unlikely to do more than scratch the surface. Another problem is that even quite restrictive search criteria can return tens of thousands of hits making it hard to identify relevant content.

This is where “vertical” search engines come in. These are search engines that focus on a specific topic, industry, or type of content with the intention of providing greater relevance and additional functionality than the “horizontal” search engines.

I recently received some data about the success of GlobalSpec since it launched an expanded Engineering Web search engine some 100 days ago:

* More than 1.1 million users have registered to use GlobalSpec.

* Almost 15,000 new users sign up for free each week – a jump of 50% in four months.

* GlobalSpec provides information on 66 million searchable parts through 23 million searchable pages.

* GlobalSpec has indexed more than 100,000 engineering-only sites, providing access to what it calls the “Hidden Web” of targeted engineering-specific information.

* More than 1.1 million searchable product families are available through GlobalSpec from 51,500 suppliers.

The Globalspec database is not only very large, it is also very structured focusing on vendor product specs for specific hierarchical product categories (for example, “Sensors, Transducers and Detectors > Acceleration and Vibration Sensing > Accelerometers”), links to vendor Web site, automated Requests for Quotes, and vendor e-mail contact.

Of course Google isn’t standing still as illustrated by the launch of Froogle, its Web shopping product search engine. While Froogle is not as narrowly focused as GlobalSpec, it is a serious step towards verticality. How vertical can Google go? As far as it wants but I would suspect that the company knows what its core competencies are and it may well feel, to quote science fiction author Robert Heinlein, that, “specialization is for insects.”

Even so, what GlobalSpec demonstrates is that the Web search business is by no means locked down by Google and the other big players and that successful business models can be built on niche search engines even if you are, at least compared to Google, sort of like an insect.