Fast shows off near real time indexing

Fast Search & Transfer Monday will launch an upgraded version of its Alltheweb.com search engine that offers near real time searches in over 3,000 online news sources.

Entering "Anthrax," for example, can bring up news stories that were updated less than an hour ago from both national and local news organizations from various countries, and in a multitude of languages. The advanced search allows a user to set language preferences and also to pick what sections of the news to search.

Fast has added a new search "vertical," which allows for specific news searches. However, the most recent results from the news vertical are also displayed in a gray box on top of the general search results page.

"We want to show our customers that we can crawl fast; this is a subsecond index," said Rob Rubin, executive vice president and general manager of Fast's Internet division. "We are now continuously spidering and indexing over 3,000 news sources."

Crawling, or spidering, is the process whereby a software agent follows links from one Web page to another and collects information on those pages. That information is then indexed to make it searchable.

Fast sees crawling at high speed as its trademark. The general search index is refreshed every nine to 12 days. Google, Fast's main competitor, reports it refreshes about once a month and AltaVista admits it is months behind with refreshing, resulting in outdated search results.

"The news vertical is a significant freshness improvement," said Rubin.

Other new features on the site, which acts as a showcase for Fast's technology, are linguistic analysis of search queries, expanded customization options and Fast Topics, which uses the Open Directory Project, a human-edited directory run by volunteers, to bundle search results by topic.

Linguistic analysis, or antiphrasing, removes clutter from a search query to deliver better results. The query "What is the weather in Las Vegas?" for example, will be stripped of the "what is" and "Las" will be tied to "Vegas," with "weather" as an extra term.

Although Fast is not a widely known brand, it powers the search engines of Web portal company Terra Lycos, Europe's largest Internet service provider, T-Online International, Europe's second largest ISP Tiscali, eBay and many others.

Fast last month announced an enterprise product, Fast Data Search, which allows large businesses to search the content on their own servers. Fast Data Search has been integrated, for example, with BroadVision's enterprise self-service applications.

The IDG News Service is a Network World affiliate.