Putting Yahoo Pipes to work

Last week I began discussing Yahoo Pipes, a service provided by (as you might have guessed) Yahoo. This service is designed to route and transform data taken from one or more online sources and generate an output in RSS 2.0, RSS 1.0 (RDF), JSON and Atom formats.

I discussed the range of input source choices (generically referred to as "feeds") for Pipes, now I want to cover what you can do with the data.

Pipes provides a number of operator modules that do things such as counting the number of items in a feed, filtering out feed items according to string matching rules, removing items with duplicate strings, performing a Regular Expression operation on feed items and creating RSS feeds.

There are also modules for building URLs, manipulating strings (including one that extracts the linguistically most important terms from feeds and another that will translate items into other languages which will surely annoy the American Translation Association).

As if that weren't enough, there are also modules to define and format dates, convert text strings to geographic locations and work with numbers.

As I mentioned last week, you create a Pipe using the Pipes Web-based IDE by dragging modules onto the canvas, setting any parameters the modules might have, and then "wiring" them together by dragging from the output of one module to the input of the next. To test the Pipe you just click on Refresh at the bottom of the IDE screen and the Pipe's results are displayed.

So, what can you use the Pipes service for? You might want to use Pipes, for example, to filter the flood of IT news and track the main topics at the top of the industry news.

I created a pipe (after much wrangling; the Pipes documentation could be better) that starts by ingesting the Network World Netflash feed using the Fetch Data module (I couldn't use the Fetch Feed module as the Network World Feed is in XML format). I had to give the path or branch of the XML data I wanted to analyze, which the Netflash feed names "rss:item". The pipe then sends the item branches to a Loop module that, in turn, contains a Term Extractor module. The Loop module is configured to extract the data from the child nodes named "rss:description" which the Term Extractor then strips down to just the significant phrases.