Aggregating RSS feeds with Drupal

One of the greatest opportunities available to web-based communities is the ability to share information. All you really need is a set of guidelines for how that information is to be presented, and once you have that, the rest is easy. So easy in fact that it is now possible for you to include news and articles of interest on your site from many well-known sources with just a few clicks.

One of the greatest opportunities available to Web-based communities is the ability to share information. All you really need is a set of guidelines for how that information is to be presented, and once you have that, the rest is easy. So easy in fact that it is now possible for you to include news and articles of interest on your site from many well-known sources with just a few clicks.

What makes it so easy to include other people’s news, documents, articles, or any other content easy is a standard called Rich Site Summary, or Really Simple Syndication (RSS). This allows aggregators (programs that consume RSS feeds) to understand how to present content on Web pages because of the way in which the RSS feed is structured. Drupal comes with one of these aggregators built in — simply enable it in the modules section under administer, and you will find that a few extra menu items pop up, allowing you to administer and view the content once it has been added.

Let’s assume that the demo site would like to provide some news relating to wildlife from the National Geographic Web site. We pop along to the news site at and look for where the RSS feed icons are presented on the site. In this case they look like the following:

Clicking on the RSS link here will bring up the following page — luckily Drupal takes care of all the dirty work for us, and we don’t even need to understand the XML in order to consume this feed:

This may not mean much to you at the moment, but it serves to confirm that there is a feed available for use at the address given in the URL. Knowing this, we can now head over to our site and look for the aggregator link under administer in order to begin adding this feed to the site. This will bring up the as yet blank list of feeds along with everything else we need. On this page, clicking on the add category tab brings up the following form:

Assuming you intend to make use of a variety of feeds, it is probably prudent to categorize your content right from the start. Remember, providing access to timely and interesting news is a value_added service for your site and encourages users to return time and again. With this in mind, it is probably quite valuable to have a variety of relevant feeds available.

Once you are done, click Submit, and then click on the add feed tab to bring up the following:

As you can see, we have:

Specified a title for the new feed

Supplied Drupal with the location of the RSS feed (check this against the URL of the RSS feed page we visited on the actual National Geographic site a bit earlier)

Given an Update interval of one day, making this a daily news feed

Associated this feed with the newly created Commercial category

There are a number of things to consider when filling out this form. First, you need to ensure that you are not infringing any licensing issues that will be supplied by the creator of the source feed. If there are restrictions as to what can and cannot be done with the feed, please ensure you abide by those restrictions.

Next, there is no point in setting an update interval of one hour if you are only running the cron script once a day — the cron can at most only update the script once a day in this case. By the same token, there is no point in using the cron to update your feed every ten minutes if the feed itself is only updated on a weekly basis — check with the providers of the feed how often they recommend you update your feed.

Having taken these factors into account you can now click on Submit to add the feed to your site. Now the aggregator homepage has something to tell us, and should look something like this:

As yet there are no items in the feed because the cron.php script hasn’t been run. You can either wait for it to be accessed by your scheduled task or crontab, or if you are not keen to hang around, simply navigate to http://localhost/drupal/cron.php in your browser (if the path to your cron.php script is different, ensure you change this URL to suit your site). Remember that cron.php is just a script, with no HTML; so you will not see anything displayed on this page even if it is working correctly.

Once you have done that, click refresh on the aggregator page to see the results. You should get something like this, assuming everything has worked correctly:

Success! From this you can see that the feed now contains 9 items as of the last update, which occurred 6 seconds ago. As well as this, you can edit the feed, remove items from the feed, or manually update the feed by clicking update items — in this case, the feed can be modified to reflect any changes on the source site. That’s all there is to it!

You can, of course, now view the content of the feed on the site. For the demo site, permission to access the feeds has been granted to anonymous and authenticated users alike. So when someone visits the site and clicks on the news aggregator link in the main menu, they are presented with something like the following:

Now all users have instant access to all the content provided by National Geographic. Nothing stops you from gathering information from any number of other feeds, and what is interesting is that provided you are not infringing any licenses, you can make your feeds available to other sites. Doing this is easy (remember to enable the syndication block before trying to work with it). Scroll down to the bottom of the page until you see a small icon that looks like this:

Clicking on it brings up the feed page that is provided at the following URL (in the case of the demo site): http://localhost/drupal/aggregator/rss, and in this way content can be syndicated. You aren’t limited to syndicating online feeds you have obtained — any content can be syndicated.

It’s worth noting that your sources can be revealed in the navigation column by clicking on the sources link under news aggregator. In this case, we are presented with the following; note that I have added a feed from the WWF (World Wide Fund for Nature, formerly World Wildlife Fund):

This presents us with a nice summary of feeds from each source. So far so good, but what does the more link do? Clicking on the more link for the WWF source brings up the following page:

From here it is possible to click on the name of the feed in order to go to the source Web site, in this case http://www.panda.org. The feeds associated with this source are listed on the page below the title box for easy access to feeds by source. Clicking on the orange syndicate icon at the top right of the box takes the browser to the actual feed page. In this case it is http://www.panda.org/rss/news.cfm?Press%20Office.

As well as this, there are two tabs at the top of the page. The categorize tab opens up the following screenshot, which allows us to put individual feed items into a variety of categories (assuming you have a variety of categories available).

This screenshot shows the second feed item being assigned two categories, namely Commercial and Special. In this case, the Special category was created with a specific purpose in mind. Because there is news every now and then that warms one’s heart, all items that are tagged in the Special category should be displayed in a special block on the Web site so that every user can see the good news when viewing any page on the site.

The final tab on the sources page simply brings up the same page that was used in order to create the feed. From here you can make any changes you require to the feed with ease.

Since we have not yet covered taxonomy and categorization, you may wish to come back to what follows in this section once you have read Chapter 7 as there may be a few things that are not clear now. For example, we add a module to display content based on its category later on in this chapter, which is why we can work with the Special category in the manner shown next. The information presented here is simply to give you an idea of what you will be able to do.

Remembering that it is entirely possible you would like to display some of the latest feeds in a block somewhere on the site, head on over to the blocks category under the administer link and enable one of the new blocks so that it displays its latest items — notice that it is also possible to display the latest feeds in a category too:

Once this is done, you will see the new feed, along with a selection of its items on your Web pages. You can configure the number of news items displayed in the Block specific settings on its configuration page (click on configure to open this) — for our purposes, the default option of five feeds is just fine.

Now, assuming that a user has blogging permissions and the feed license allows your site to do so, you can add feed items to blogs, by clicking on the little b icon that appears next to each item, as shown here:

Adding a news item to your blogs gives the article more permanence as it will not be lost when or if the list of items in the feed is cleared. Of course, if you are going to build up lots of content over time, then it is highly likely that you will need to make use of the archive module.

The aggregator has one more important section to look at — configuration! If you click on aggregator under the settings link in the main menu, the following page is displayed:

This interface provides control over the type of HTML that is allowed within the feeds that are to be consumed by your site. This should be sufficient for most if not all the feeds you want to aggregate, but on the off chance that there are problems with some feeds, it is likely that they are using tags that are not specified in this section. When in doubt, leave it as is!

Next, the number of items to be shown with each feed in general as well as how long to hold on to old feed items are presented, and it is easy enough to make sensible selections here. The final section stipulates whether the category selection interface (on the categorize tab of the sources page) should use checkboxes or the multiple selector. We have already seen checkboxes in use, and they are fine for the purposes of the demo site. If, however, multiple selector was enabled, the category selections would look like this:

Before we move on, it is worth mentioning again that if at some stage you feel the urge to syndicate your site’s content, you can enable the syndicate block to force Drupal to show the orange syndicate icon. Further, if you would like to enhance the control of the site’s syndication, you can take a look at the syndication module provided.

This article has been adapted from the book Drupal: Creating Blogs, Forums, Portals, and Community Websites by David Mercer.

In addition to the Aggregator module, Chapter 6: Basic Content also mentions Archive, Similar Entries, Taxonomy content-related modules in detail.

For further details, please visit here. To read through summaries of each chapter of the book, you can visit its Free Online Edition here.