FeedTree version 0.7.2 is available for download. It is strongly recommended that all users update to the latest version, as it includes a number of fixes which improve the stability of the network (and which work best when more of the network is running the fixed code).

In this release:

• Improvements to bootstrap selection, reducing the likelihood of “all bootstraps are faulty” messages: [184]
• Fixes to the FreePastry p2p substrate: [188] [192]
• Minor bug fixes and tweaks (see the project timeline for a complete list).

Version 0.7.1 of the FeedTree proxy and publisher is out (download). This is a bugfix release containing improvements to the FreePastry layer (changes [179] and [180]) and the way FeedTree uses it [178].

For users with multiple network interfaces (and therefore multiple IP addresses on their local network), the command-line ftproxy client now supports the -I flag, allowing you to specify which address FeedTree should use [173]. When debugging connection issues, ftproxy also allows you to see Pastry networking traces (flag -D) and to save all that logging info to a file (flag -L) [176].

Bugs confirmed fixed since 0.7.0: #37 #38 #42.

The key algorithm that makes FeedTree possible is the Scribe peer-to-peer multicast system. Scribe trees are each organized around a topic, in this case, the URL of a feed; each subscriber in the Pastry network routes a message incrementally toward the node nearest the topic, and the union of all those routes (with arrows reversed!) becomes an effective multicast tree.

An example of a lovely Scribe tree, taken from current network statistics, is the current multicast tree for Digg’s main feed:

It’s not exactly a balanced tree, because it’s constructed organically from Pastry routes. But it’s effective: it’s cheap to maintain given the Pastry infrastructure, and it fans out quickly, so updates propagate rapidly from the root down to every subscriber. And because there are at least 3 nodes polling Digg’s feed at the moment, there are new stories for all FeedTree proxy users once every 10 minutes or so. (That’s a lot to digg!)

Key:

• rectangle = Scribe topic (SHA-1 hash of feed URL)
• oval = subscriber
• dotted oval = helper (non-subscribing member of tree)

Welcome to everyone visiting from Slashdot! The story generated some good discussion, including

• FT isn’t BitTorrent (but how they might work together)
• why it’s OK that there’s some conceptual overlap with NNTP
• a good response to a reasonable question: what’s the problem, exactly?

It wasn’t really the bump in users I was hoping for; at our peak this afternoon we had about 35 nodes, and we’re currently back down to 30 or so. This is still a huge increase from what we had before, and the Scribe trees seem to be operating well. (Those of you subscribed to this feed ought to see a prompt update as soon as this entry is posted!)

The power law is in full effect, judging by my network stats; it appears that there are twice as many users subscribed to Slashdot’s feed as the next most-popular feed, and there are a whole lot of feeds with just one subscriber.

Here’s a little bonus for all the new users out there: a few of those web badges that seem to be so enduringly popular. If you’re pushing RSS or Atom updates via FeedTree, drop in a icon to let your users know they can get faster updates (and save you bandwidth).

Version 0.7.0 of the FeedTree software for users and publishers is now available for download. This version collects a number of bug fixes and new features; a complete list of checkins is here. Some highlights:

• Proxy: New statistics and info in the web UI (screenshots)
• Proxy: Support for conditional HTTP GET for feeds being polled
• Proxy: All feeds are now formatted as RSS 2.0 on the client side for better compatibility with feed reading apps (this might be switched back to Atom 1.0 as adoption and toolkit support improve)
• Proxy: More robust handling of content feed elements, so that RSS 2.0 fulltext feeds aren’t truncated for proxy users
• Publisher: New one-time configuration script, as well as /etc/init.d-style run control scripts for Linux installations
• Publisher: New, more flexible format for publisher.conf (based on JSON, see [141])
• Publisher: Debian package installation option (see downloads page)

(It is strongly recommended that all current users download the new version; please report any issues to the mailing list or the developers.)

It’s worth noting that the website has received some attention as well; the new FeedTree for users and for publishers pages give a high-level schematic overview of the problems FeedTree solves.

Above: Diagrams from the new introduction pages, inspired by the original FeedTree poster.

I’ve been playing around with ways to take advantage of FeedTree’s prompt feed updates. On my desktop, where I run the FeedTree client proxy, I get new links from the Digg diggall feed and the Reddit new links feed every 6 minutes on average. These feeds are a firehose of links that are about to become hot; between the two, there are about 60 new URLs posted per hour. With FeedTree it’s actually possible to keep on top of this stuff as soon as it’s “discovered”.

It turns out I don’t always have time to read these feeds and hit all those URLs immediately. (OK, I rarely have time.) I typically wait for popular links to bubble up to the frontpages of Digg and Reddit, by which time they’re often unavailable, their webservers slaughtered by hordes of Digg readers.

It occurred to me that a program which listens to FeedTree (for really up-to-the-minute feed updates) and automatically populates the CoralCDN distributed Web cache. That way, Coral can snapshot the pages before they’re swamped, and by the time I get around to seeing these links, there’ll be a Coralized version (thanks, Mike!) all ready to go.

So I wrote Calcium, a Python program that does exactly that. It’s running on my desktop, passing new soon-to-be-hot links through FeedTree and over to Coral. Now everyone will benefit from automatically pre-Coralized Digg and Reddit links!

(If you want to play with it, grab a copy from the Subversion browser or right out of the repository.)

This is pretty exciting! WordPress users who want to push out their blog updates with FeedTree now have a two-step process:

1. Install the FeedTree publishing tool. (This is easier than ever now that there’s a helpful configuration script.)
2. Install the new FeedTree plugin for WordPress. Tell the plugin the URL to your certificate and the certificate’s fingerprint, and it takes care of tweaking your RSS2 and Atom feeds to add the appropriate FeedTree tags.

Read the rest of this entry »

If you’ve been meaning to try pushing out feed updates but haven’t figured out the configuration, there’s a new configure-publisher console tool that will walk you through it. It asks you a few questions, calls keytool and openssl to do the crypto for you, and then emits a publisher.conf that’s ready to use with the FeedTree publishing tool.

You can read more about the configuration tool on the FeedTreePublisher wiki page, and you can fetch the latest version of the configuration tool directly from the Subversion repository.

By installing the publisher, you’ll ensure that FeedTree users see new feed entries the minute you publish them. By signing your feed, you ensure that those rapid updates can be verified to be authentic. Give it a try, and let us know if you have any problems by posting to the users list.

FeedTree is a research project, and all research projects need data: preferably lots and lots of it. Each FeedTree node sends a small amount of anonymized data back to a statistics server; it’s enough to reconstruct the Scribe multicast trees that distribute new feed events. Since the FT network is still pretty small, these “trees” are usually more like sprouts, but I noticed this morning that the Reddit tree looked pretty interesting. So here’s a snapshot (PDF) of the current Scribe tree for the main Reddit RSS feed. As the size of the network increases, these trees (which are constructed organically as new subscribers join the group) will look more and more interesting.

Version 0.6.2 of the FeedTree client software is now available for Windows, Mac, Linux, and any other Java 1.4/1.5 platform. Many minor fixes, including the elimination of a large memory leak (oops!) and an update to the latest version of FreePastry (which itself contains a number of performance-related improvements).

In addition to the new release, I’ve added a few new documents to the wiki, including improved FeedTree proxy documentation, a slightly expanded FAQ, and everyone’s favorite: screenshots!.

Direct 0.6.2 download links: Windows / Mac OS X / JAR and publisher tool