Reference: Scalability of Syndication Feeds

I do mean to pick on the general notion that the Web can’t scale enough for syndication’s purposes; the Web provably does scale, and like gangbusters.

"Our hourly RSS surge has all the characteristics of a distributed DoS attack [...]"

Is anybody working on a solution?

Because Bloglines has a vested interest in increasing RSS (in the generic sense) adoption, we're looking at ways we can help. We are working on a couple of projects right now, and we're of course open to suggestions.

Jeremy Zawodny: "Thoughts and ideas out there?"

Something that came up as a partial solution at one of the blogger meetups was the HTTP delta proposal.

one of the proposed solutions is to build on top of a protocol that fits the syndication model better than HTTP. one called NNTP already exists and is in widespread use as the foundation for network newsgroups

We've seen similiar problems over the years. RSS (or as it should be called, "Speedfeed") is such a useful thing, it's unfortunate that it's ultimately just very stupid.

"Once all the covered wagons show up and an RSS reader is folded into Microsoft's Internet Explorer or Outlook, we're doomed," Murphy said.

The functionality is clearly there in HTTP. The word is clearly not getting out to everywhere it should be.

For any technology, the statement “X doesn’t scale” is a myth.

RSS is broken, is what happened. It's not scalable when 10s of thousands of people start subscribing to thousands of separate RSS feeds and start pulling down those feeds every few minutes

Whatever duct tape you put on this problem (e.g. make the subscribers poll less frequently) will get ripped off again when the growth to the next order of magnitude will happen. And if atom/rss is indeed such a great thing, this growth is indeed something that will happen.

Now, instead of thousands of individual desktop PCs independently scanning news sources, blogs and web sites for updated feeds, Bloglines will make low-bandwidth requests to each site on behalf of the universe of subscribers and cache any updates to its master web database. Bloglines will then redistribute the latest content to all the individuals subscribed to those feeds via the linked desktop applications.