antipaucity

fighting the lack of good ideas

helping a magpierss-powered site perform better

I rely on MagpieRSS to run one of my websites. (If you'd like to see the basic code for the site, see my GitHub profile.)

One of the drawbacks to Magpie, and dynamic websites in general, is they can be bottlenecked by external sources – in the case of Magpie, those sources are the myriad RSS feeds that Datente draws from.

To overcome some of this sluggishness, and to take better advantage of the caching feature of Magpie, I recently started a simple cron job to load every page on the site every X minutes – this refreshes the cache, and helps ensure reader experience is more performant. By scheduling a background refresh of every page, I cut average page load times by nearly a factor of 10! While this is quite dramatic, my worst-performing page was still taking upwards of 10 seconds to load a not-insignificant percentage of the time (sometimes more than a minute!) 🙁

Enter last week's epiphany – since RSS content doesn't change all that often (even crazy-frequent-updating feeds rarely exceed 4 updates per hour), I could take advantage of a "trick", and change the displayed pages to be nearly static (I still have an Amazon sidebar that's dynamically-loaded) – with this stupidly-simple hack, I cut the slowest page load time from ~10-12 seconds to <1: or another 10x improvement!

"What is the 'trick'," you ask? Simple – I copied every page and prefixed it with a short character sequence, and then modified my cron job to still run every X minutes, but now call the "build" pages, redirecting the response (which is a web page, of course) into the "display" pages. In other words, make the display pages static by building them in the background every so often.

If you'd like to see exactly how I'm doing this for one page (the rest are similar), check out this stupidly-short shell script:

(time (/bin/curl -f http://datente.com/genindex.php > ~/temp.out)) 2>&1 | grep real

(The time is in there for my cron reports.)

Compare the run time to the [nearly] static version:

(time (/bin/curl -f http://datente.com/index.php > ~/temp.out)) 2>&1 | grep real

ifttt & box drive my desktop backgrounds … with a little cron happiness

I love that OS X lets me change my background on a schedule (I use every 30 minutes now).

But I don’t like having to find pictures to populate my desktop menagerie with.

Enter completely SFW backgrounds via RSS feeds!

Using IFTTT, I watch for new items from a variety of daily photo feeds, and upload the new items to a folder in my Box account. I have that folder set to be the source for my desktop backgrounds, and bingo bango we have automated new images coming to enjoy!

The recipe I’m using is available for you to grab here. (I have several running, but you can use any RSS feed you’d like.)

Also, to ensure I don’t end up with duplicate images (eg from the Bing images feed), I have the following running as a cron job (thanks to Unix.SE for helping me figure it out):

md5 -r * | sort | awk 'BEGIN{lasthash = ""} $1 == lasthash {print $2} {lasthash = $1}' | xargs rm

That script removes any files with duplicate MD5 sums from the folder I keep the images in (note – you should put the actual path to your folder in your cron job).

everything with a webui should publish rss

RSS is far from dead – it’s ubiquitous.

What astonishes me, though, is that not all applications that have a WebUI don’t publish feeds via RSS (or Atom – same difference).

OpenNMS and Nagios (via a plugin) will push alerts via RSS – which is fantastic: there’s no reason everyone shouldn’t be able to filter what alerts they look at. I’m sure some other tools will do this, too.

But why don’t all WebUI-based applications support updates and content via RSS? Several of the applications I routinely work with have no possibility of getting data out with an industry-standard format – they use custom APIs (APIs are excellent – and RESTful ones are better, but they’re no RSS).

What benefits could come from every webapp being RSS-enabled? I can think of a few right-off:

  • quick user-by-user customization of content viewing
  • user-preferred interface for content viewing
  • lighter-weight interface for app access
  • quick flexibility

Is you’re developing a webapp, or you’re giving an app a WebUI – make sure you give the ability to get information out via RSS.