antipaucity

fighting the lack of good ideas

helping a magpierss-powered site perform better

I rely on MagpieRSS to run one of my websites. (If you'd like to see the basic code for the site, see my GitHub profile.)

One of the drawbacks to Magpie, and dynamic websites in general, is they can be bottlenecked by external sources – in the case of Magpie, those sources are the myriad RSS feeds that Datente draws from.

To overcome some of this sluggishness, and to take better advantage of the caching feature of Magpie, I recently started a simple cron job to load every page on the site every X minutes – this refreshes the cache, and helps ensure reader experience is more performant. By scheduling a background refresh of every page, I cut average page load times by nearly a factor of 10! While this is quite dramatic, my worst-performing page was still taking upwards of 10 seconds to load a not-insignificant percentage of the time (sometimes more than a minute!) 🙁

Enter last week's epiphany – since RSS content doesn't change all that often (even crazy-frequent-updating feeds rarely exceed 4 updates per hour), I could take advantage of a "trick", and change the displayed pages to be nearly static (I still have an Amazon sidebar that's dynamically-loaded) – with this stupidly-simple hack, I cut the slowest page load time from ~10-12 seconds to <1: or another 10x improvement!

"What is the 'trick'," you ask? Simple – I copied every page and prefixed it with a short character sequence, and then modified my cron job to still run every X minutes, but now call the "build" pages, redirecting the response (which is a web page, of course) into the "display" pages. In other words, make the display pages static by building them in the background every so often.

If you'd like to see exactly how I'm doing this for one page (the rest are similar), check out this stupidly-short shell script:

(time (/bin/curl -f http://datente.com/genindex.php > ~/temp.out)) 2>&1 | grep real

(The time is in there for my cron reports.)

Compare the run time to the [nearly] static version:

(time (/bin/curl -f http://datente.com/index.php > ~/temp.out)) 2>&1 | grep real

the seven stages of expertise

I recently found The Seven Stages of Expertise in Software Engineering.

  • Stage 1: Innocent
    • barely knowledgeable if at all
  • Stage 2: Exposed
    • seeking knowledge
  • Stage 3: Apprentice
    • has read case studies and tries to apply those techniques
  • Stage 4: Practitioner
    • can actually apply concepts learned in one context to a not-identical context
  • Stage 5: Journeyman
    • professional understanding and application of the field; can mentor
  • Stage 6: Master
    • moved from “whats” and “hows” to “whys”; can mentor very effectively
  • Stage 7: Researcher
    • the teacher, presenter, mentor, speaker, evangelist, writer, authority

Presented firstly in the humorous guise of The Seven Stages of Expertise in Bear Hunting, Meilir Page-Jones makes a highly-compelling case for progressive advancement in [nearly] any field.

Some of the ideas seem similar to what Malcolm Gladwell brings in Outliers (review) or Robert Greene does in Mastery (review). Which seems to only lend more credence to those other works, given that this article is © 1998.

everything with a webui should publish rss

RSS is far from dead – it’s ubiquitous.

What astonishes me, though, is that not all applications that have a WebUI don’t publish feeds via RSS (or Atom – same difference).

OpenNMS and Nagios (via a plugin) will push alerts via RSS – which is fantastic: there’s no reason everyone shouldn’t be able to filter what alerts they look at. I’m sure some other tools will do this, too.

But why don’t all WebUI-based applications support updates and content via RSS? Several of the applications I routinely work with have no possibility of getting data out with an industry-standard format – they use custom APIs (APIs are excellent – and RESTful ones are better, but they’re no RSS).

What benefits could come from every webapp being RSS-enabled? I can think of a few right-off:

  • quick user-by-user customization of content viewing
  • user-preferred interface for content viewing
  • lighter-weight interface for app access
  • quick flexibility

Is you’re developing a webapp, or you’re giving an app a WebUI – make sure you give the ability to get information out via RSS.

don’t implement your scheduler in a pure queue design

Recently came across a seriously funky issue with one of HP’s products (don’t laugh – I know there’s loads of funkiness in HP tools).

HP Cloud Service Automation (3.1) allows you to schedule requests in the future. It also allows you schedule end dates for subscriptions.

That’s neat.

Here’s the problem: if you delete a Service Offering (which allows something to be requested in the CSA catalog) while an active subscription is using it – any active subscriptions get delinked, and become unmanageable with the tool.

Now the dopey scheduler comes into play.

CSA’s scheduler works by putting all future items into a FIFO queue. What this ends up meaning is that if you have an item that fails (because, for example, you’re now past a scheduled end date, but the subscription is unlinked from an offering, so it can’t unprovision it), all other pending items fail, too. Even ones that should happen “immediately” – because “immediately” is still added to the scheduling queue… behind the erroring item(s). And since those items have errored, nothing can move forward.

This is stupid. (And yes – RFEs have already been filed over these problems in the product.)

Instead, have the scheduler put all items into a table – at the appointed time, iterate through the table and run everything you can – if it won’t run, flag it as an error, and move on.

This is how cron works. Why would you not use a commonly-accepted, reliable way of doing things? Oh yeah – you’re HP.

If you’re planning to write something for your product/software/tool – see if anyone else has done it before, and then try to mimic methods that work… please!

finding your niche

“What do you want to be when you grow up?”

You’ve probably been asked that questions hundreds of times in your life – parents, friends, teachers, yourself, movies. It’s a common theme.

For most of us, the decision gets made sometime in our late teens or during college: doctor, mom, lawyer, electrician, plumber, teacher, policeman, musician, actor, soldier, nurse, preacher, engineer, contractor, etc.

But I’d venture to guess that *most* people don’t truly know what they want to do until they’ve been doing something else for a while: I still don’t entirely know what I want to do for a career for the long term – if you’d asked me 5 years ago (as I was in interviews at the end of 2007 – beginning of 2008), I would have said that I wanted to be running a support organization, working towards a professional services operations role. 3 years prior to that, I would have said platform/application architect for flexible large systems. My best guess for what I want to be doing in 2 years now is being an IT/Enterprise data, virtualization, and automation architect for large environments (which happens to line-out with my current title and ‘career path’ with my current employer) – or a US Representative / Senator for my Congressional District / State.

However, the most successful and fulfilled people I’ve met (not necessarily by total ‘wealth’ or accumulated money) have all followed a Blue Ocean Strategy – they’ve invented their own job, or even their own business. That business might not be unique (eg MMM‘s contracting work), but it’s something they’ve decided to do for themselves.

If you’ve not heard of The Personal MBA, you need to learn more about it – start with their list of top business books, and read what you find interesting (and a couple you don’t think would be).

Expand your horizons – browse a good bookstore’s magazine racks, and buy one or two per month that are on topics you know nothing about, don’t like, don’t think you’re interested in, etc.

Visit your local library or bookstore and grab the first book in the history section that starts with an “A” in the title – then go for “B”, “C”, etc. Then do it from some other section of the shelves – maybe relationships, scifi, teen, romance, home improvement, etc, etc.

I am convinced college is not the best path for everyone: there are trade schools, military training, family businesses, farming, etc. I am convinced that going to college straight out of highschool is almost always a Bad Idea™ – at the very least, get a summer job: maybe get a “real” job for a couple years while you figure out what even interests you. Take the ASVAB (Armed Services Vocational Aptitude Battery). Take the ACT (it’s a better predictor of collegiate and work success than the SAT in my opinion). If you’re in college, take the GRE either just after your freshman year, or just before your junior year – the content will be most fresh then (if you’re taking enough general education classes and are not over-focused on your major).

As you start to find out what you’re good at, and what you enjoy to do, do everything you can to improve your communication skills. If you do nothing else at a college, take writing classes – take every class you can that will make you write. Communication is the single most important skill you can have: someone who can write and speak well will go far further than one who can’t. Take public speaking classes. Take classes you need to make presentations for, and follow the 10-20-30 rule. Brevity is highly key, and concision will get you much further than verbosity.

Blog. Blog about what you’re doing, what you’re interested in, what you’d like to do, where you’ve been, etc etc. The more you write, especially if you intend for what you write to be read, the better you will get at it. Aim to write frequently if you’re going to write at all – maybe it’s every first of the month, maybe it’s every Monday, maybe it’s every day, or maybe it’s every 4th of July: but give yourself a schedule and stick to it. Write for personal reasons, write for fun, and write for work.

Teach. When you learn something new, teach it to someone else. Whether you teach by writing, speaking, or showing – teach what you have learned. After communication, the ability to teach someone else to do what you are doing the most important thing you can learn how to do. You never want to become irreplaceable. To be irreplaceable is to be unpromotable. Teach at least one person how to do one aspect of your job as often as possible – spread your responsible skills across your team, and two things happen: first, you can take a vacation; second, you can move up (or out) more easily. The more you teach, like intentionally writing, the better you should get at it – especially if you intend for those you have taught to be able to teach others.

Learn. Strive to learn something new frequently. If you can do it every day, that’s awesome – but just once a month will help keep your mind sharp, and help you become even more valuable to wherever you choose to work (whether it is for someone else, or on your own). Any time your employer wants to pay for training for you, take it – you never know when it may come in handy. I am a proponent of the “Lifelong Learner” – and work to make sure I am learning something new all the time.

Review. Don’t ‘merely’ learn – review what you have learned before. You can do this via blogging and teaching, but take time to reread texts and materials you’re intimately familiar with: this is what David was doing when he wrote, “But his delight is in the law of the Lord, And in His law he meditates day and night.”

No one can ever tell you what your niche is – not really. Maybe you want to be a lawyer – or not. Spend some time to figure out what you’re good at, and what you’d really like to do: shadow people in various careers; interview friends, family, coworkers, classmates, etc.

Life is too short to not try to spend it doing what you want.