Category Archives: insights

what level of abstraction is appropriate?

Every day we all work at multiple levels of abstraction.

Perhaps this XKCD comic sums it up best:

If I'm such a god, why isn't Maru *my* cat?
abstraction

But unless you’re weird and think about these kinds of things (like I do), you probably just run through your life happily interacting at whatever level seems most appropriate at the time.

Most drivers, for example, don’t think about the abstraction they use to interact with their car. Pretty much every car follows the same procedure for starting, shifting into gear, steering, and accelerating/decelerating: you insert a key (or have a fob), turn it (or push a button), move the drive mode selection stick (gear shift, knob, etc), turn a steering wheel, and use the gas or brake pedals.

But that’s not really how you start a car. It’s not really how you select drive mode. It’s not really how you steer, etc.

But it’s a convenient, abstract interface to operate a car. It is one which allows you to adapt rapidly to different vehicles from different manufacturers which operate under the hood* in potentially very different ways.

The problem with any form of abstraction is that it’s just a summary – an interface – to whatever it is trying to abstract away. And sometimes those interfaces leak. You turn the key in your car and it doesn’t start. Crud. What did I forget to do, or is the car broken? Did I depress the brake and clutch pedal? Is it in Park? Did I make sure to not leave the lights on overnight? Did the starter motor seize? Is there gas in the tank? Did the fuel pump quit? These are all thoughts that might run through your mind (hopefully in decreasing likelihood of probability/severity) when the simple act of turning the key doesn’t work like you expect.

For a typical computer user, the only time they’ll even begin to care about how their system really works is when they try to do something they expect it to do … and it doesn’t. Just like drivers don’t think about their cars’ need for the fuel injector system to make minute adjustments thousands of times per second, most people don’t think about what it actually takes to go from typing “www.google.com” in their browser bar to getting the website returned (or how their computer goes from off to “ready to use” after pushing the power button).

Automation provides an abstraction to manual processes (be it furniture making or tier 1 operations run book scenarios). And abstractions are good things .. except when they leak (or outright break).

Depending on your level of engagement, the abstraction you need to work with will differ – but knowing that you’re at some level of abstraction (and, ideally, which level) is vital to being the most effective at whatever your role is.

I was asked recently how a presentation on the benefits of automation would vary based on audience. The possible audiences given in the question were: engineer, manager, & CIO. And I realized that when I’ve been asked questions like this before, I’ve never answered them wrong, but I’ve answered them very inefficiently: I have never used the level of abstraction to solve the general case of what this question is really getting at. The question is not about whether or not you’re comfortable speaker to any given “level” of customer representative (though it’s important). It is not about verifying you’re not lying about your work history (though also important).

No. That question is about finding out if you really know how to abstract to the proper level (in leakier fashions as you go upwards assumed) for the specific “type” of person you are talking to.

It is vital to be able to do the “three pitches” – the elevator (30 second), the 3 minute, and the 30 minute. Every one will cover the “same” content – but in very different ways. It’s very much related to the “10/20/30 rule of PowerPoint” that Guy Kawasaki promulgates: “a PowerPoint presentation should have ten slides, last no more than twenty minutes, and contain no font smaller than thirty points.” Or, to quote Winston Churchill, “A good speech should be like a woman’s skirt; long enough to cover the subject and short enough to create interest.”

The answer that epiphanized for me when I was asked that question most recently was this: “I presume everyone in the room is ‘as important’ as the CIO – but everyone gets the same ‘sales pitch’ from me: it’s all about ROI. The ‘return’ on ‘investment’ is going to look different from the engineer’s, manager’s, or CIO’s perspectives, but it’s all just ROI.”

The exact same data presented at three different levels of abstraction will “look” different, even though it’s conveying the same thing – because the audience’s engagement is going to be at their level of abstraction (though hopefully they understand at least to some extent the levels above (and below) themselves).

A simple example: it currently takes a single engineer 8 hours to perform all of the tasks related to patching a Red Hat server. There are 1000 servers in the datacenter. Therefore it takes 8000 engineer-hours to patch them all.

That’s a lot.

It’s a crazy lot.

But I’ve seen it countless times in my career. It’s why patching can so easily get relegated to a once-a-year (or even less often) cycle. And why so many companies are woefully out-of-date with their basic systems from known issues. If your patching team consists of 4 people, it’ll take them a year to patch all 8000 systems – and then they just have to start over again. It’d be like painting the Golden Gate Bridge – an unending process.

Now let’s say you happen to have a management tool available (could be as simple as pssh with preshared SSH keys, or as big and encompassing as Server Automation). And let’s say you have a local mirror of RHN – so you can decide just what, exactly, of any given channel you want to apply in your updates.

Now that you have a central point from which you can launch tasks to all of the Red Hat servers that need to be updated, and a managed source from which each will source their updates, you can have a single engineer launch updates to dozens, scores, even hundreds of servers simultaneously – bringing them all up-to-date in one swell foop. What had taken a single engineer 8 hours is still 8 – but it’s 8 in parallel: in other words, the “same” 8 hours is now touching scores of machines instead of 1 at a time. The single engineer’s efficiency has been boosted by a factor of, say, 40 (let’s stay conservative – I’ve seen this number as high as 1000 or more).

Instead of it taking 8000 engineer-hours to update all 1000 servers, it’s now only 200. Your 4 engineer patching team can now complete their update cycle in well under 2 weeks. What had taken a full year, is now being measured in days or weeks.

The “return on investment” at the abstraction level of the engineer is they have each been “given back” 1900 hours a year to work on other things (which helps make them promotable). The team’s manager sees an ROI of >90% of his team’s time is available for new/different tasks (like patching a new OS). The CIO sees an ROI of 7800 FTE hours no longer being expended – which means the business’ need for expansion, with an associated doubling of server estate, is now feasible without having to double his patching staff.

Every abstraction is like that – there is a different ROI for a taxi driver on his car “just working” than there is for a hot rodder who’s truly getting under the hood. But it’s still an ROI – one is getting his return by being able to ferry passengers for pay, and the other by souping-up his ride to be just that little (or lot) bit better. The ROI of a 1% fuel economy improvement by the fuel injector system being made incrementally smarter in conjunction with a lighter engine block might only be measured in cents per hour driving – but for FedEx, that will be millions of dollars a year in either unburned fuel, or additional deliveries (both of which are good for their bottom line).

Or consider the abstraction of talking about financial statements (be they for companies or governments) – they [almost] never list revenues and expenditures down to the penny. Not because they’re being lazy, but because the scale of values being reported do not lend themselves well to such mundane thinking. When a company like Apple has $178 billion in cash on hand, no one is going to care if it’s really $178,000,102,034.17 or $177,982,117,730.49. At that scale, $178 billion is a close-enough approximation to reality. And that’s what an abstraction is – it is an approximation to the reality being expressed down one level. It’s good enough to say that you start your car by turning the key – if you’re not an automotive engineer or mechanic. It’s good enough to approximate the US Federal Budget at $3.9 trillion or maybe $3900 billion (whether it should be that high is a totally different topic). But it’s not a good approximation to say $3,895,736,835,150.91 – it may be precise, but it’s not helpful.

I guess that means the answer to the question I titled this post with is, “the level of abstraction appropriate is directly related to your ‘function’ in relation to the system at hand.” The abstraction needs to be helpful – the minute it is no longer helpful (by being either too approximate, or too precise), it needs to be refined and focused for the audience receiving it.


*see what I did there?

owncloud vs pydio – more diy cloud storage

Last week I wrote a how-to on using Pydio as a front-end to a MooseFS distributed data storage cluster.

The big complaint I had while writing that was that I wanted to use ownCloud, but it doesn’t Just Work™ on CentOS 6*.

After finishing the tutorial, I decided to do some more digging – because ownCloud looks cool. And because it bugged me that it didn’t work on CentOS 6.

What I found is that ownCloud 8 doesn’t work on CentOS 6 (at least not easily).

The simple install guide and process really is about version 8, and the last one that can be speedy-installed is 7. And as everyone knows, major version releases often make major changes in how they work. This appears to be very much the case with ownCloud going from 7 to 8.

In fact, the two pages needed for installing ownCloud are so easy to follow, I see no reason to copy them here. It’s literally three shell commands followed by a web wizard. It’s almost too easy.

You need to have MySQL/MariaDB installed and ready to accept connections (or use SQLite) – make a database, user, and give the user perms on the db. And you need Apache installed and running (along with PHP – but yum will manage that for you).

If you’re going to use MooseFS (or any other similar tool) for your storage backend to ownCloud, be sure, too, to bind mount your MFS mount point back to the ownCloud data directory (by default it’s /var/www/html/owncloud/data). Note: you could start by using local storage for ownCloud, and only migrate to a distributed setup later.

Pros of Pydio

  • very little futzing needed to make it work with CentOS 6
  • very clean user management
  • very clean webui
  • light system requirements (doesn’t even require a database)

Pros of ownCloud

  • apps available for major mobile platforms (iOS, Android), desktop)
  • no futzing needed to work with CentOS 7
  • very clean user management
  • clean webui

Cons of Pydio

  • no interface except the webui

Cons of ownCloud

  • needs a database
  • heavier system requirements
  • doesn’t like CentOS 6

What about other cloud environments like Seafile? I like Seafile, too. Have it running, in fact. Would recommend it – though I think there are better options now than it (including ownCloud & Pydio).


*Why do I keep harping on the CentOS 6 vs 7 support / ease-of-use? Because CentOS / RHEL 7 is different from previous releases. I covered that it was different for the Blue Grass Linux User Group a few months ago. Yeah, I know I should be embracing the New Way™ of doing things – but like most people, I can be a technical curmudgeon (especially humorous when you consider I work in a field that is about not being curmudgeonly).

Guess this means I really need to dive into the new means of doing things (mostly the differences in how services are managed) – fortunately, the Fedora Project put together this handy cheatsheet. And Digital Ocean has a clew of tutorials on basic sysadmin things – one I used for this comparison was here.

jump start your brain by doug hall

I’m happy I didn’t pay for this copy of Jump Start Your Brain.

I’m saddened someone else did in order to give it to me.

The core of Doug Hall’s creative self-help book from 1996 is decent: get outside yourself, remember what it’s like to be a kid, have fun, don’t take yourself too seriously, and be willing to take calculated risks.

The problem is that summary could be said of pretty much any 3-5 page group of the book, and the rest of the pages seem to be filled with text, quotes, and graphics to show you that you can’t be effectively creative if you’re stagnant in your thinking.

Save yourself the trouble of buying (or even reading) this book, and instead take its core advice:

Maybe version 2.0 is better? I dunno. Not really psyched to find out.

But the blog looks nice.

charge for carry-on luggage

Airlines over the past several years have begun charging for all kinds of things that used to be “free” (they weren’t ever “free”, they just hid the cost in your ticket price).

One of the worst offenders to this list of fees, though, is the inane charge for your first checked bag whereas carry-on baggage is free. Southwest doesn’t charge for your first two checked bags – and other airlines won’t if you have status or book your flight with their branded credit card – which is the model all airlines should use. But they need to add charging for anything for than your FAA-recognized “personal item”.

Why? Because finding overhead bin space for bulky carry-on bags is what slows most boardings to a crawl. And it is what makes most travelers most frustrated when getting on the plane – not in the first or second boarding groups? They’re going to check your bag(s) for you anyway because all the bin space is taken. (Add-in the ridiculous seat pitch, and you can hardly put anything but a small backpack or purse down by your feet anyway.)

My solution: give the first (and maybe second) checked bags away for free. But charge heavily for carry-on baggage that is more than a personal item (ie your laptop case or purse). (I’d allow an exception for items purchased in-airport from the duty-free shops – they can be carried-on free, too.) By “heavily”, I mean at least $50.

And I would eliminate that crazy practice of gate-checking your bag when getting onto a commuter flight: just check the bag and don’t bottleneck the jetway getting on and off for the rest of us who weren’t as narcissistic as to think bringing our roll-aboards onboard was a good idea.

With the TSA  suggesting everyone arrive at least 2 hours before their flight, there is no reason you wouldn’t have time to check your bags. And with the hassle of trying to navigate a crowded terminal dragging your wheelie duffel behind you, everyone should love the idea of just getting it at baggage claim.

“But what about lost bags?” I hear you ask. Lost and misdirected baggage happens. But it’s pretty rare. It’s something that has happened to me the sum total of 3 times in my flying life (the last 18 years, several of which included flying frequently for work). And of those 3 instances, only 1 ended up with the bag going to the wrong airport – each of the other two ended up with the bag arriving before I did.

Frontier Airlines gets it right (almost – on the carry-on aspect they do, but they still charge for checked bags). Mash Southwest’s checked policy with Frontier’s charging for carry-ons, and you would have a worlds-better flight experience.

The other major benefit to this plan: your time going through TSA will be shorter – the fewer bags that have to be scanned, the less time it will take to get through.

apps on the network

{This started as a Disqus reply to Eric’s post. Then I realized blog comments shouldn’t be longer than the original post 🙂 }

The app-on-network concept is fascinating: and one I think I’ve thought about previously, too.

Hypothetically, all “social networks” should have the same connections: yet there’s dozens upon dozens (I use at least 4 – probably more, but I don’t realize it). And some folks push the same content to all of them, while others (including, generally, myself) try to target our shares and such to specific locations (perhaps driving some items to multiple places with tools like IFTTT).

Google’s mistake with Google+ was thinking they needed to “beat” Facebook: that’s not going to happen. As Paul Graham notes:

“If you want to take on a problem as big as the ones I’ve discussed, don’t make a direct frontal attack on it. Don’t say, for example, that you’re going to replace email. If you do that you raise too many expectations…Maybe it’s a bad idea to have really big ambitions initially, because the bigger your ambition, the longer it’s going to take, and the further you project into the future, the more likely you’ll get it wrong…the way to use these big ideas is not to try to identify a precise point in the future and then ask yourself how to get from here to there, like the popular image of a visionary.”

That’s where folks who get called things like The Idea Guy™ go awry: instead of asking questions, you try to come up with ideas – like these 999. And if you can’t/don’t, you think you’ve failed.

Social networks should be places where our actual social interactions can be modeled effectively. Yet they turn into popularity contests. And bitch fests. And rant centers. Since they tend towards the asymmetric end of communication, they become fire-and-forget locales, or places where we feel the incessant need to be right. All the time. (Add services like Klout and Kred, and it gets even worse.)

I would love to see a universal, portable, open network like the one Eric describes. All the applications we think run on social networks (like Farmville) don’t. They run on top of another app which runs on “the network”.

Layers on layers leads to the age-old problem of too many standards, and crazy amounts of abstraction. Peeling-back the layers of the apps atop the network could instead give us the chance to have a singular network where types of connections could be tagged (work, fun, school, family, etc, etc – the aspect of G+ that everyone likes most: “circles”). Then the app takes you to the right subset of your network.

Of course – this all leads to a massive problem: security.

If there is only One True Social Network, we all end up entrusting everything we put there to be “safe”. And while some of still follow the old internet mantra, “if you wouldn’t put it on a billboard, don’t put it on a website,” the vast majority of people – seemingly especially those raised coincident to technology’s ubiquitization – think that if they put it somewhere “safe” (like Facebook), that it should be “private”.

After all, the One True Social Network would also be a social engineer’s or identity thief’s Holy Grail – the subversive access to all  of someone’s personal information would be their nirvana.

And that, I think, is the crux of the matter: regardless of what network (or, to use Eric’s terminology, what app-atop-the-network) we use, privacy, safety, and security are all forefront problems.

Solve THAT, and you solve everything.

Or maybe you just decide privacy/security doesn’t matter, and make it all public.

what to automate

I have been in the world of automation for quite a while. Specifically in the realms of server, datacenter, and cloud automation – but I’ve been interested and/or involved in other tasks that tend towards automation (even for a short period of time) for far longer than just my post-college time in the world of HPSA and its related ilk.

One of the first questions customers ask us when we arrive onsite (heck – even way back in the technical presales cycle) is NOT what can be automated, but rather what should we automate and/or what can we automate first.

Analyzing the environment and finding some prime, low-hanging fruit to target in an initial automation push is vital.

To quote Donald Knuth, “We should forget about small efficiencies, say about 97% of the time; premature optimization is the root of all evil.”

In the realm of automating, that means picking on tasks that, while the tools at hand can make quick work of them, are done so infrequently as to not warrant an immediate focus, since the ROI on infrequently-done tasks is not going to be readily seen* should be skipped.

This is part of where being a good architect comes to bear.

No.

That’s not true.

This is where being a good listener and collator comes to bear. In a future post I’ll talk more fully about the art of architecting – but for today’s topic, let’s focus on the true key personality traits you must display to get a successful project started, implemented, and running.

You need to listen. You do not need to “hear” – you need to process what is being said, ask it back, take notes, ask for clarifications, etc. In the counseling world, this is called “active listening“. In the rest of life, it’s called being an attentive, thoughtful, caring, intelligent, adult human being.

When you hear a customer say they have a real problem with some task or other (beware – managementspeak coming!) – ie, they have “pain points” in various places, ask about what those individual tasks are actually comprised of. Investigate what can be touched today, what can be planned-for tomorrow, and what needs to be tabled for a future engagement (for you architects and sales folks reading, this translates into “what can we sell them later – after this project is successful?” – how can we build and strengthen this relationship?).

Take these notes and conversations you have to your colleagues and tease-out coherent lines of attack. Collate all the notes form everyone involved into commonalities – what has everyone heard a customer say? What did only one guy hear? What order did each person hear them in?

After you’ve listened, after you’ve taken your notes, after you’ve powwowed with your colleagues – then comes the fun part of any engagement: the actual automation!

Bring your cleaned-up and trimmed-down notes back to the customer in an easily-digestible form, and give a solid plan for what we will do now, what we want to do soon, and what really needs to wait to be done til later. Put an N, S, or L next to each item on your list. – it’s a first-cut priority draft. Then ask your customer for how they view those tasks, and listen to what they say are their priorities (including “real” dates, if any exist). You may need to reorganize your list, but keeping it involved in all project discussions will show you’re truly paying attention to them.

And at the end of the day, everyone’s favorite topic is themselves. Always – even shy people want to hear themselves bragged-up, talked-about, promoted, and given attention.

When you showcase your individual focus and attention on your customer, it will show in their willingness to accept you into their closer rings of trust – their readiness to receive you as a “trusted advisor”, which is what you want to be for them: you want to be who they can talk to about problems they’re seeing in their environment (current or potential) so you can bring your expertise to bear on their issues.

The role of any consultant who wants to be more than a mere grunt is not so much technical or business acumen, but that of their business therapist and/or best friend. You want to be able to say with Frasier Crane, “I’m listening”. And you want them to know that you really are.

Some of the early steps you can take today to bring yourself there are to:

  • avoid electronic distraction in meetings
  • document everything you do for work
  • be detailed
  • know industry trends, what competitors are doing, etc
  • treat everyone you come in contact with at a customer as if they were the most important person there
  • anticipate what you may be asked, and where you want to go
  • never speak authoritatively about that which you do not know
  • learn – be a “Lifelong Learner”: the day you stop learning is the day you stop growing, and the day you stop being reliable to others

*Unless, of course, those infrequent tasks are only infrequent because they’re “hard”, and therefore automating them will yield a solid ROI by allowing them to be done more often

first experiment follow-up

I’ve been attempting a “reactive”/”consumptive” reading experiment recently.

The first book I tried it on was the Henry Petroski’s horrid To Engineer is Human (my review). That turned into a failure as I couldn’t stomach his writing, and so “reacting” to it was going to pretty much be an exercise in futility.

So I’ve ditched that book – maybe someone else will not find it so poor a read.

Many of the books I read (and review) I get from my local library. All of which, therefore, are poor candidates for consumptive reading in the sense Ryan Holiday used the term in his blog post.

But as I dove through his writing a bit more, I saw his mention of a “commonplace book“.

“A commonplace book is a central resource or depository for ideas, quotes, anecdotes, observations and information you come across during your life and didactic pursuits. The purpose of the book is to record and organize these gems for later use in your life, in your business, in your writing, speaking or whatever it is that you do.”

Specifically, he was taught how to do one by Robert Greene (author of Mastery, The 48 Laws of Power, etc), and he cites various individuals in history who have maintained them. It’s also something that Roald Dahl mentioned obliquely in his book The Wonderful Story of Henry Sugar and Six More (one of my favorites by him (PDF)) in “Lucky Break” – namely, that he always keeps something on which to write nearby (a notebook, a scrap of envelope – even the dust on his car bumper) so that whenever an idea strikes him, he can jot it down in case it was good enough to actually write about:

“Sometimes, these little scribbles will stay unused in the notebook for five or even ten years. But the promising ones are always used in the end. And if they show nothing else, they do, I think, demonstrate from what slender threads a children’s book or short story must ultimately be woven. The story builds and expands while you are writing it.”

This got me to thinking about how I might integrate the idea myself – though, of course, in a slightly different way. And that’s where I am progressing to now: instead of “consuming” all the books I read, ones I find interesting I’m taking notes on in a composition book (specifically a quad-ruled one, as those are my favorite). I’ve found so far it’s helped form better reviews. It’s also not the only place I’ll keep those notes – many will end up on this blog. Others will end up on Twitter. Others maybe in email signatures, or Facebook posts, or wherever.

In our amazingly digitized world, writing by hand seems, well, old-fashioned and trite. Or hipster-ish and cool. (Depends on who sees you doing it, I think.) Sometimes I’ve already found my notes being done electronically – via SMS to myself, or draft blog posts, or just a quick Notes session on my laptop.

Anyways, where I’m going with all this is instead of always being a mere passive consumer of writing, I’m trying to be a bit more “thoughtful” about it 🙂