Tag Archives: automation

programming your home by mike riley

Mike Riley’s entry in The Pragmatic Programmers series, Programming Your Home – automating with Arduino, Android, and your computer – was a lot of fun.

While I am not really in a position to do many of the mini projects given in the book (wrong type of house plus we rent), reading some of the project ideas did give me some inspiration for other activities. One of those is a Buffer-like tool I’m now writing to queue tweets over-and-above what the free level of Buffer will allow (and on a different schedule from my Buffer-fed queue). In conjunction with python-twitter, cron, and simple email messages, I’ve got a system started to which I can email things I would like to be posted, and they will go out when the cron job runs.

The Arduino is an impressive embedded platform – one that has also rekindled another long-time interest I’ve had in robotics. Years back, I recall seeing Sally Struthers advertising for one of those learn-at-home groups, and one of the options was robotics. (By “years back”, I mean 20+ years ago – probably more like 25 years ago, at this point.) I used to own a copy of Robot Builder’s Bonanza – and read it cover-to-cover a couple times. I loved watching Battlebots on TV. I’ve always wanted to buy/use LEGO Mindstorms.

Using robots to automate daily activities (and, of course, for fun) has been a fascination since I first saw Lost In Space and myriad other scifi shows and movies.

Riley does a great job of not demanding you be an expert programmer (or even a programmer at all) with the fully-implemented code examples in the book. He also does a good job of indicating what you’ll likely have to tweak on your own – and what you can probably just leave alone in the examples. Add to this the “extra credit challenges”, and I highly recommend it to anyone interested in home automation, embedded development, robotics, or just general programming/scripting.

There are some other interesting Python snippets throughout the book – that don’t have to be used in the context of an Arduino (like using Google’s SMTP server (via authentication)).

automation {gp}

The way people moved up the ladder in IT during my early days (starting in 1975) was to take on new projects that allowed them time to master the new software and become the local expert. As you became the local expert on many new software products, management became very comfortable giving you more and more of these new software projects. Now, at this critical time you needed to train your replacement for some of the software that you had become the local expert because you could not maintain forward momentum with the tremendous drag caused by being the local expert for so many software offerings. Of course, you would get management to agree to lighten your workload because your current workload didn’t allow you to work on their next “Pet” project (after all you are now their go to guy). Also, you would push to give up the software that was either too time consuming and/or not part of your future automation plans. If you didn’t give up something (holding onto knowledge is your power trip), you would fail to meet management’s expectations and someone else would get the new projects and your plans would stall (cut off from new software knowledge and isolated).

When you start to automate, you automate your existing manual processes usually by using a wrapper because it is the fastest way of showing progress. However, after demonstrating that you can automate longstanding processes, you now have the evidence to convince management to allow you to automate your next project (beginning to end) because you will become the local expert not only on the new software but also on the automation software. After successfully completing the automation beginning to end, you are in a position to push for the policy that states that all new software/applications will be automated as part of its installation/configuration steps. Now, you push for automation friendly software/applications because wrappers will no longer be acceptable automation options.

The philosophy of automation document was compiled during IBM user group meetings from around 1987 through 1989 when automation was a very hot topic.

The Philosophy of Automation

Automation is not a technical problem it is a people problem.

When you initially automate, you convert your process flow documents into executable code that consistently runs on a prearranged schedule, or through a monitor, or an error message. These executable processes enforce your process rules every time.

You cannot automate processes that are all over the place.

When you automate your processes they will be transformed.

Because you automate your processes automation never ends

Automate as close to the source as possible.

Problems with automated processes occur infrequently but are more difficult to solve than manual processes.

The combining of problem, change, and asset management with automated process management and root cause analysis, improves quality and allows you to consistently meet your service levels.

All automation code is throwaway code.

Automation is very exciting

Automation is very rewarding.

Everyone is on the automation team

Guest Post from one of my colleagues, Dave B

automating or automation?

I have been working in the realm of “automation” – specifically data center automation – for several years.

Merriam-Webster defines “automating” thusly:

  1. to operate by automation
  2. to convert to largely automatic operation <automate a process>

Notice the subtle difference with M-W’s definition of “automation“:

  1. the technique of making an apparatus, a process, or a system operate automatically
  2. the state of being operated automatically
  3. automatically controlled operation of an apparatus, process, or system by mechanical or electronic devices that take the place of human labor

These words tend to be used interchangeably – but they are different. Most of the customers I have worked with think they are “doing automation”, when in actuality they are only barely starting to “automate”.

What do I mean by this?

Most customers that bring automation tools in-house (whether simple cron jobs or complex tools like HPSA) take their current, manual processes and merely write wrappers around them to make them “automatic”. That is the first step of automation – but it’s only the first step.

Too many people try to take new tools and make them fit their current processes, procedures, and policies – rather than seeing what policies, procedures, and processes are either made redundant by the new tools, or can be improved, shortened, or – wait for it – automated!

Think of a physical example – you’re a Shaker Cabinetmaker in the late 1800s. You’re making end tables. Cutting dovetails with a dovetail saw. Sanding with a block and sandpaper. Cutting pieces to size by hand. Drilling mortises and cutting tenons with a manual auger and small saw. Lathing on a treadle-powered lathe.

Jump forward 100 years. You’re WGBH in Boston wanting to come up with a how-to program to air weekly. You find a fellow named Norm Abram, and pay him to do the show. You put New Yankee Workshop on TV for years. Norm visits places like Hancock Shaker Village for inspiration. But Norm doesn’t use loads of hand tools – he uses radial arm saws, drill presses, table saws, routers, joiners, etc. Why? Because he wants to make a prototype or three in a few days. And then wants to film a show in two (or occasionally longer for big projects) and be able to give you the confidence, along with a set of measured drawings, that you could, more-or-less, replicate what he did in your own home.

Which approach is better? Neither – they both yield end tables. Which approach is more repeatable – especially by someone with little experience? That’d be Norm’s method: the automated method. But the New Yankee Workshop is only slightly down the road of automating the entire process.

Automation (in the furniture world, at least) is found at Ikea. Thousands of identical Ingos and Karls roll out the door every year. Produced by automated machinery.

The job of the Information Technologist has a great deal of art to it – but it’s also a science: at the core of everything that a computer does is logic (perhaps poor logic, but logic nonetheless). Everyone in information technology (especially, though it’s applicable to myriad other industries) should be striving to make themselves replaceable – because no one is irreplaceable. I’ve seen it come true in scores of settings: the person who makes themselves “irreplaceable” never gets promoted, and is eventually replaced by someone else: either management removes him or finds a way around him, or he leaves the organization.

Therefore, preemptively make yourself replaceable. This was Ken Moellman’s campaign when he ran for State Treasurer in KY: to eliminate the very job he was running for.

There’s a secret to making yourself replaceable – and that is that when you can show that you’re replaceable, especially in the world I work in, you tend to be promoted. You also tend to grow because you’re learning more. Because you learn more and grow, you become more valuable. Because you become more valuable, you can move up the chain as you like.

Do artisan works still have a market? Of course – go look at any “artsy” type store that showcases “local craftsmen” in almost any part of the country: they’re offering their hard work for your consideration … at a hard price.

Is the artisan chair fundamentally any better than the Ikea chair? Maybe, maybe not. It sure looks better. But it’s not as repeatable.

And repeatable tasks get automated because if you have to do it twice, you need to write it down. And if you have to do it more than twice, you need a process anyone can follow. But processes are always open for refinement and replacement. The process to get a piece of lumber from a tree is not completely dissimilar to day from how we did it before the advent of the sawmill – but with laser-guided blades, the sawmill of 2013 can optimize lumber out of a log in a way very very few people ever could .. and can cut the log into its constituent boards faster than any person.

The process for manually provisioning a RHEL server is pretty simple – but shortly after introducing Anaconda, Red Hat introduced the kickstart (modeled after jumpstart). Microsoft, likewise, has unattend.txt and unattend.xml (for either the DOS or WinPE methods of installing). And SuSE, HPUX, and AIX have their systems (AutoYaST, Ignite, and NIM).

Why do these tools exist? It’s so administrators can rapidly deploy machines without having to do a lot of extra setup work by hand. The same can be asked for why do chainsaws, table saws, and circular saws exist? It’s so you can cut wood more rapidly and more repeatably than by hand. You can fell a tree with an axe. You can fell a tree with a saw. But for a single person, felling a tree with a chainsaw is best. Or you can use a Tiger Cat.

The first step of automating needs to be building wrappers to reliably repeat manual processes.

The second, and far more important, step in the paradigm shift from manual methods to automation, is to systematically go through all of your processes, procedures, and policies and see what can be refined, what can be replaced, and, most importantly, what can be removed.

What legacy activities are you doing that should be eliminated, updated, or cleaned-up?

delivering solutions – “shipping is a feature!”

Back in 2009, Joel Spolsky wrote an article called The Duct Tape Programmer. Of everything he has written, I think this is the very pinnacle, and it is summed in one simple sentence in the middle: “Shipping is a feature.”

I’ve referenced this article twice before (in Feb and Sep of ’11).

Why is this so important in my mind?

I went back to school in 2003 to complete my bachelor’s degree in CIS. I had graduated in 2001 with an AAS in CIS from HVCC, and after finding nothing in 2 years of searching better than the job I had at Hertz, I decided a 4 year degree might help. I graduated from Elon University in Dec 2006 with my newly-minted BA in CIS. During my tenure at school I discovered that I didn’t really like the development end of Computer Science, and I instead preferred the analytical and integrational aspects of systems work – tying disparate tools together, improving internal workflow, etc – to help make individuals’ lives better and easier. In other words, I enjoyed finding ways to automate time-consuming and repetitive tasks to allow myself (and others) to focus on more interesting work – like figuring out how to automate more tasks to move up the chain.

I worked for a few places while I was at school (two different departments at the school itself, a pair of non-profits, and some freelance side work doing web site development). When I graduated, therefore, it was only natural that I ended-up with a pair of offers to work with automation tools – one from a company called Opsware, and one from a place called Network General. For a variety of reasons, I chose Opsware.

It wasn’t long after I started in Support for Opsware’s Server Automation [System] product that I became more and more sold on the product, and grew bored doing support – troubleshooting is fun, but with the paucity of good support tickets*, large similarity of cases coming from customers, etc .. it just wasn’t “me”. Shortly after HP purchased Opsware I put in to move from Support to Professional Services – to, hopefully, get a chance to work with harder integrational problems than I would ever see helping people over the phone and via email.

Beginning March of 2008^ I moved from Support to ProServe, and did start to get a taste for the bigger systemic problems that could be solved with the Opsware HP BTO suite. While with HP, I had the opportunity to do the global delivery of HPSA 7.5 for HSBC – performing both installation and onsite mentoring/training in Chicago, NYC, London, and Hong Kong. I also did the replacement install of HPSA 7.0 (a non-upgrade-to release) for Home Depot in Atlanta to manage their 2200 stores. There were some other customers I worked with, too – but those were the two biggest.

One of the issues that has arisen with [nearly] every customer I have ever worked with it that they want what they’ve agreed to pay for in the Statement of Work (SOW) signed, sealed, and delivered by the end of the project – and if it’s not, they want good reasons to sign a CO (change order) to modify the SOW.

And it’s no surprise. When you cost someone nearly 7 cents per second to work for them, they want to see results!

One of the constraints, therefore, that needs to be constantly watched is scope creep – the insidious tendency for all projects to go beyond their intended purpose (violating law 47 of the 48 laws), exceed budget, and never deliver what is really needed.

My primary goal when I work with a customer is not, perhaps paradoxically, to “make them happy”. One thing I learned when working in support is that the customer is never right!. You may have to pretend that they’re kinda right – but they’re always wrong. They do not know what they want. They do not know what they need. And they certainly do not know what is wrong if you ask them.

My primary goal when I work with a customer is to deliver what they have paid for. When possible, I will change course slightly (following proper CO processes) – but I want them to get what they have agreed to pay for. Ideally, especially now that I am in the architecture end of the world much more than ‘just’ delivery, I can work with them in the pre-sales process to get the SOW to something that approximates what they need. But I always aim to give them what they have paid for. Everything else is window dressing.

At the end of a project – whether as outside consultants, students, internal employees, at home, for work, etc – what needs to be seen is what was paid for.

Ship. Deliver.

Without those two, nothing gets done.

* I have grown so frustrated with support processes that I spent time a couple years ago writing a small eBook that includes a section on how to make good tickets. I’ve also written on ways to improve your support organization before.
^ Just realized that means I’ve been doing ProServe or PS-like work for 5 years running, and have been with the automation suite for more than 6 now.
! Before you become too concerned – I do realize there are a few good customers out there. But they are just that – few, and VERY far between.

geeks night out

Last week I went to the Geeks’ Night Out at Beerworks in Lexington.

One of the people I met was a junior EE major at UK named Robyn who, along with two of her friends, is looking to start a home automation company (The Unity Box) – but not in the realm of a company like Ambiance (interestingly, a company I was going to start with as a junior developer back in 2001 when the bottom dropped out of the tech bubble – but that’s another story). They want to make a small control device along the lines of a DVR or Roku that would control smart outlets/switches in your home (and be able to learn a la the Nest about your habits (with, of course, manual overrides for non-pattern events)).

Yesterday I found the WeMo from Belkin.

Robyn – looks like you and Belkin should talk: they’ve done the part about the smart outlets 🙂


I have been deeply involved in data center management and automation for well over 5 years.

Most companies still view automation the Wrong Way™, though – and it’s a hard mindset to change. Automation is NOT about reducing your headcount, or reducing hiring.

Automation is used to:

  • improve the efficiency of business tasks
  • improve employee productivity
  • reduce human error
  • ensure consistency, and auditability
  • improve/ensure repeatability
  • replace “fire fighting” with planning and proactivity
  • ensure an organization can pass the bus test (which disturbingly-few can)
  • free engineers to work on interesting, engineering problems – not day-to-day busywork

Cringely has an article on this topic this week, entitled “An IT labor economics lesson from Memphis for IBM“.

How can a company 1/100,000th the size of IBM afford to have monitoring?  Well, it seems DBADirect has its own monitoring tools and they are included as part of their service.  It allows them to do a consistently good job with less labor.  DBADirect does not need to use the cheapest offshore labor to be competitive.  They’ve done what manufacturing companies have been doing for 100+ years – automating!

Even today IBM is still in its billable hours mindset.  The more bodies it takes to do a job the better.  It views monitoring and automation tools as being a value added, extra cost option.  It has not occurred to them you could create a better, more profitable service with more tools and fewer people.  When you have good tools, the cost of the labor becomes less important.

Any company that fails to realize that throwing more people at the problem is rarely the answer (something former IBMer Fred Brooks wrote about as a post-mortem of the OS/360 project in The Mythical Man-Month), is doomed to fail – consistently, and tragically.

And yet IBM is still in the mindset of the 1960s and raw, manual labor in an increasingly-connected, -compliant, -complex, and –cloudy world. They are still trying to solve problems the Risk way – throw a gob o’ guys at the problem, and roll over your opponents through sheer numbers.

In many ways, it is sad to see the demise of once-great companies like IBM. There’s the loss of competition, the passing of the Old Guard, etc.

But it’s also a huge opportunity for new businesses to come in, compete, and clean-up in sectors the Bug Guys can’t (or won’t) touch well.