antipaucity

Slashdot has a post on gamification in the workplace today.

One of the myriad replies was from a poster, gomoX, who was pushing his company’s gamified tech support tool (invgate.com/en/service-desk/gamification). I’m all for product placement and pushing when it’s relevant (and here it most certainly was), but I don’t like the general concepts in that particular tool.

gomoX started well, too:

Bad system:
* 10 points for solving a ticket
* 1 point por replying to a ticket
* 4 points for chipping into another tech’s tickets (allegedly to help out)
* -20 points for reopened ticket
* -100 points for SLA missed

but then goes into describing (and then having shredded by many responders) their “Good system”:

* 1 point for solving a ticket
* 15, 10, 0, -10, -20 points for 5, 4, 3, 2 and 1-star customer ratings on those tickets
* -100 points for SLA missed
* 200 points bonus for doing 10 5-star tickets in a row
* 1000 points bonus for doing those 10 5-star tickets in a row in less than one hour

It even starts to become fun! And if you plug gamification throughout the whole system, even this (taken from a “Knowledge Week” quest that lasted through a specific week in an InvGate Service Desk instance):
* 10 points for creating a Knowledge Base article
* 15, 10, 0, -10, -20 points for 5, 4, 3, 2 and 1-star customer ratings on those articles
* 20 points for having the article you created used by other techs to solve a ticket
* 50 points for having the article you created used by customers to figure out the ticket themselves

I’ve written in the past about support organizations, and have a guide on effective support cases available, too. And I stand by my previous assertions that “gaming” and the metrics mindset are a Bad Thing™ – when they’re the BASIS of management reviews, promotions, etc.

The big problem with the InvGate concept is stated so cheerfully, I had to read it twice:

You get a performance metric in the amount of points an agent gathered during a period of X … It even has a “ka-ching” sound effect when you get points!

Seriously? a ‘”ka-ching” sound effect’? Who does this encourage? Certainly not any of the professionals I’ve ever worked with!

Maybe there are groups for which this would work – but none that I would want to deal with over anything important or business critical.

There are ways in which gentle, informal “competition” can be a Good Thing™ … but those are few and far between in the professional environment of support work.

^{_{A friend of mine pointed me at a [potentially] NSFW site with “badges” you can earn that was pretty funny (excluding the cussing).}}

Dave Winer had an interesting take on the recent Twitter-NYPD flare-up.

Personally, the thought of any government organization demanding records without a warrant is abhorrent.

However, since the entire point of Twitter is to make your tweets public … then what is there to subpoena? They’re all out there – visible to the world… Unless the user has deleted them (and, from my understanding, they are “real” deletes (unlike facebook “deletes” which may or may not go anywhere)).

So, NYPD – why are you not just looking at the tweets that are available publicly? Why are you trying to demand data that may or may not exist, and without a warrant?

Lastly, to Mr Winer’s comment that “the government has no business investing taxpayer dollars in private companies”: there’s a couple big problems therein. First, since it was in reference to the Library of Congress, we should make sure that in addition to not “investing” in archiving tweets, they also not invest in archiving books, journals, newspapers, etc – after all, those are also coming from “private companies”. Second, if the government shouldn’t be investing taxpayer dollars in private companies, then where, exactly, do you propose the “government” get what it needs to operate? By fiat? By dictatorial claim? No – those aren’t good public relations moves. The government needs to obtain the services and goods it needs to continue its functions from private industry (or we need to abandon this whole ‘capitalism’ thing and go for a pure central economy wherein all produced goods and services are provided by the government).

Diagram.ly – it’s sorta like Visio, but free, and web-based.

Meetings.io – like webex, including conference calling and file and screen sharing.

Qama - a calculator that doesn’t give an answer until you provide a “reasonable” guess.

Udacity – a free computer science program.

Urbanchickens - dedicated to raising chickens in “non-traditional” environments (like cities).

In conjunction with the customized PXE process I wrote about previously, it could be highly desirable to be able to return a server to the unprovisioned server pool in HP’s Server Automation.

This is a specifically-Linux procedure: though I’m sure something similar can be done with Windows^*.

run an ad-hoc script against a target server that contains the following:

dd if=/dev/zero of=/dev/sda bs=512 count=1
sleep 1
nohup reboot

This will erase the MBR and partition table, and then reboot the server.

Before it reboots, however, you need to deactivate and delete the server from SA – otherwise it will not register correctly.

If you’ve already enabled (or not disabled) PXE booting, when it reboots, it will pick the default entry off the PXE menu, skipping the hard drive as there is no valid boot record available to it.

Why would you want to do this?

Well, let’s say you’re doing a lot of build testing (verifying ks.cfg or unattend.xml files, for example) – this could be useful.

Or, maybe you want to get your build process completely streamlined and you’re working with the MBC functionality in SA – again, rapid recycling of machines is highly desirable.

In a later post I’ll discuss freeing the VM from SA in the process (ie, removing it from the ESXi host to fully release resources).

^{*_{In fact, you may be able to run fdisk /mbr on a Windows server – but I haven’t tried.}}

At least the publicly-searchable knowledge base had something.

I tried searching for “Kernel panic – not syncing: Fatal Exception” and “RIP” and “cpuid4_cache_lookup”.

And wouldn’t you know it! There’s a known issue if you try to install RHEL >5.4 x64 on ESXi 4.x if it’s running on AMD 6000 series CPUs. Guess what – we’re on ESXi 4.1 on AMD 61xx CPUs and were trying to install RHEL 5.8 x64.

Sigh.

The only workaround is to install 5.4 and then upgrade – that’s dumb. There needs to be a fix from somewho for this problem!

I found an interesting reference that indicates that for x86, the maximum number of CPUs the Linux kernel can handle is 255. I’d presume that it’s the same for Windows.

I’m curious – right now this is a rather large limit: but it won’t be for long! 16 core CPUs are available from AMD today. I see motherboards that will support 4 sockets (there may be some that go higher). That’s a total of 64 CPUs visible to the OS.

With AMD and Intel pushing the number of cores ever higher, how soon will we hit the practical limits of the x86-64 architecture – or is the limit of 255 not a “core” limitation of the architecture?

I’m curious to find this out – and to see how vendors will handle these upcoming limitations.

I recently discovered an interesting “feature” of symbolic links in conjunction with NFS mounts: they don’t work!

For example, let’s say you have the following NFS export:

/media/files

Inside of that export, you have the following path:

/media/files/isos/osmedia/linux/ubuntu

In *this* directory, you have a symlink called ‘current‘, which points to a different location:

current -> /media/osmedia/linux/ubuntu/11.10

Here’s the rub: if anyone tries to mount the root (or sub path) of that NFS export (say /media/files/isos/osmedia) at a local point (say, /mnt/osmedia), the symlink will fail to work correctly because there is no path /media/osmedia/linux/ubuntu/11.10 on the local machine!

That’s not cool.

Here is another case where bind mounts will save your bacon!

Instead of using symlinks on the NFS host, use bind mounts – that way the path will still work when exported.

Most typically in the past, I have seen mounts for CDs in RHEL work thusly:

mount /dev/hdc /mnt

In RHEL6 you need to do the following:

mount /dev/sr0 /mnt

I’d really like to know why Red Hat made the change, but knowing is useful

Came across this little gem recently:

/path/to/vmware-tools-distrib/vmware-install.pl --default EULA_AGREED=yes

That’s been a life saver a lot recently

Last time I talked about storage robustifiers.

In the context of a couple applications with which I am familiar, I want to discuss ways to approach balancing storage types and allocations.

Storage Overview

Core requirements of any modern server, from a storage standpoint, are the following:

• RAM
• swap
• Base OS storage
• OS/application log storage
• Application storage

Of course, many more elements could be added – but these are the ones I am going to consider today. From a cost perspective, hardware is almost always the least expensive part of an application deployment – licensing, development, maintenance, and other non-”physical” costs will typically far outweigh the expenses of hardware.

RAM

RAM is cheap, comparatively. Any modern OS will lap-up as much memory as is given to it, so I always try to err on the side of generous.

swap

After having found an instance where Swap Really Mattered™, I always follow the Red Hat guidelines which state that swap space should be equal to installed RAM plus 2 gigabytes. For example, if 16GB of RAM is installed, swap should at least equal 18GB. Whether swap should be on a physical disk or in a logical volume is up for debate, but do not chintz on swap! It is vital to the healthy operation of almost all modern systems!

Base OS

This will vary on a per-platform basis, but a common rule-of-thumb is that Linux needs about 10GB for itself. Windows 2008 R2 requests a minimum of 20GB, but 40GB is substantially better.

OS/application logs

Here is another wild variable - though most applications have pretty predictable log storage requirements. For example, HP’s Server Automation (HPSA) tool will rarely exceed 10GB in total log file usage. Some things, like Apache, may have varying log files depending on how busy a website is.

Application

Lastly, and definitely most importantly, is the discussion surrounding the actual space needed for an application to be installed and run. On two ends of the spectrum, I will use Apache and HPSA for my examples.

The Apache application only requires a few dozen megabytes to install and run. The content served-up by Apache can, of course, be a big variable – a simple, static website might only use a few dozen megabytes. Whereas a big, complex website (like StackOverflow) might be using a few hundred gigabytes (in total with any dynamically-generated content which might be in a database, or similar)^*.

The best way to address varying storage needs, in my opinion, is to use a robustifying tool like LVM - take storage presented to the server, and conglomerate it into a single mount point (perhaps /var/www/html) so that as content needs change, it can be grown transparently to the website.

Likewise, with HPSA, there are several base storage needs – for Oracle, initial software content, the application itself, etc. Picking-up on a previous post on bind mounts, I think it a Very Good Thing™ to present a mass of storage to a single initial mount point, like /apps, and then put several subdirectories in place to hold the “actual” application. Storage usage for the “variables” of HPSA – Software Repository, OS Media, Model Repository (database) – is very hard to predict, but a base guideline is that you need 100GB in total to start.

Choosing Storage Types

My recommendations

This is how I like to approach storage allocation on a physical server (virtual machines are a whole other beast, which I’l address in a future post) for HP Server Automation:

Base OS

I firmly believe this should be put on local storage – ideally a pair of mirror RAIDed 73GB drives (these could be SSDs, to accelerate boot time, but otherwise the OS, per se, should not be being “used” much. You could easily get away with 36GB drives, but since drives are cheap, using slightly more than you “need” is fine.

swap

Again, following the Red Hat guidelines (plus my patented fudge growth factor), ideally I want swap to be on either a pair of 36GB or a pair of 73GB drives – not RAIDed (neither striping, nor mirroring swap makes a great deal of sense). Yes, this means you should create a pair of swap partitions and present the whole shebang to the OS.

OS/application logs

Maybe this is a little paranoid, but I like to have at least 30GB for log space (/var/log). I view logs to be absolutely vital in the monitoring and troubleshooting arenas, so don’t chintz here!

Application

HPSA has four main space hogs, so I’ll talk about them as subheadings.

Oracle

It is important that the database has plenty of space – start at 200GB (if possible), and present it as a logically-managed volume group, preferably made up of one or more [growable] LUNs from a SAN.

Note: Thin-provisioning is a perfectly-acceptable approach to use, by the way (thin provisioning present space as “available” but not yet “allocated” from the storage device to the server).

Core application

The application really doesn’t grow that much over time (patches and upgrades do cause growth, but they are pretty well-defined).

Since this is the case, carve 50-60GB and present it as a [growable] LUN via LVM to the OS.

OS Media

Depending on data retention policies, number of distinct OS flavors you need to deploy, and a few other factors, it is a Good Idea™ to allocate a minimum of 40GB for holding OS media (raw-copied content from vendor-supplied installation ISOs). RHEL takes about 3.5GB per copy, and Windows 2008 R2 takes about the same. Whether this space is presented as an NFS share from a NAS, or as a [growable] LUN under an LVM group from a SAN isn’t vitally-important, but having enough space most certainly is.

Software Library

This is truly the biggest wildcard of them all – how many distinct packages do you plan to deploy? How big are they? How many versions needs to be kept? How many target OSes will you be managing?

I prefer to start with 50GB available to the Library. But I also expect that usage to grow rapidly once the system is in live use – Software Libraries exceeding 300GB are not uncommon in my field. As with the OS Media discussion, it isn’t vitally-important whether this space is allocated from a NAS or a SAN, but it definitely needs to be growable!

Closing comments (on HPSA and storage)

If separate storage options are not available for the big hogs of SA, allocating one, big LVM volume (made up of LUNs and/or DAS volumes), and then relying on bind mounts is a great solution (and avoids the issue of needing to worry about any given chunk of the tool exceeding its bounds too badly – especially if other parts aren’t being as heavily-used as might have been anticipated).

^{*_{Yes, yes, I know – once you hit a certain size, the presentation and content layers should be split into separate systems. For purposes of this example, I’m leaving it all together.}}