This is a specifically-Linux procedure: though I’m sure something similar can be done with Windows^*.

run an ad-hoc script against a target server that contains the following:

dd if=/dev/zero of=/dev/sda bs=512 count=1
sleep 1
nohup reboot

This will erase the MBR and partition table, and then reboot the server.

Before it reboots, however, you need to deactivate and delete the server from SA – otherwise it will not register correctly.

If you’ve already enabled (or not disabled) PXE booting, when it reboots, it will pick the default entry off the PXE menu, skipping the hard drive as there is no valid boot record available to it.

Why would you want to do this?

Well, let’s say you’re doing a lot of build testing (verifying ks.cfg or unattend.xml files, for example) – this could be useful.

Or, maybe you want to get your build process completely streamlined and you’re working with the MBC functionality in SA – again, rapid recycling of machines is highly desirable.

In a later post I’ll discuss freeing the VM from SA in the process (ie, removing it from the ESXi host to fully release resources).

^{*_{In fact, you may be able to run fdisk /mbr on a Windows server – but I haven’t tried.}}

For example, I have seen people list multiple full-time jobs that they could not have had at the same time (eg, both at one employer and also at the company that acquired their old employer).

I’ve also seen people claim to have accomplished things or be in a role that is either flat-out wrong, or worded in a weaselly way that looks like they’ve accomplished a lot more than they really did (eg showing only their current title at their current employer, but listing the start date as their initial hire date, and only listing their current accomplishments/roles (or listing all of them, but implying they did something that other individuals were actually responsible for)).

I’ve also seen LinkedIn profiles that are spartanly-populated – which is cool, that kinda follows my personal philosophy of never putting anything on my resume I don’t want to be asked about. But the ones that are full of – at the very least – questionable entries on their work history seem very troubling to me.

First, a little background. Most applications have specific directory structure requirements. Many have wildly varying space requirements depending on how a given install is utilized. For example, HPSA can use anywhere from 40-400-4000 gigabytes of space for its deployable Software Library, depending on a given customer’s individual needs. Likewise, the database can be quite small (under 15GB), or massive (exceeding 1TB) – again depending on usage patterns and use cases. Because of varying space and directory requirements, an old sysadmin trick is to use symbolic links to create fake directories so that an application can take advantage of the structure it needs, but the admins can keep their file systems structured they way they prefer (sometimes this is a side effect of various corporate IT standards).

This is cool because it allows all logs for all non-standard applications to be housed in a common locale.

The drawback is that if the application checks on the location of its logs and they’re not *really* at /var/log/appname, it may fail to start. When you look at the details of a symlink, it shows that it is merely a pointer to a different place, and is not, in fact, a directory. Eg, if you have a symlink at /var/log/appname that really points to a directory at /apps/logs/appname, a symlink does not have the first bit set to ‘d‘, because it is not a directory, is is set to ‘l‘. That can be a problem.

Without creating separate partitions for each application’s logs, after all that was why we have /apps/logs created, how can the dilemma be solved?

Enter mount --bind. Bind mounts take an existing directory path (which might be a mount point itself), and remounts it to a new location. Like this: mount --bind /apps/logs/appname /var/log/appname.

This also effectively treats the ‘from’ path as a partition.

And, since it *is* a directory, when the application checks on the location of its logs, it will not fail.

When combined with growable space (the subject of a future post) at /apps, this provides a very flexible approach to varied storage requirements for applications.

The final component of properly utilizing bind mounts is to add the bind to the file system table, fstab (/etc/fstab):

/path/to/orig /path/to/new bind rw,defaults 0 0

Specifically in the context of the tool I run on a day-to-day basis, HP’s Server Automation, here is what I will tend to do for a simple install, as it allows the flexibility to have any given sub-set of the product use as much space as it needs, without being overly tied to specific partitions and mounts points:

/dev/{device-name} /apps/hpsa ext3 defaults 0 0
/apps/hpsa/u01 /u01 bind rw,defaults 0 0
/apps/hpsa/u02 /u02 bind rw,defaults 0 0
/apps/hpsa/u03 /u03 bind rw,defaults 0 0
/apps/hpsa/u04 /u04 bind rw,defaults 0 0
/apps/hpsa/varoptoracle /var/opt/oracle bins rw,defaults 0 0
/apps/hpsa/etcoptopsware /etc/opt/opsware bind rw,defaults 0 0
/apps/hpsa/varlogopsware /var/log/opsware bind rw,defaults 0 0
/apps/hpsa/varoptopsware /var/opt/opsware bind rw,defaults 0 0
/apps/hpsa/optopsware /opt/opsware bind rw,defaults 0 0
/apps/hpsa/media /media/opsware bind rw,defaults 0 0


Next time I’ll cover strategies for storage allocation.

• blogging
• tweets
• etc

I know it’s not horribly scientific, but if you’d be willing to participate in this inquiry, please leave a comment.

I’ll be putting up a more formal poll in the coming days.

Thanks!

And whether you go “standard” or not, please, please, please document everything you do!

Those of us who will come after you will be intensely grateful for your brief efforts to ease our jobs!

---------
rwxrwxrwx
uuugggooo

Where “u”, “g”, and “o” indicate “user”, “group”, and “other” while “rwx” indicates whether the item can be read, written, or executed.

Also of note are file ownerships in the form of

user:group

So a file might be owned by warren:warren with read/write privileges only for the owner: rw-rw----.

That’s pretty handy.

How many of you know there are hidden permissions that do not show up in a normal use of ls -l?

I didn’t until last week – and I’ve been using Linux for more than a decade

Turns out that – at least with ext2, etx3, and ext4 filesystems – there are a variety of other attributes that can be assigned to a file. One of them is the immutable flag^*.

(Using a related command to ls, lsattr, a person can view the extended attribute flags of a file.)

In specific, the immutable flag disallows even the owner from deleting his/her own file! There was the source of a very ornery issue I ran into while doing an upgrade last week of a customer’s management environment. The upgrade installer ran into an issue wherein it interrupted its own process. But when it did, it left the filesystem in an inconsistent state – specifically with regards to an SSL certificate file and the machine identifier file (agent.srv and mid, if you’re curious) having the immutable flag set. When attempting to pick back up with the upgrade, the installer failed more [apparently] asymptomatically (and esoterically) than I had ever seen before.

After several exchanges with senior product support, a working fix of running chattr -i /path/to/file was provided. Neither myself, nor any of the senior admins at my customer had ever heard of chattr. So off to the man pages. Turns out that chattr is quite the command – but it’s not well publicized (at least, I don’t think it is).

If you run into a file you own, but cannot delete, be sure to inspect the extended attributes with lsattr, and fix any mis-set flags with chattr.

^{*_{The entire list of alterable flags: “append only (a), compressed (c), no dump (d), extent format (e), immutable (i), data journalling (j), secure deletion (s), no tail-merging (t), undeletable (u), no atime updates (A), synchronous directory updates (D), synchronous updates (S), and top of directory hierarchy (T).” Non-modifiable, but visible with lsattr, flags: “huge file (h), compression error (E), indexed directory (I), compression raw access (X), and compressed dirty file (Z)”.}}

Being terminated over charging a movie to your room? Seems harsh (getting the $9.95 back from the employee would seem to be easier) – but breaking the rule is breaking the rule.

Being terminated over buying gum? Ok, so I WOULD terminate somebody over that … but I hate the stuff

But it’s repulsive, revolting, and wrong
chewing and chewing all day long
The way that a cow does*

There are a host of ways listed in the article – that I find truly shocking – to cheat on expense reports: blank receipts? buying gifts and then selling them on eBay? double-billing? Wow. The sheer effort taken by some people to cheat is astonishing!

Where I work now has a corporate credit card issued to every traveling employee. The only time we submit non-AmEx charges is if a place doesn’t accept AmEx: it’s just way easier to use the corporate card than it is to try to give all the supporting documentation of a personal card. Plus, there’s the benefit that it’s not my personal limit that is being affected if a customer delays in paying a bill.

Everyone that works where I do now also follows the expense guidelines we have – don’t exceed the IRS per diem rate for your region (on average). If you want to eat someplace nice for dinner – that’s fine. Just eat someplace less expensive the next day. Sticking within the rules isn’t that hard … so why would you want to try to evade them and end up with employment history issues like termination on your record?

OS Prov is most typically done using PXE (or the similar bootp process). SA provides a PXE server that gives a boot menu to network-booted systems. That menu contains a variety of choices: linux, windows, winpe, etc.

In most environments, one particular OS will be dominant, and typically one particular version of that OS – whether it be RHEL 5 x64 or Windows 2008 R2, it’s usually just one that makes up the lion’s^* share of systems on the network.

If you know that you will be provisioning, say, Windows 2008 R2 90% of the time, it would be nice to not have to always have to pick it manually from the PXE boot menu.

Here’s what you need to do to make that happen (presuming you want to use Build Plans):

Edit the following file:

/opt/opsware/boot/tftpboot/pxelinux.cfg/default

For example, if you have this at the beginning of the file:

"prompt 1
default local
timeout 100
display pxelinux.msg
implicit 0"

Change it to this for the OGFS version of winpe64:

"prompt 1
default winpe64-ogfs
timeout 100
display winpe64-ogfs.msg
implicit 0"

You can use the above process – modified, of course – for any of the available boot images.

^{*_{And if you’re provisioning Mac OS X 10.7, “Lion” makes up all the share}}

1. for folks with no “real” experience yet, I ask them to rank themselves on a few key technologies on the “Google scale”
• the range is 0..10 where a 0 is no knowledge, 1 is some, 10 is “you wrote the book”, 9 is you could’ve written the book, or you edited/contributed
• on a few occasions, I have had folks ask to change their ranking from their initial [overconfident] statement to one that is much closer to inline with their true experience/comfort/knowledge level – and that’s OK in my book – honesty is always the best policy here
2. a couple quick “about us” questions – open-ended inquiries that let the candidate tell me what they’ve done for work
• this verifies their resume
• gets them warmed-up for the rest of the call
• allows the candidate to brag on something
3. perhaps a couple quick probes to find out more about a specific experience
4. a few basic / intermediate questions to assess candidate’s technical chops (ie, verify that their resume is accurate)
• this goes along with my personal rule of “never put anything on a resume you don’t want to be asked about”
5. open-ended, intentionally-vague questions to gauge problem solving ability, and methodologies
• see how they go about refining the problem statement (if at all)
• gauge estimation skills
• gauge teamwork and delegation aptitude
6. a few intermediate/advanced questions about an area they *don’t* know anything about – to gauge their response to unfamiliar/stressful situations
• in my field in particular, it is impossible to know every new technology or even (probably) to be truly 100% aware of those that you do use every single day
7. a few intermediate/advanced questions in their now-articulated fields of expertise (presuming I have any)
• this verifies more of their stated (and unstated) job experience, and helps determine at what title/work level they should start
8. lifestyle/workstyle questions
• how much they enjoy travel
• how they handle last-minute demands and “requests” by customers and management
9. a few questions to gauge flexibility of response to changing requirements
• for example, switching a project from being Solaris-based to Windows-based part way into implementation because a new CIO has come in, or new licensing is available, etc
10. open time for them to ask me whatever they may wish to know that I can tell them
• this usually ends-up being very short because the candidate was stressed-out over the interview, and can’t think of anything about the company they want to know on the spot

What I try to NEVER ask:

• “trivia” questions – I bet there are C questions even K&R couldn’t answer

• I guarantee I can ask you a question about your area of expertise you cannot answer…just like I guarantee you could do the same to me
• since that is the case, trivia questions are pretty pointless, and more of an ego stroke to the asker than anything else
• pointless “MindTrap“, lateral-thinking questions

• riddles are fun – but only add to the stress of the interview
• pointless problem-solving and estimation problems

• for example, “how would you move Mt Fuji”, “why are manhole covers round”, or “how many gallons of water flow into New York Harbor from the Hudson River per hour”
• estimation problems are wonderful tools and games to play, but not in an interview
• illegal questions

• sometimes they slip out, but it’s never intentional

I adjust my questioning to fit the situation, timing, and candidate responses – so it’s [somewhat] different every time.

When the interview is done, I write-up my evaluation of the candidate and send it on to the hiring manager. In line with Joel Spolsky‘s “Guerilla Guide to Interviewing“, I make sure to put my firm conclusion of Hire/No-Hire near the top, and again at the bottom – with my reasoning in between.

One thing I have noticed about almost every interview I have ever taken or given is that I end up learning something in the process – and not just about the candidate (or company). It’s important to listen to both how and the candidates responds to questions, and what they say.

So, if you ever get the chance to interview with me, you have an idea of how I’m going to run the show

Stateless merely means that any given request has nothing to do with the previous, or the next request. This enables the world wide web, as web servers do not need to keep track of who is receiving data, nor ow much they have: they get a request, and ship data to the requestor.

It is up to the requestor (often a web browser) to handle the incoming data.

If not every part of a web page, for example, is sent, the browser will display what it can.

This is analogous to a creditor sending you a bill (request), and you sending a check back to them – once the bill has been sent, the creditor knows nothing about the state of the bill until he receives a payment. Likewise, once the check is dropped in the mail, the payor knows nothing about his bill until the check clears his bank.

Why is this important? Because of an oft-repeated “request for enhancement” to the product I use on a daily basis. When the implementors of Opsware SAS were picking how a user should communicate with the system, they picked to run everything over http(s). They chose to utilize http because it’s commonplace, well-understood, and easy to work with.

One of the things about statelessness is that you cannot know how many people are using a given web page at the same time. Google cannot tell anyone how many people are actually looking at www.google.com at this moment. They can tell you how many loaded it,and how many just presses “Search”, but they can’t know what percentage of the loaders promptly went elsewhere – either to a different page, or a different room in their home.

One way around the statelessness of http is to utilize cookies or session data – but that merely adds a check layer to the interaction, it does not provide true “statefulness”.

Several times during my time in Support at Opsware (and after HP’s acquistion), I would have a customer who was looking for the ability to determine who was logged-in at any given time (in similar fashion to running `w` or `who` or `finger` on a Linux/Unix system). This could be important to know whether a user is “doing something” before doing an application restart.

However, since communication is all done via http, there can be no state known in the tool. Once you load a web page, it is being viewed/rendered on your local machine in your web browser – the server could be shut off, your network connection removed, or any of a host of other simulations of restarting the application. And your browser would be none-the-wiser, nor should it be: it has the data it requested/received, and you’re doing something with it.

This carries over to the product I work with. Jobs might be scheduled by a user to run every day at 0200 – but he doesn’t need to be actively logged-in to have them run. Likewise, someone may have logged-in, but is not “doing” anything currently (maybe they’re at lunch).

Another case of why technical intricacies matter