fighting the lack of good ideas

storage strategies – part 3

In part 2, I introduced SAN/NAS devices, and in part 1, I looked as the more basic storage type, DAS.

Today we’ll look at redundancy and bundling/clustering of storage as a start of a robust storage approach. Before I go any further, please note I am not a “storage admin” – I have a pretty broad exposure to the basic technologies and techniques, but the specifics of individual appliances, vendors, etc is beyond my purview 🙂

One of the oldest robustifiers is RAID – a Redundant Array of Inexpensive Disks. The basic theory is that you have a series of identical disks, and data is mirrored and/or striped across them to both accelerate writes and reads, and to provide a small level data safety: if one drive in a mirrored set dies, a replacement can be swapped-in, and no data will be lost (though performance will be degraded as the array is rebuilt).

Another robustifier is logical volume management. Under Linux, the tool is called LVM2. A logical volume manager collects varied partitions (whether from LUNs or DAS, standalone or RAID), and presents them as a unified partition (or “volume”) to the OS. One of the benefits of LVM is that as new storage is added, the volumes can be expanded transparently to the system – that also means that any applications that are running on the system can be “fixed” (if, for example, they are running low on space) without the need to stop the application, shut off the server, add storage, start the server, figure out where to mount the new storage, and then start the application again.

Robustifiers have as their main goal to make system more reliable, reduce downtime, and overall make a system more performant. As such, they are one part of an overall strategy to improve application/system responsiveness and reliability.

Next time we’ll look at how to figure out how to balance types of storage, specifically in the context of a couple applications with which I am familiar.

storage strategies – part 2

Continuing my series on storage strategies and options (see part 1), today I want to briefly look at SAN and NAS options.

First, storage area networks. SANs are “dedicated network[s] that provides access to consolidated, block level data storage”. Storage presented to a target server appears to the machine as if it is a “real” drive (ie, as if it were DAS) – which means it behaves exactly the same from the OS’s point of view.

Second, network-attached storage. A NAS is “file-level computer data storage connected to a computer network providing data access to heterogeneous clients”. Storage presented by a NAS host can be mounted on a target server, but won’t support being installed-to*, since the space is presented at a Samba or NFS share. A NAS device differs from a “mere” file server that happens to be publishing NFS or Samba mounts in that there are dedicated management interfaces for handling quotas, publishing policies (eg which protocols should be used for different network segments), etc.


  • flexibly-deploy storage “on-demand” to target machines
  • the storage appliance can be serviced – typically – while servers are utilizing it
  • with technologies like vMotion from VMware, storage can be migrated “live” from one device to another based on load


  • most expensive – though pricing varies widely depending on manufacturer, protocols exposed, size, redundancy, management interfaces, etc
  • dedicated storage technologies require dedicated storage admins over-and-above dedicated system admins to utilize and maintain properly

*The caveat here is if a NAS share has been mounted as a storage device for virtual machines through a hypervisor like VMware – then the vmdks could be stored on a NAS device, and the VMs will be running off the remote mount.

storage strategies – part 1

In follow-up to my previous article about bind mounts, here is the first in a series on storage strategies (while everything contained in this series is applicable to desktops and laptops, the main thrust will be towards servers). Today we’ll look at local/simple storage options (DAS – both the spinning and solid-state varieties).

The most basic form of utilizable storage is the direct-attached (or DAS) variety. In short, DAS covers any drives that are physically connected to a computer – hard drives, SSDs, etc.

Spinning disks, aka hard drives, are the most common form of DAS – and are extremely similar between consumer and professional levels (the only main differences are price* and guaranteed reliability): they’re a very mature, stable technology, and, excepting recent problems in Thailand, have been pretty cheap for a long time.

Depending on the server, there will be anywhere from 2-12 (or more) disk slots. If the disks are sized equally, they can be either added to a RAID (a topic for a future post on fault-tolerance), or used individually.

A more recent alternative to hard drives have been solid-state storage. SSDs use flash memory – which can be more fault-tolerant than disk drives because there are no moving parts. While, as with all devices, there is a failure rate with flash storage, it [generally] fails more gracefully than a spinning disk, as individual cells of the flash will wear-out/become inaccessible, rather than a platter  physically crashing.


  • “traditional” storage, which makes installation simple
  • storage performance can be easily isolated
  • generally-speaking, it’s the fastest storage option
  • cheapest storage option


  • if storage needs have been predicted too low, it can be costly and time-consuming to increase
  • if storage needs have been predicted too high, a server could be “wasting” lots of space
  • when failures happen, recovery can be a very time-consuming process

*There is an excellent question and answer on Server Fault that covers why costs of enterprise/professional storage are high, and I won’t rehash much of that information in this series.

binding your mounts

Over the past several years, I have grown quite fond of the ability to do bind mounts on Linux.

First, a little background. Most applications have specific directory structure requirements. Many have wildly varying space requirements depending on how a given install is utilized. For example, HPSA can use anywhere from 40-400-4000 gigabytes of space for its deployable Software Library, depending on a given customer’s individual needs. Likewise, the database can be quite small (under 15GB), or massive (exceeding 1TB) – again depending on usage patterns and use cases. Because of varying space and directory requirements, an old sysadmin trick is to use symbolic links to create fake directories so that an application can take advantage of the structure it needs, but the admins can keep their file systems structured they way they prefer (sometimes this is a side effect of various corporate IT standards).

This is cool because it allows all logs for all non-standard applications to be housed in a common locale.

The drawback is that if the application checks on the location of its logs and they’re not *really* at /var/log/appname, it may fail to start. When you look at the details of a symlink, it shows that it is merely a pointer to a different place, and is not, in fact, a directory. Eg, if you have a symlink at /var/log/appname that really points to a directory at /apps/logs/appname, a symlink does not have the first bit set to ‘d‘, because it is not a directory, is is set to ‘l‘. That can be a problem.

Without creating separate partitions for each application’s logs, after all that was why we have /apps/logs created, how can the dilemma be solved?

Enter mount --bind. Bind mounts take an existing directory path (which might be a mount point itself), and remounts it to a new location. Like this: mount --bind /apps/logs/appname /var/log/appname.

This also effectively treats the ‘from’ path as a partition.

And, since it *is* a directory, when the application checks on the location of its logs, it will not fail.

When combined with growable space (the subject of a future post) at /apps, this provides a very flexible approach to varied storage requirements for applications.

The final component of properly utilizing bind mounts is to add the bind to the file system table, fstab (/etc/fstab):

/path/to/orig /path/to/new bind rw,defaults 0 0

Specifically in the context of the tool I run on a day-to-day basis, HP’s Server Automation, here is what I will tend to do for a simple install, as it allows the flexibility to have any given sub-set of the product use as much space as it needs, without being overly tied to specific partitions and mounts points:

/dev/{device-name} /apps/hpsa ext3 defaults 0 0
/apps/hpsa/u01 /u01 none bind 0 0
/apps/hpsa/u02 /u02 none bind 0 0
/apps/hpsa/u03 /u03 none bind 0 0
/apps/hpsa/u04 /u04 none bind 0 0
/apps/hpsa/varoptoracle /var/opt/oracle none bind 0 0
/apps/hpsa/etcoptopsware /etc/opt/opsware none bind 0 0
/apps/hpsa/varlogopsware /var/log/opsware none bind 0 0
/apps/hpsa/varoptopsware /var/opt/opsware none bind 0 0
/apps/hpsa/optopsware /opt/opsware none bind 0 0
/apps/hpsa/media /media/opsware none bind 0 0

Next time I’ll cover strategies for storage allocation.