antipaucity

fighting the lack of good ideas

merging centos iso images

Thanks to @Anon on Unix.SE for the pointer on how to do this. And to @Andy‘s comment on @mmckinst‘s answer for the warning about additional packages you may need.

As my three readers know, I run a CentOS mirror. One of the idiosyncrasies of CentOS, like its upstream RHEL, is that DVD ISOs aren’t always just one image – for example, the 6.6 x64 image comes on two ISOs. I suppose this has something to do with the “normal” or “simple” capacity of a DVD disc, but it’s annoying.

Enter the mkdvdiso.sh script (original found here) from Chris Kloiber & Phil Schaffner.

The process I used to combine these two ISOs into one is as follows:
yum install isomd5sum createrepo mkisofs
mkdvdiso.sh /full/path/to/original/isos /full/path/to/destination.iso

For posterity, and in case the CentOS wiki dies, below is the mkdvdiso.sh script:

#!/bin/bash

# by Chris Kloiber 
# Mods under CentOS by Phil Schaffner 

# A quick hack that will create a bootable DVD iso of a Red Hat Linux
# Distribution. Feed it either a directory containing the downloaded
# iso files of a distribution, or point it at a directory containing
# the "RedHat", "isolinux", and "images" directories.

# This version only works with "isolinux" based Red Hat Linux versions.

# Lots of disk space required to work, 3X the distribution size at least.

# GPL version 2 applies. No warranties, yadda, yadda. Have fun.

# Modified to add sanity checks and fix CentOS4 syntax errors

# TODO:
#   Add checks for available disk space on devices holding output and
#       temp files.
#   Add optional 3rd parameter to specify location of temp directory.
#   Create .discinfo if not present.

OS_VER=\
$((test -e /etc/fedora-release && rpm -qf /etc/fedora-release --qf "FC%{VERSION}") \
|| (test -e /etc/redhat-release && rpm -qf /etc/redhat-release --qf "EL%{VERSION}") \
|| echo OS_unknown)

case "$OS_VER" in
  EL[45]*|FC?)
        IMPLANT=/usr/lib/anaconda-runtime/implantisomd5
        if [ ! -f $IMPLANT ]; then
            echo "Error: $IMPLANT Not Found!"
            echo "Please install anaconda-runtime and try again."
            exit 1
        fi
        ;;
  EL6*|FC1?)
        IMPLANT=/usr/bin/implantisomd5
        if [ ! -f $IMPLANT ]; then
            echo "Error: $IMPLANT Not Found!"
            echo "Please install isomd5sum and try again."
            exit 1
        fi
        ;;
  OS_unknown)
        echo "Unknown OS."
        exit 1
        ;;
  *)
        echo "Fix this script for $OS_VER"
        exit 1
esac

if [ $# -lt 2 ]; then
        echo "Usage: `basename $0` source /destination/DVD.iso"
        echo ""
        echo "        The 'source' can be either a directory containing a single"
        echo "        set of isos, or an exploded tree like an ftp site."
        exit 1
fi

DVD_DIR=`dirname $2`
DVD_FILE=`basename $2`

echo "DVD directory is $DVD_DIR"
echo "ISO file is $DVD_FILE"

if [ "$DVD_DIR" = "." ]; then
    echo "Destinaton Directory $DVD_DIR does not exist"
    exit 1
else
    if [ ! -d "/$DVD_DIR" ]; then
        echo "Destinaton Directory $DVD_DIR must be an absolute path"
        exit 1
    else
        if [ "$DVD_FILE" = "" ] || [ -d "$DVD_DIR/$DVD_FILE" ]; then
            echo "Null ISO file name."
            exit 1
        fi
    fi
fi

which mkisofs >&/dev/null
if [ "$?" != 0 ]; then
    echo "mkisofs Not Found"
    echo "yum install mkisofs"
fi

which createrepo >&/dev/null
if [ "$?" != 0 ]; then
    echo "createrepo Not Found"
    echo "yum install createrepo"
fi

if [ -f $2 ]; then
    echo "DVD ISO destination $2 already exists. Remove first to recreate."
    exit 1
fi

# Make sure there is enough free space to hold the DVD image on the filesystem
# where the home directory resides, otherwise change ~/mkrhdvd to point to
# a filesystem with sufficient free space.

cleanup() {
    [ ${LOOP:=/tmp/loop} = "/" ] && echo "LOOP mount point = \/, dying!" && exit
    [ -d $LOOP ] && rm -rf $LOOP 
    [ ${DVD:=~/mkrhdvd} = "/" ] && echo "DVD data location is \/, dying!" && exit
    [ -d $DVD ] && rm -rf $DVD 
}

cleanup
mkdir -p $LOOP
mkdir -p $DVD

ls $1/*.iso &>/dev/null
if [ "$?" = 0 ]; then

    echo "Found ISO CD images..."

    CDS=`expr 0`
    DISKS="1"

    [ -w / ] || {   # Very portable, but perhaps not perfect, test for superuser.
        echo "Only 'root' may use this script for loopback mounts" 1>&2
        exit 1
    }

    for f in `ls $1/*.iso`; do
        mount -o loop $f $LOOP
        cp -av $LOOP/* $DVD
        if [ -f $LOOP/.discinfo ]; then
            cp -av $LOOP/.discinfo $DVD
            CDS=`expr $CDS + 1`
            if [ $CDS != 1 ] ; then
                DISKS=`echo ${DISKS},${CDS}`
            fi
        fi
        umount $LOOP
    done
else
    if [ -f $1/isolinux/isolinux.bin ]; then

        echo "Found FTP-like tree..."

        if [ -e $1/.discinfo ]; then
            cp -av $1/.discinfo $DVD
        else
# How does one construct a legal .discinfo file if none is found?
            echo "Error: No .discinfo file found in $1"
            cleanup
            exit 1
        fi
        cp -av $1/* $DVD
    else
        echo "Error: No CD images nor FTP-like tree found in $1"
        cleanup
        exit 1
    fi
fi

if [ -e $DVD/.discinfo ]; then
    awk '{ if ( NR == 4 ) { print disks } else { print ; } }' disks="ALL" $DVD/.discinfo > $DVD/.discinfo.new
    mv $DVD/.discinfo.new $DVD/.discinfo
else
    echo  "Error: No .discinfo file found in $DVD"
    cleanup
    exit 1
fi

rm -rf $DVD/isolinux/boot.cat
find $DVD -name TRANS.TBL | xargs rm -f

cd $DVD
createrepo -g repodata/comps.xml ./
mkisofs -J -R -v -T -o $2 -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 8 -boot-info-table $DVD
if [ "$?" = 0 ]; then

    echo ""
    echo "Image complete, create md5sum..."

#  $IMPLANT --force $2
# Don't like forced mediacheck? Try this instead.
    $IMPLANT --supported-iso --force $2

    echo "Start cleanup..."

    cleanup

    echo ""
    echo "Process Complete!"
    echo "Wrote DVD ISO image to $DVD_DIR/$DVD_FILE"
    echo ""
else
    echo "ERROR: Image creation failed, start cleanup..."

    cleanup

    echo ""
    echo "Failed to create ISO image $DVD_DIR/$DVD_FILE"
    echo ""
fi

setting-up etherpad in centos 6

To add to my tutorial collection, here’s how to setup EtherPad on CentOS 6 (x64). As in the IRC tutorial, I used a Digital Ocean VM for this 🙂

What is EtherPad? It’s an open-source collaborative text editor that works like Google Docs – ie, all editors/viewers can see changes from everyone else in realtime.

Here’s how I did it: (props to the EtherPad docs and this other tutorial on Node.js)

  • acquire a CentOS server – I used Digital Ocean
  • make sure you have enough swap space (if you use the smallest Digital Ocean “droplet”, you really need to give yourself 2-4G swap)
    • dd if=/dev/zero of=/swapfile bs=8192 count=524288
    • mkswap swapfile
    • swapon swapfile
    • add this line to the end of your /etc/fstab:
      • /swapfile swap swap defaults 0 0
  • run the following as root:
    • yum -y install gzip git-core curl python openssl-devel && yum groupinstall
    • yum -y install screen gcc gcc-c++ make wget curl
      • note – it’s not always called “gcc-c++”; make sure you use the correct package name for your platform
      • you can find out the correct package name by doing a `yum provides */g++` search
    • yum -y upgrade
      • bring everything up to date – it’s equivalent to `yum -y update –obsoletes`
    • adduser etherpad
    • su – etherpad
    • git clone git://github.com/ry/node.git
    • cd node && ./configure && make && make install
    • cd
    • git clone git://github.com/ether/etherpad-lite.git
    • cd etherpad-lite
    • screen bin/run.sh
  • load http://127.0.0.1:9001 in your browser (substitute the IP/DNS name of your server as appropriate – mine was http://107.170.150.57:9001)
  • for maintenance purposes, you should also do a `git pull origin` for EtherPad periodically

Some notes on the above:

  • if you already have swap space and/or don’t want to worry about it (though I recommend you do), you can skip it
  • I’ve put the `-y` option after every `yum` call presuming you really mean to run it, and you don’t care about dependencies
    • if you aren’t using a fresh server (EtherPad certainly doesn’t require it’s own), you may want to be a little more cautious about the `yum` commands 🙂
  • you should create a start-up script to ensure EtherPad is running if you need to reboot
  • the EtherPad docs have all kinds of further things you can do – this is just a “get it going and start using it” tutorial 🙂

automatically extract email attachments with common linux tools

I had need to automatically process emails to a specific address to pull attachments out, and this is how I did it:

$ yum install mpack

$ cat extract-attach.sh 
#!/bin/bash
rm -rf ~/attachtmp
mkdir ~/attachtmp
mv ~/Maildir/new/* ~/attachtmp
cd ~
munpack ~/attachtmp/*
rm -rf ~/attachtmp

$ crontab -l
*/5 * * * *	~/extract-attach.sh

Why, you may ask? Because I get a report a few times per day to the email address in question.


Note – this runs in my crontab every 5 minutes on a CentOS 6 x64 server; I’m sure the process is similar/identical on other distros, but I haven’t personally tried.

setting up an unreal irc server on centos 6

Ever want to run an IRC server? I recently set one up at irc.datente.com using a Digital Ocean VM running CentOS 6.5 x64.

Here’s what I did, if you want to replicate the process for yourself (full documentation available from Unreal’s website):

  • acquire CentOS 6.5 x64 server (as I mentioned, I used Digital Ocean)
  • `yum -y install screen wget gcc`
  • `yum -y upgrade`
  • `adduser unreal`
  • `su – unreal`
  • download Unreal to your server (http://www.unrealircd.com/downloads/unreal/source – `wget http://www.unrealircd.com/downloads/Unreal3.2.10.2.tar.gz`)
  • `tar zxf Unreal*.gz`
  • `cd Unreal*`
  • `make clean`
  • `./Config`
    • answer prompts – most can be left default
  • `make`
  • `cp doc/example.conf unrealircd.conf`
  • edit unrealircd.conf (use your editor of choice)
    • see sample config file below for what I did (minus passwords / emails)
  • if all has gone well, start Unreal
    • `screen ./unreal start`
  • create a startup script to ensure Unreal launches on reboot as user `unreal`

That’s it. Thankfully, while the config file isn’t pleasant to play with, it’s a lot better than it used to be.

loadmodule "src/modules/commands.so";
loadmodule "src/modules/cloak.so";

include "help.conf";
include "badwords.channel.conf";
include "badwords.message.conf";
include "badwords.quit.conf";
include "spamfilter.conf";

me
{
        name "your.irc.server.tld";
        info "Your IRC Server";
        numeric 1;
};

admin {
        "Your Name";
        "yournick";
        "your@email.tld";
};

class           clients
{
        pingfreq 90;
        maxclients 500;
        sendq 100000;
        recvq 8000;
};

class           servers
{
        pingfreq 90;
        maxclients 10;          /* Max servers we can have linked at a time */
        sendq 1000000;
        connfreq 100; /* How many seconds between each connection attempt */
};

allow {
        ip             *@*;
        hostname       *@*;
        class           clients;
        maxperip 25;
};

/* Passworded allow line */
allow {
        ip             *@255.255.255.255;
        hostname       *@*.passworded.ugly.people;
        class           clients;
        password "f00Ness";
        maxperip 1;
};

allow channel {
        channel "#WarezSucks";
        class "clients";
};

oper youroperatornick {
        class           clients;
        from {
                userhost bob@smithco.com;
        };
        password "yourpassword";
        flags
        {
                netadmin;
                can_zline;
                can_gzline;
                can_gkline;
                global;
        };
};

listen         *:6697
{
        options
        {
// uncomment this line if you chose to compile Unreal with SSL support
//              ssl;
                clientsonly;
        };
};

listen         *:8067;
listen         *:6667;

/* not linking to any other servers right now
link            hub.mynet.com
{
        username        *;
        hostname        1.2.3.4;
        bind-ip         *;
        port            7029;
        hub             *;
        password-connect "LiNk";
        password-receive "LiNk";
        class           servers;
                options {
                        /* Note: You should not use autoconnect when linking services */
                        autoconnect;
                        ssl;
                        zip;
                };
};
*/

ulines {
        services.roxnet.org;
        stats.roxnet.org;
};

drpass {
        restart "I-love-to-restart";
        die "die-you-stupid";
};

log "ircd.log" {
        /* Delete the log file and start a new one when it reaches 20MB, leave this out to always use the 
           same log */
        maxsize 20971520;
        flags {
                oper;
                connects;
                server-connects;
                kills;
                errors;
                sadmin-commands;
                chg-commands;
                oper-override;
                spamfilter;
        };
};

alias NickServ { type services; };
alias ChanServ { type services; };
alias OperServ { type services; };
alias HelpServ { type services; };
alias StatServ { type stats; };

alias "identify" {
        format "^#" {
                target "chanserv";
                type services;
                parameters "IDENTIFY %1-";
        };
        format "^[^#]" {
                target "nickserv";
                type services;
                parameters "IDENTIFY %1-";
        };
        type command;
};

alias "services" {
        format "^#" {
                target "chanserv";
                type services;
                parameters "%1-";
        };
        format "^[^#]" {
                target "nickserv";
                type services;
                parameters "%1-";
        };
        type command;
};

alias "identify" {
        format "^#" {
                target "chanserv";
                type services;
                parameters "IDENTIFY %1-";
        };
        format "^[^#]" {
                target "nickserv";
                type services;
                parameters "IDENTIFY %1-";
        };
        type command;
};

alias "glinebot" {
        format ".+" {
                command "gline";
                type real;
                parameters "%1 2d Bots are not allowed on this server, please read the faq at http://www.example.com/faq/123";
        };
        type command;
};

files
{
        /* The Message Of The Day shown to users who log in: */
        /* motd ircd.motd; */

        /*
         * A short MOTD. If this file exists, it will be displayed to
         * the user in place of the MOTD. Users can still view the
         * full MOTD by using the /MOTD command.
         */
        /* shortmotd ircd.smotd; */

        /* Shown when an operator /OPERs up */
        /* opermotd oper.motd; */

        /* Services MOTD append. */
        /* svsmotd ircd.svsmotd; */

        /* Bot MOTD */
        /* botmotd bot.motd; */

        /* Shown upon /RULES */
        /* rules ircd.rules; */

        /*
         * Where the IRCd stores and loads a few values which should
         * be persistent across server restarts. Must point to an
         * existing file which the IRCd has permission to alter or to
         * a file in a folder within which the IRCd may create files.
         */
        /* tunefile ircd.tune; */

        /* Where to save the IRCd's pid. Should be writable by the IRCd. */
        /* pidfile ircd.pid; */
};

/*
tld {
        mask *@*.fr;
        motd "ircd.motd.fr";
        rules "ircd.rules.fr";
};
*/

/* note: you can just delete the example block above,
 * in which case the defaults motd/rules files (ircd.motd, ircd.rules)
 * will be used for everyone.
 */

ban nick {
        mask "*C*h*a*n*S*e*r*v*";
        reason "Reserved for Services";
};

ban ip {
        mask 195.86.232.81;
        reason "Delinked server";
};

ban server {
        mask eris.berkeley.edu;
        reason "Get out of here.";
};

ban user {
        mask *tirc@*.saturn.bbn.com;
        reason "Idiot";
};

ban realname {
        mask "sub7server";
        reason "sub7";
};

except ban {
        /* don't ban stskeeps */
        mask           *stskeeps@212.*;
};

deny dcc {
        filename "*sub7*";
        reason "Possible Sub7 Virus";
};

deny channel {
        channel "*warez*";
        reason "Warez is illegal";
        class "clients";
};

vhost {
        vhost           i.hate.microsefrs.com;
        from {
                userhost       *@*.image.dk;
        };
        login           stskeeps;
        password        moocowsrulemyworld;
};

set {
        network-name            "Datente";
        default-server          "irc.datente.com";
        services-server         "irc.datente.com";
        stats-server            "irc.datente.com";
        help-channel            "#datente";
        hiddenhost-prefix       "rox";
        /* prefix-quit          "no"; */
        /* Cloak keys should be the same at all servers on the network.
         * They are used for generating masked hosts and should be kept secret.
         * The keys should be 3 random strings of 5-100 characters
         * (10-20 chars is just fine) and must consist of lowcase (a-z),
         * upcase (A-Z) and digits (0-9) [see first key example].
         * HINT: On *NIX, you can run './unreal gencloak' in your shell to let
         *       Unreal generate 3 random strings for you.
         */
        cloak-keys {
                "aoAr1HnR6gl3sJ7hVz4Zb7x4YwpW";
                "aaAr1HnR6gl3sJ7hVz4Zb7x4YwpW";
                "aeAr1HnR6gl3sJ7hVz4Zb7x4YwpW";
        };
        /* on-oper host */
        hosts {
                local           "locop.roxnet.org";
                global          "ircop.roxnet.org";
                coadmin         "coadmin.roxnet.org";
                admin           "admin.roxnet.org";
                servicesadmin   "csops.roxnet.org";
                netadmin        "netadmin.roxnet.org";
                host-on-oper-up "no";
        };
};

set {
        kline-address "your@email.tld";
        modes-on-connect "+ixw";
        modes-on-oper    "+xwgs";
        oper-auto-join "#opers";
        options {
                hide-ulines;
                /* You can enable ident checking here if you want */
                /* identd-check; */
                show-connect-info;
        };

        maxchannelsperuser 10;
        /* The minimum time a user must be connected before being allowed to use a QUIT message,
         * This will hopefully help stop spam */
        anti-spam-quit-message-time 10s;
        /* Make the message in static-quit show in all quits - meaning no
           custom quits are allowed on local server */
        /* static-quit "Client quit";   */

        /* You can also block all part reasons by uncommenting this and say 'yes',
         * or specify some other text (eg: "Bye bye!") to always use as a comment.. */
        /* static-part yes; */

        /* This allows you to make certain stats oper only, use * for all stats,
         * leave it out to allow users to see all stats. Type '/stats' for a full list.
         * Some admins might want to remove the 'kGs' to allow normal users to list
         * klines, glines and shuns.
         */
        oper-only-stats "okfGsMRUEelLCXzdD";

        /* Throttling: this example sets a limit of 3 connection attempts per 60s (per host). */
        throttle {
                connections 3;
                period 60s;
        };

        /* Anti flood protection */
        anti-flood {
                nick-flood 3:60;        /* 3 nickchanges per 60 seconds (the default) */
        };

        /* Spam filter */
        spamfilter {
                ban-time 1d; /* default duration of a *line ban set by spamfilter */
                ban-reason "Spam/Advertising"; /* default reason */
                virus-help-channel "#help"; /* channel to use for 'viruschan' action */
                /* except "#help"; channel to exempt from filtering */
        };
};


defaulting pxe boots with hpsa 10.0

In follow-up to my last post, which itself was a commentary on an earlier topic, I have the additional steps you need to do the previous procude (which is to edit /opt/opsware/boot/tftpboot/pxelinux.cfg/default):

/etc/init.d/opsware-sas stop smartboot

Edit file.

/etc/init.d/opsware-sas start smartboot

network install of centos 6

I wanted to try something different when playing with CentOS 6 recently, so I did a network install. Other than one very small detail, the install is identical to installing off a normal ISO.

Here’s the difference:

  1. use the netinstall.iso (eg http://centos.datente.com/media/6/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)
  2. when it asks for the URL to grab your image from, use something like http://centos.datente.com/media/6/os/x86_64 (make sure it’s the full path (this example is off my mirror for an x64 install))
  3. proceed as usual 🙂

automatically returning a host to the unprovisioned server pool in hpsa

In conjunction with the customized PXE process I wrote about previously, it could be highly desirable to be able to return a server to the unprovisioned server pool in HP’s Server Automation.

This is a specifically-Linux procedure: though I’m sure something similar can be done with Windows*.

run an ad-hoc script against a target server that contains the following:

dd if=/dev/zero of=/dev/sda bs=512 count=1
sleep 1
nohup reboot

This will erase the MBR and partition table, and then reboot the server.

Before it reboots, however, you need to deactivate and delete the server from SA – otherwise it will not register correctly.

If you’ve already enabled (or not disabled) PXE booting, when it reboots, it will pick the default entry off the PXE menu, skipping the hard drive as there is no valid boot record available to it.

Why would you want to do this?

Well, let’s say you’re doing a lot of build testing (verifying ks.cfg or unattend.xml files, for example) – this could be useful.

Or, maybe you want to get your build process completely streamlined and you’re working with the MBC functionality in SA – again, rapid recycling of machines is highly desirable.

In a later post I’ll discuss freeing the VM from SA in the process (ie, removing it from the ESXi host to fully release resources).


*In fact, you may be able to run fdisk /mbr on a Windows server – but I haven’t tried.