Tag Archives: linux

improve your entropy pool in linux

A few years ago, I ran into a known issue with one of the products I use that manifests when the Red Hat Linux server it’s running on has a low entropy pool. And, as highlighted in that question, the steps I found 5 years ago didn’t work for me (turns out modifying the t parameter from ‘1’ to ‘.1’ did work (rngd -r /dev/urandom -o /dev/random -f -t .1), but I digress (and it’s no longer correct in CentOS 7 (the ‘t’ option, that is))).

In playing around with the Mozilla-provided SSL configurator, I noticed a line in the example SSL config that referenced “truerand”. After a little Googling, I found an opensource implementation called “twuewand“.

And a little more Googling about adding entropy, and I came across this interesting tutorial from Digital Ocean for “haveged” (which, interestingly-enough, allowed me to answer a 6-month-old question on Server Fault about CloudLinux).

Haveged “is an attempt to provide an easy-to-use, unpredictable random number generator based upon an adaptation of the HAVEGE algorithm. Haveged was created to remedy low-entropy conditions in the Linux random device that can occur under some workloads, especially on headless servers.”

And twuewand “is software that creates hardware-generated random data. It accomplishes this by exploiting the fact that the CPU clock and the RTC (real-time clock) are physically separate, and that time and work are not linked.”

For workloads that require lots of entropy (generating SSL keys, SSH keys, PGP keys, and pretty much anything else that wants lots of random (or strong pseudorandom) seeding), the very real problem of running out of entropy (especially on headless boxes or virtual machines) is something you can face quite easily / frequently.

Enter solutions like OpenRNG which are hardware entropy generators (that one is a USB dongle (see also this skh-tec post)). Those are awesome – unless you’re running in cloud space somewhere, or even just a “traditional” virtual machine.

One of the funny things about getting “random” data is that it’s actually very very hard to get. It’s easy to describe, but generating “truly” random data is incredibly difficult. (If you want to have an aneurysm (or you’re like me and think this stuff is unendingly fascinating), go read the Wikipedia entry on “Cryptographically Secure Pseudo Random Number Generator“.)

If you’re in a situation, though, like I was (and still am), where you need to maintain a relatively high quantity of fairly decent entropy (probably close to CSPRNG level), use haveged. And run twuewand occasionally – at the very least when starting Apache (at least if you’re running HTTPS – which you should be, since it’s so easy now).

knoppix remastering virtual appliance

In preparation for an upcoming post on remastering Knoppix, I have made a VirtualBox virtual appliance based on the Knoppix v7.6.1 DVD all set for remastering.

/dev/sda holds the raw files.

/dev/sdb1 is a swap partition.

To use the appliance, download the Knoppix 7.6.1 DVD. You’ll need a boot environment for remastering, and Knoppix has the tools you need to remaster it.

Make sure you mount /dev/sda somewhere memorable. And that you run a swapon /dev/sdb1 before you start.

Have fun.

Download the .ova appliance from me here.

keep your wordpress installs up-to-date

I run several websites on my server – nothing heavy, just some various vhosts for Apache.

Many (but not all) of them run WordPress.

At some unknown point (and I haven’t kept the crap that was being used around), over 100,000 files were uploaded to the root directory of one of the websites (the only one, apparently, I did not have cron’d to keep up-to-date with the latest-and-greatest version of WordPress) – most of these were random-named HTML or JavaScript files. Sometime late Thursday night / early Friday morning of last week, some number of those were triggered which launched a DDoS (distributed denial-of-service) attack against a hosting company in England.

After a relatively short period of time (on the order of a couple hours at most), this otherwise-low-traffic site generated 48MB in Apache httpd logs (normal for a given day is on the order of a few dozen to couple hundred kilobytes).

My hosting provider, with no warning, “locked” my server, and sent me an administrative message with the following cryptic email:

Your server with the above-mentioned IP address has carried out an attack on another server on the Internet.

This has placed a considerable strain on network resources and, as a result, a segment of our network has been adversely affected.

Your server has therefore been deactivated as a precautionary measure.

A corresponding log history is attached at the end of this email.

10:00:21.645887 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.646166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649416 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649421 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74.54988 > 85.233.160.139.8888: UDP, length 8192

Gee, thanks, hosting company – that was informative.

After several hours of back-and-forth with their support group, I was finally able to get a rescue boot environment enabled, a KVM session to that environment, and could start diagnosing the problem(s). First, of course, were the normal checks of dmesg, /var/log/messages, and the like. there was running dig to find out who was being attacked (how I found the target IP belonged to a hosting provider in the UK). Nothing. I was also Googling similar error messages, and finally found a clue (though cannot recall where) that malicious JavaScript can cause messages like those provided to me to be trapped by external logging systems.

This led me to look in /var/log/httpd instead of just /var/log. And there is where I found the unusual log file for my LUG’s website here in Kentucky – bglug-access_log was 48 megabytes. And bglug-error_log was 4.3 MB. As I mentioned above, a typical access_log for that site is closer to ~100 KB.

Opening the ginormous log file showed a host of HTTP 200 response codes for things that looked nothing like WordPress files (things like “qdlrdi-casio-parliament-90treaty.html”). There shouldn’t be HTTP 200 (OK) response codes for non-WordPress files, because it’s a WordPress-powered website.

Running a file listing to screen failed (in the rescue boot environment) – but doing an ls -l > files.out, and then a wc -l files.out showed over 105,000 files in the root directory of the BGLUG website.

To get my server back up and online as quickly as possible, I edited the Apache vhosts.conf and disabled the Blue Grass Linux User Group site and contacted my hosting company as to what the root cause of the issue was, and what I had done to fix it (both needed for them to reenable my system).

After getting the server back online normally, I was able to clear-out all the junk that had been transparently uploaded into the LUG’s site.

One of the biggest annoyances of the whole process (after not having been given any warning from my hosting provider, but just a summary disconnect) was that permissions on the directory for the website were “correct” to have disallowed uploading random junk to the server:
drwxr-xr-x 6 bglug apache 5611520 Apr 11 13:24 bglug

The user bglug had not been compromised (it hasn’t even logged-in in a few months) – and neither was the apache group (which, of course, cannot login, but still).

Apparently, some part of the version of WordPress the site was running (or a plugin) was compromised, and allowed a malicious attacker to upload junk to the server, and spawn this DDoS on my server.

Moral of the story? Keep all your software up-to-date, and monitor your logs for suspicious activity – not sure monitoring would’ve done me good in this case, but it’s a Good Practice™ anyway.

merging centos iso images

Thanks to @Anon on Unix.SE for the pointer on how to do this. And to @Andy‘s comment on @mmckinst‘s answer for the warning about additional packages you may need.

As my three readers know, I run a CentOS mirror. One of the idiosyncrasies of CentOS, like its upstream RHEL, is that DVD ISOs aren’t always just one image – for example, the 6.6 x64 image comes on two ISOs. I suppose this has something to do with the “normal” or “simple” capacity of a DVD disc, but it’s annoying.

Enter the mkdvdiso.sh script (original found here) from Chris Kloiber & Phil Schaffner.

The process I used to combine these two ISOs into one is as follows:
yum install isomd5sum createrepo mkisofs
mkdvdiso.sh /full/path/to/original/isos /full/path/to/destination.iso

For posterity, and in case the CentOS wiki dies, below is the mkdvdiso.sh script:

#!/bin/bash

# by Chris Kloiber 
# Mods under CentOS by Phil Schaffner 

# A quick hack that will create a bootable DVD iso of a Red Hat Linux
# Distribution. Feed it either a directory containing the downloaded
# iso files of a distribution, or point it at a directory containing
# the "RedHat", "isolinux", and "images" directories.

# This version only works with "isolinux" based Red Hat Linux versions.

# Lots of disk space required to work, 3X the distribution size at least.

# GPL version 2 applies. No warranties, yadda, yadda. Have fun.

# Modified to add sanity checks and fix CentOS4 syntax errors

# TODO:
#   Add checks for available disk space on devices holding output and
#       temp files.
#   Add optional 3rd parameter to specify location of temp directory.
#   Create .discinfo if not present.

OS_VER=\
$((test -e /etc/fedora-release && rpm -qf /etc/fedora-release --qf "FC%{VERSION}") \
|| (test -e /etc/redhat-release && rpm -qf /etc/redhat-release --qf "EL%{VERSION}") \
|| echo OS_unknown)

case "$OS_VER" in
  EL[45]*|FC?)
        IMPLANT=/usr/lib/anaconda-runtime/implantisomd5
        if [ ! -f $IMPLANT ]; then
            echo "Error: $IMPLANT Not Found!"
            echo "Please install anaconda-runtime and try again."
            exit 1
        fi
        ;;
  EL6*|FC1?)
        IMPLANT=/usr/bin/implantisomd5
        if [ ! -f $IMPLANT ]; then
            echo "Error: $IMPLANT Not Found!"
            echo "Please install isomd5sum and try again."
            exit 1
        fi
        ;;
  OS_unknown)
        echo "Unknown OS."
        exit 1
        ;;
  *)
        echo "Fix this script for $OS_VER"
        exit 1
esac

if [ $# -lt 2 ]; then
        echo "Usage: `basename $0` source /destination/DVD.iso"
        echo ""
        echo "        The 'source' can be either a directory containing a single"
        echo "        set of isos, or an exploded tree like an ftp site."
        exit 1
fi

DVD_DIR=`dirname $2`
DVD_FILE=`basename $2`

echo "DVD directory is $DVD_DIR"
echo "ISO file is $DVD_FILE"

if [ "$DVD_DIR" = "." ]; then
    echo "Destinaton Directory $DVD_DIR does not exist"
    exit 1
else
    if [ ! -d "/$DVD_DIR" ]; then
        echo "Destinaton Directory $DVD_DIR must be an absolute path"
        exit 1
    else
        if [ "$DVD_FILE" = "" ] || [ -d "$DVD_DIR/$DVD_FILE" ]; then
            echo "Null ISO file name."
            exit 1
        fi
    fi
fi

which mkisofs >&/dev/null
if [ "$?" != 0 ]; then
    echo "mkisofs Not Found"
    echo "yum install mkisofs"
fi

which createrepo >&/dev/null
if [ "$?" != 0 ]; then
    echo "createrepo Not Found"
    echo "yum install createrepo"
fi

if [ -f $2 ]; then
    echo "DVD ISO destination $2 already exists. Remove first to recreate."
    exit 1
fi

# Make sure there is enough free space to hold the DVD image on the filesystem
# where the home directory resides, otherwise change ~/mkrhdvd to point to
# a filesystem with sufficient free space.

cleanup() {
    [ ${LOOP:=/tmp/loop} = "/" ] && echo "LOOP mount point = \/, dying!" && exit
    [ -d $LOOP ] && rm -rf $LOOP 
    [ ${DVD:=~/mkrhdvd} = "/" ] && echo "DVD data location is \/, dying!" && exit
    [ -d $DVD ] && rm -rf $DVD 
}

cleanup
mkdir -p $LOOP
mkdir -p $DVD

ls $1/*.iso &>/dev/null
if [ "$?" = 0 ]; then

    echo "Found ISO CD images..."

    CDS=`expr 0`
    DISKS="1"

    [ -w / ] || {   # Very portable, but perhaps not perfect, test for superuser.
        echo "Only 'root' may use this script for loopback mounts" 1>&2
        exit 1
    }

    for f in `ls $1/*.iso`; do
        mount -o loop $f $LOOP
        cp -av $LOOP/* $DVD
        if [ -f $LOOP/.discinfo ]; then
            cp -av $LOOP/.discinfo $DVD
            CDS=`expr $CDS + 1`
            if [ $CDS != 1 ] ; then
                DISKS=`echo ${DISKS},${CDS}`
            fi
        fi
        umount $LOOP
    done
else
    if [ -f $1/isolinux/isolinux.bin ]; then

        echo "Found FTP-like tree..."

        if [ -e $1/.discinfo ]; then
            cp -av $1/.discinfo $DVD
        else
# How does one construct a legal .discinfo file if none is found?
            echo "Error: No .discinfo file found in $1"
            cleanup
            exit 1
        fi
        cp -av $1/* $DVD
    else
        echo "Error: No CD images nor FTP-like tree found in $1"
        cleanup
        exit 1
    fi
fi

if [ -e $DVD/.discinfo ]; then
    awk '{ if ( NR == 4 ) { print disks } else { print ; } }' disks="ALL" $DVD/.discinfo > $DVD/.discinfo.new
    mv $DVD/.discinfo.new $DVD/.discinfo
else
    echo  "Error: No .discinfo file found in $DVD"
    cleanup
    exit 1
fi

rm -rf $DVD/isolinux/boot.cat
find $DVD -name TRANS.TBL | xargs rm -f

cd $DVD
createrepo -g repodata/comps.xml ./
mkisofs -J -R -v -T -o $2 -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 8 -boot-info-table $DVD
if [ "$?" = 0 ]; then

    echo ""
    echo "Image complete, create md5sum..."

#  $IMPLANT --force $2
# Don't like forced mediacheck? Try this instead.
    $IMPLANT --supported-iso --force $2

    echo "Start cleanup..."

    cleanup

    echo ""
    echo "Process Complete!"
    echo "Wrote DVD ISO image to $DVD_DIR/$DVD_FILE"
    echo ""
else
    echo "ERROR: Image creation failed, start cleanup..."

    cleanup

    echo ""
    echo "Failed to create ISO image $DVD_DIR/$DVD_FILE"
    echo ""
fi

setting-up etherpad in centos 6

To add to my tutorial collection, here’s how to setup EtherPad on CentOS 6 (x64). As in the IRC tutorial, I used a Digital Ocean VM for this 🙂

What is EtherPad? It’s an open-source collaborative text editor that works like Google Docs – ie, all editors/viewers can see changes from everyone else in realtime.

Here’s how I did it: (props to the EtherPad docs and this other tutorial on Node.js)

  • acquire a CentOS server – I used Digital Ocean
  • make sure you have enough swap space (if you use the smallest Digital Ocean “droplet”, you really need to give yourself 2-4G swap)
    • dd if=/dev/zero of=/swapfile bs=8192 count=524288
    • mkswap swapfile
    • swapon swapfile
    • add this line to the end of your /etc/fstab:
      • /swapfile swap swap defaults 0 0
  • run the following as root:
    • yum -y install gzip git-core curl python openssl-devel && yum groupinstall
    • yum -y install screen gcc gcc-c++ make wget curl
      • note – it’s not always called “gcc-c++”; make sure you use the correct package name for your platform
      • you can find out the correct package name by doing a `yum provides */g++` search
    • yum -y upgrade
      • bring everything up to date – it’s equivalent to `yum -y update –obsoletes`
    • adduser etherpad
    • su – etherpad
    • git clone git://github.com/ry/node.git
    • cd node && ./configure && make && make install
    • cd
    • git clone git://github.com/ether/etherpad-lite.git
    • cd etherpad-lite
    • screen bin/run.sh
  • load http://127.0.0.1:9001 in your browser (substitute the IP/DNS name of your server as appropriate – mine was http://107.170.150.57:9001)
  • for maintenance purposes, you should also do a `git pull origin` for EtherPad periodically

Some notes on the above:

  • if you already have swap space and/or don’t want to worry about it (though I recommend you do), you can skip it
  • I’ve put the `-y` option after every `yum` call presuming you really mean to run it, and you don’t care about dependencies
    • if you aren’t using a fresh server (EtherPad certainly doesn’t require it’s own), you may want to be a little more cautious about the `yum` commands 🙂
  • you should create a start-up script to ensure EtherPad is running if you need to reboot
  • the EtherPad docs have all kinds of further things you can do – this is just a “get it going and start using it” tutorial 🙂