fighting the lack of good ideas
We will be meeting at the Beaumont Library Branch at 6:30p on 11 May.
Our speaker is the LUG’s own Nathaniel McCallum, one of the FreeIPA maintainers – and all-around nice guy.
Come out and support the LUG, learn something new, and meet cool people.
I run several websites on my server – nothing heavy, just some various vhosts for Apache.
Many (but not all) of them run WordPress.
At some unknown point (and I haven’t kept the crap that was being used around), over 100,000 files were uploaded to the root directory of one of the websites (the only one, apparently, I did not have cron’d to keep up-to-date with the latest-and-greatest version of WordPress) – most of these were random-named HTML or JavaScript files. Sometime late Thursday night / early Friday morning of last week, some number of those were triggered which launched a DDoS (distributed denial-of-service) attack against a hosting company in England.
After a relatively short period of time (on the order of a couple hours at most), this otherwise-low-traffic site generated 48MB in Apache httpd logs (normal for a given day is on the order of a few dozen to couple hundred kilobytes).
My hosting provider, with no warning, “locked” my server, and sent me an administrative message with the following cryptic email:
Your server with the above-mentioned IP address has carried out an attack on another server on the Internet.
This has placed a considerable strain on network resources and, as a result, a segment of our network has been adversely affected.
Your server has therefore been deactivated as a precautionary measure.
A corresponding log history is attached at the end of this email.
10:00:21.645887 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.646166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649416 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649421 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74.54988 > 85.233.160.139.8888: UDP, length 8192
Gee, thanks, hosting company – that was informative.
After several hours of back-and-forth with their support group, I was finally able to get a rescue boot environment enabled, a KVM session to that environment, and could start diagnosing the problem(s). First, of course, were the normal checks of dmesg, /var/log/messages, and the like. there was running dig to find out who was being attacked (how I found the target IP belonged to a hosting provider in the UK). Nothing. I was also Googling similar error messages, and finally found a clue (though cannot recall where) that malicious JavaScript can cause messages like those provided to me to be trapped by external logging systems.
This led me to look in /var/log/httpd instead of just /var/log. And there is where I found the unusual log file for my LUG’s website here in Kentucky – bglug-access_log was 48 megabytes. And bglug-error_log was 4.3 MB. As I mentioned above, a typical access_log for that site is closer to ~100 KB.
Opening the ginormous log file showed a host of HTTP 200 response codes for things that looked nothing like WordPress files (things like “qdlrdi-casio-parliament-90treaty.html”). There shouldn’t be HTTP 200 (OK) response codes for non-WordPress files, because it’s a WordPress-powered website.
Running a file listing to screen failed (in the rescue boot environment) – but doing an ls -l > files.out, and then a wc -l files.out showed over 105,000 files in the root directory of the BGLUG website.
To get my server back up and online as quickly as possible, I edited the Apache vhosts.conf and disabled the Blue Grass Linux User Group site and contacted my hosting company as to what the root cause of the issue was, and what I had done to fix it (both needed for them to reenable my system).
After getting the server back online normally, I was able to clear-out all the junk that had been transparently uploaded into the LUG’s site.
One of the biggest annoyances of the whole process (after not having been given any warning from my hosting provider, but just a summary disconnect) was that permissions on the directory for the website were “correct” to have disallowed uploading random junk to the server:
drwxr-xr-x 6 bglug apache 5611520 Apr 11 13:24 bglug
The user bglug had not been compromised (it hasn’t even logged-in in a few months) – and neither was the apache group (which, of course, cannot login, but still).
Apparently, some part of the version of WordPress the site was running (or a plugin) was compromised, and allowed a malicious attacker to upload junk to the server, and spawn this DDoS on my server.
Moral of the story? Keep all your software up-to-date, and monitor your logs for suspicious activity – not sure monitoring would’ve done me good in this case, but it’s a Good Practice™ anyway.
Attached is the presentation for my talk on initial CentOS/RHEL 6.x server configuring.
I am now running an official Haiku mirror: haiku.datente.com.
Alpha 4.1 has been released, and you can get a copy from any of the mirrors.
After several months, the Bluegrass Linux User Group (see also G+ page) is resuming regular meetings in conjunction With Collexion – a hackerspace in Lexington KY.
Our first (next) topic is “The Raspberry Pi“, presented by Jordan Keyes (creator of twil.tv (also on youtube: http://youtube.com/twildottv)), at 1845 (6:45p) at Collexion. We’ll plan to “start” at 1900 (7p).
Collexion is located 109 E. Loudon Ave in Lexington.
We are looking forward to a good time being had by all.
Please bring a small donation for dinner (planning to order-in pizza) – any leftover cash will be left as a “thank you” to the kind folks at Collexion for letting us use their space.
Also to note: the new regular meeting day will be the third Thursday of each month.
It’s been several months, but we are planning to have another Bluegrass Linux User Group meeting next month!
It looks like there’s a good space to meet near downtown, so while the exact Saturday has not been picked yet, we are looking to reboot the LUG in a few weeks.
If anyone has suggestions for topics, speakers, etc … please leave a comment. If you’re able to give the talk – please tell us that, too 🙂
August Schell Recent Paragraphs
