Tag Archives: wordpress

bglug, technical, warning

keep your wordpress installs up-to-date

3 Comments

I run several websites on my server – nothing heavy, just some various vhosts for Apache.

Many (but not all) of them run WordPress.

At some unknown point (and I haven’t kept the crap that was being used around), over 100,000 files were uploaded to the root directory of one of the websites (the only one, apparently, I did not have cron’d to keep up-to-date with the latest-and-greatest version of WordPress) – most of these were random-named HTML or JavaScript files. Sometime late Thursday night / early Friday morning of last week, some number of those were triggered which launched a DDoS (distributed denial-of-service) attack against a hosting company in England.

After a relatively short period of time (on the order of a couple hours at most), this otherwise-low-traffic site generated 48MB in Apache httpd logs (normal for a given day is on the order of a few dozen to couple hundred kilobytes).

My hosting provider, with no warning, “locked” my server, and sent me an administrative message with the following cryptic email:

Your server with the above-mentioned IP address has carried out an attack on another server on the Internet.

This has placed a considerable strain on network resources and, as a result, a segment of our network has been adversely affected.

Your server has therefore been deactivated as a precautionary measure.

A corresponding log history is attached at the end of this email.

10:00:21.645887 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.646166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649416 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74 > 85.233.160.139: ip-proto-17
10:00:21.649421 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: 176.9.40.74.54988 > 85.233.160.139.8888: UDP, length 8192

Gee, thanks, hosting company – that was informative.

After several hours of back-and-forth with their support group, I was finally able to get a rescue boot environment enabled, a KVM session to that environment, and could start diagnosing the problem(s). First, of course, were the normal checks of dmesg, /var/log/messages, and the like. there was running dig to find out who was being attacked (how I found the target IP belonged to a hosting provider in the UK). Nothing. I was also Googling similar error messages, and finally found a clue (though cannot recall where) that malicious JavaScript can cause messages like those provided to me to be trapped by external logging systems.

This led me to look in /var/log/httpd instead of just /var/log. And there is where I found the unusual log file for my LUG’s website here in Kentucky – bglug-access_log was 48 megabytes. And bglug-error_log was 4.3 MB. As I mentioned above, a typical access_log for that site is closer to ~100 KB.

Opening the ginormous log file showed a host of HTTP 200 response codes for things that looked nothing like WordPress files (things like “qdlrdi-casio-parliament-90treaty.html”). There shouldn’t be HTTP 200 (OK) response codes for non-WordPress files, because it’s a WordPress-powered website.

Running a file listing to screen failed (in the rescue boot environment) – but doing an ls -l > files.out, and then a wc -l files.out showed over 105,000 files in the root directory of the BGLUG website.

To get my server back up and online as quickly as possible, I edited the Apache vhosts.conf and disabled the Blue Grass Linux User Group site and contacted my hosting company as to what the root cause of the issue was, and what I had done to fix it (both needed for them to reenable my system).

After getting the server back online normally, I was able to clear-out all the junk that had been transparently uploaded into the LUG’s site.

One of the biggest annoyances of the whole process (after not having been given any warning from my hosting provider, but just a summary disconnect) was that permissions on the directory for the website were “correct” to have disallowed uploading random junk to the server:
drwxr-xr-x 6 bglug apache 5611520 Apr 11 13:24 bglug

The user bglug had not been compromised (it hasn’t even logged-in in a few months) – and neither was the apache group (which, of course, cannot login, but still).

Apparently, some part of the version of WordPress the site was running (or a plugin) was compromised, and allowed a malicious attacker to upload junk to the server, and spawn this DDoS on my server.

Moral of the story? Keep all your software up-to-date, and monitor your logs for suspicious activity – not sure monitoring would’ve done me good in this case, but it’s a Good Practice™ anyway.

commentary, education, ideas, insights, technical, work

organizational knowledge capture, retention, and dissemination

3 Comments

Knowledge capture, retention, and dissemination has been an interest of mine for a long time. I have written about various aspects of it before.

The most vital commodity any organization has is the knowledge of its members – it does not matter if it is a historical society, company, church, or school: the organizational knowledge base is vital to ongoing health of the organization.

I love the picture of the “Tree of Wisdom“: at the ground there is a meadow of data, from this data information roots are gathered, the roots grow into knowledge branches, and at the end is the application of that knowledge in wisdom leaves.

Data is easy to come by.

Information similarly so.

Knowledge, taking information and transforming it into a more-usable form, is important.

When to apply that knowledge – aka using wisdom – is the topic for another post.

Capturing Knowledge

There are a host of available tools for capturing knowledge – text files, brown bags, PowerPoint, SharePoint, blogs, Plone, wikis, etc. The “best” one to use is the one you use.

Culture

Getting team members to contribute to organizational knowledge pools can be difficult – unless it is an organizational priority .. a part of the organization’s culture.

Incorporating this culture switch (if it’s not already innate to the organization) needs to be done not merely as a top-down directive, but encouraged via bottom-up interest.

Retaining (Managing) Knowledge

Now that you’ve captured (or started capturing) the organization’s data, managing it becomes the next task of import.

For example, should the KB article written 5 years ago be updated, replaced, or left alone?

Who is responsible for managing all of the information that has been collected? Will it be self-managed and -directed, will there be a curation team, will it be a combination?

Who determines the process for taking “internal” knowledge and “promoting” it to “outside” knowledge?

How are these roles going to be managed as the team changes memberships through people leaving, entering, and shifting in the organization?

For extremely small organizations, formal curation may be unnecessary. Perhaps since everyone knows everyone else, or the knowledge domain is so small, everyone’s individual contributions will remain fairly static and the “promotion” path will merely be proofreading (eg a historical society’s archives – the archives may be extensive, but the material doesn’t ‘change’ all that much (excepting being added-to, of course)).

For very big organizations (like the MSDN documentation available on microsoft.com), many layers of curation are likely going to be needed – proofreading, formatting, verifying, etc.

Finding the right balance of self-direction and organizational management can be tricky.

Disseminating Knowledge – Getting The Word Out

All of the captured knowledge in the world is useless if you can’t find it – and knowing where to look is vital. A close second to knowing where to look is how to find it.

Where is it?

There needs to be a solid document, landing page, directory, table of contents, etc so that new members (or folks who forget) can find the tribal knowledge that exists in the organization.

As a part of the new-hire\introduction\etc process\period, be sure to tell new members where information can be found, and who to talk to about certain major topics.

Finding it once you know where to look

“Search is a hard problem.” Google’s own Udi Manber said that. Anna Paterson at Stanford wrote, “Writing Your Own Search Engine Is Hard.”

Search in general may be hard, but many tools handle at least basic (and some fuzzy) searching well – OSQA, WordPress, Plone, Drupal, and many others. If, in addition to categorization, a tag taxonomy is employed, quickly finding content relevant to the searcher’s wants\needs can become easier.

“A tag is a keyword or label that categorizes your question with other, similar questions. Using the right tags makes it easier for others to find and answer your question.” {SO description}

Knowledge contributors should be the primary agents of tagging. However, consumers should be able to suggest additional tags. Administrators\curators should be able (under unusual, but well-defined, circumstances) to remove tags.

The human factor

For any given topic / knowledge region in the organization’s realm, there need to be established “experts” and “mentors” who will help guide new individuals through the fog to locate the buoys to be able to navigate themselves into a clearer understanding of the new world they have been made a part of.

Apprenticing upcoming experts into the organization is the single most vital aspect of the knowledge capture process – if it is not disseminated, it doesn’t matter if it is captured.