fighting the lack of good ideas

keep your wordpress installs up-to-date

I run several websites on my server – nothing heavy, just some various vhosts for Apache.

Many (but not all) of them run WordPress.

At some unknown point (and I haven’t kept the crap that was being used around), over 100,000 files were uploaded to the root directory of one of the websites (the only one, apparently, I did not have cron’d to keep up-to-date with the latest-and-greatest version of WordPress) – most of these were random-named HTML or JavaScript files. Sometime late Thursday night / early Friday morning of last week, some number of those were triggered which launched a DDoS (distributed denial-of-service) attack against a hosting company in England.

After a relatively short period of time (on the order of a couple hours at most), this otherwise-low-traffic site generated 48MB in Apache httpd logs (normal for a given day is on the order of a few dozen to couple hundred kilobytes).

My hosting provider, with no warning, “locked” my server, and sent me an administrative message with the following cryptic email:

Your server with the above-mentioned IP address has carried out an attack on another server on the Internet.

This has placed a considerable strain on network resources and, as a result, a segment of our network has been adversely affected.

Your server has therefore been deactivated as a precautionary measure.

A corresponding log history is attached at the end of this email.

10:00:21.645887 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: > ip-proto-17
10:00:21.646166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: > ip-proto-17
10:00:21.649166 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: > ip-proto-17
10:00:21.649416 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: > ip-proto-17
10:00:21.649421 14:da:e9:b3:97:dc > 28:c0:da:46:26:0d, ethertype IPv4 (0x0800), length 1514: > UDP, length 8192

Gee, thanks, hosting company – that was informative.

After several hours of back-and-forth with their support group, I was finally able to get a rescue boot environment enabled, a KVM session to that environment, and could start diagnosing the problem(s). First, of course, were the normal checks of dmesg, /var/log/messages, and the like. there was running dig to find out who was being attacked (how I found the target IP belonged to a hosting provider in the UK). Nothing. I was also Googling similar error messages, and finally found a clue (though cannot recall where) that malicious JavaScript can cause messages like those provided to me to be trapped by external logging systems.

This led me to look in /var/log/httpd instead of just /var/log. And there is where I found the unusual log file for my LUG’s website here in Kentucky – bglug-access_log was 48 megabytes. And bglug-error_log was 4.3 MB. As I mentioned above, a typical access_log for that site is closer to ~100 KB.

Opening the ginormous log file showed a host of HTTP 200 response codes for things that looked nothing like WordPress files (things like “qdlrdi-casio-parliament-90treaty.html”). There shouldn’t be HTTP 200 (OK) response codes for non-WordPress files, because it’s a WordPress-powered website.

Running a file listing to screen failed (in the rescue boot environment) – but doing an ls -l > files.out, and then a wc -l files.out showed over 105,000 files in the root directory of the BGLUG website.

To get my server back up and online as quickly as possible, I edited the Apache vhosts.conf and disabled the Blue Grass Linux User Group site and contacted my hosting company as to what the root cause of the issue was, and what I had done to fix it (both needed for them to reenable my system).

After getting the server back online normally, I was able to clear-out all the junk that had been transparently uploaded into the LUG’s site.

One of the biggest annoyances of the whole process (after not having been given any warning from my hosting provider, but just a summary disconnect) was that permissions on the directory for the website were “correct” to have disallowed uploading random junk to the server:
drwxr-xr-x 6 bglug apache 5611520 Apr 11 13:24 bglug

The user bglug had not been compromised (it hasn’t even logged-in in a few months) – and neither was the apache group (which, of course, cannot login, but still).

Apparently, some part of the version of WordPress the site was running (or a plugin) was compromised, and allowed a malicious attacker to upload junk to the server, and spawn this DDoS on my server.

Moral of the story? Keep all your software up-to-date, and monitor your logs for suspicious activity – not sure monitoring would’ve done me good in this case, but it’s a Good Practiceâ„¢ anyway.

evaluating “work from home” “opportunities”

It seems the number of advertised “work from home” “opportunities has gone ever higher since the advent of prolific social networking.

A not insignificant portion of these opportunities really are legitimate – 31, Avon, Mary Kay … – but a lot of them at the very least feel scammy.

The good ones tell you everything you need to know up-front:

  • “franchise” or licensing fees
  • buy-in cost
  • required sales to maintain active status
  • expected monthly commitment
  • growth paths
  • etc

The scammy ones do not – they have poorly-written, ambiguous, or unstated expectations, require lots of cold calling, expect you to pay-in an enormous amount with little-to-no understanding of how you will get paid later, they’re really “affiliate” marketing, etc. They’re the timeshare of the ‘independent consultant’ business. They’re the 2AM infomercial of the “work” world – you know the type, “for the low low cost of 3 easy payment of $39.95 I will teach you how to make money sending envelopes!” Btw, the way you make $1000s sending envelopes is by promising people to teach them how to make money by sending envelopes.

Many people I know have a tendency to get sucked into the more scammy of the varied wfh things – using the common catchphrases of “if you’re tired of being a Just Over Broke (aka “job”) worker, this is for you” or “in just 10-15 hours a week, earn $500-$2000 a month” or “I’m getting ready to launch a great new product, and I need you to be on the secret board of directors in the prelaunch stage” and more similar to them.

Let’s look at the the first one I mentioned: “10-15 hours per week to ‘earn’ $500-$2000 a month”. If you work (whatever this involves, it’s always left very nebulous), 40 hours a month and make $500, you’re making $12.50 an hour – about 50% above minimum wage, but you haven’t paid taxes yet – and you’re on the hook for all of your SSI (not the half you usually are by being a “real” employee). That means you pay 15.3% to SSI and Medicare (and remember, still no income taxes taken out yet). 15.3% of $500 is $76.50. Compare that to working for a “real” employer where you only pay 7.65% (because they pay more than half of it). 7.65% of $500 is $38.25. That’s a major difference.

What if you’re at the high end of the mentioned range? $2000 a month (which is only $24000 a year, btw – a third less than teachers start in the state of Kentucky), and we’ll say it took you 60 hours to earn it. That’s $33.33 an hour. If you could sustain $33.33 an hour (by, oh I don’t know, having a real job?), you’d be earning $69333 a year (2080 work hours in the year). The problem with these types of “opportunities” is that they’re not consistent. And the hours range always (in my observation) corresponds to the bare minimum of the “earnings” range. If it takes you 60 hours to make $500, you’re only making $8.33 an hour – a dollar more than minimum wage, and you’re on the hook for double the SSI/Medicare taxes – which, over the 60 hours, shows a difference of only 44 cents per hour more than minimum wage. 44 cents. Why not just get a job?

What if you’re truly successful with one of the “work from home” thingies? Well, then you start making the infomercial rounds, and you’re the guy they show with the 12 mansions, the 8 yachts, the cars, the women, etc. But you’re also not working “10-15 hours a month” – you’re engaged with the “opportunity” full-time+. You’re probably operating your “business” 70-90 hours a week.

If you’re going to work 70-90 hours a week, why not start your own company and own *everything* you do? You will, most likely, pay far less in taxes than as an independent contractor.

Are “work from home” “opportunities” all a scam? No. But do they consistently yield the earnings levels advertised for the hours put in? Not that I have witnessed.

For more information, this Money.SE question, “What warnings would you tell a friend about to enter a multi-level marketing (MLM) business venture?“, is a great resource:

  • MLM is not really a selling job
  • Be careful not to stockpile inventory, you’ll end up with $4000 dollars worth in your garage that you’ll never use
  • MLM is really a recruiting and training sales people job
  • Don’t think you are going to get rich at this part time
  • There are a lot of millionaires from MLM but they work a lot of hours recruiting and training
  • What does the business do
  • How do you make money
  • How do they make money
  • Why does this business need you
  • What do you bring to the table that the business doesn’t already have (skills, contacts, money)
  • How realistic are your time expectations – is this to be a part-time occasional endeavor, or your full-time occupation
  • Is there a product
  • Is the market saturated
  • Put as little of your own money into it as possible
  • Take as much out of it as you can as soon as you can
  • Don’t count your money as earned until you actually get it in your hands as ‘cold hard cash’
  • Remember if it’s too good to be true, it usually is – no matter how many of people assure you it’s not
  • Don’t go in thinking you’ll beat the system by trying harder than everyone else: the only way you’ll make any money is by recruiting lots of people, and selling products that can be obtained for cheaper elsewhere at a normal store
  • Make sure you are paid on volume, not people

more irrational gun maneuvering – president obama living up to [my] expectations

I was harshly criticized a few years ago when I pointed-out Mr Obama’s anti-gun stances. While several good things for gun owners did happen in his first term, irrational exuberance over the recent shooting in Connecticut has led Vice President Biden to say the following:

“The president is going go act,” said Biden, who is conducting meetings all week on gun control. “There are executive orders, executive action that can be taken. We haven’t decided what that is yet, but we’re compiling it all.”

“I want to make clear that we’re not going to get caught up in the notion that, unless we can do everything, we’re going to do nothing,” Biden said. “It’s critically important we act.”

why technical intricacies matter

I have been working on a upgrade for one of our customers for nearly a month.

Last week we spent about two hours focused on one specific problem that had been rearing its ugly head on an exceedingly-frequent basis: one of the components of the application was routinely pitching OutOfMemory errors from the Java Virtual Machine (jvm). The errors were actually being returned from WebLogic (currently an Oracle product; previously from BEA).

Much googling of the error messages returned the following Sun bug:, and the workaround:
Disable VM heap resizing by setting -Xmx and -Xms to the same value.
This will prevent us from hitting the most common sources of the vm_exit_out_of_memory exits.
The best thing to do is increase swap size on the machines encountering this error.

[If you want to skip the rest of this, feel free: the short version is we boosted swap space from 1GB to 13GB, and it works like a champ now.]

Important Things You Should Know

  • The version (1.4) and platform (32-bit) of Java is used for a variety of reasons by this product in this component
  • A 32-bit OS/machine1 can only access ~3GB of RAM (due to OS overhead and bus address mapping strategies)
  • A 64-bit OS/machine can access between 248 and 264 bytes (256TB-16EB) of memory (depending on addressing model used)
  • There are two types of memory a system can use: heap and stack
  • The jvm gets memory for itself from the host OS from the heap
  • If more memory is need by the Java application in question, and it has not yet exceeded the max (-Xmx argument) amount available to the jvm, the jvm will get more memory for itself from the system
  • The 32-bit jvm has a certain amount of overhead itself (I have seen 5-25%, depending on the application)

Environmental issues for the application in question

  • 8 CPUs
  • 32GB physical memory
  • ~9GB RAM in use, the rest unused
  • RHEL 4 64-bit
  • 1GB swap

Go check out this video while you think for a few seconds 🙂

Oh, you’re back? Welcome!

More details about the Sun jvm: when the jvm needs more memory, so long as the system can issue it, it will ask for a multiple of what it really needs (observationally about 40%, or 1.4x the “actual” request). And while it is asking for more memory, it swaps itself out to swap space (virtual memory, or a special location/partition on the drive). After it gets its new allocation, it loads itself back in from swap, and goes on its merry way.

Why does it ask for more than what the application “actually” requested? It’s a best-guess on the part of the jvm – if you have allocated 256M of RAM minimum, and 1G max, when the application asks for 257M, the jvm doesn’t want to ask for more RAM too often from the OS, so it asks for ~360M, with the theory being that if you needed 1M over your initial amount, you will likely need yet more. This continues on until the jvm has asked for as much RAM as it is allowed, or until the application quits – whichever comes first.

Last piece of useful technical data:

  • The specific component in the application I was working with asks for 256MB to start, with a cap of 1280MB (we raised that to 2560MB (2.5GB) as an initial attempt to stave-off OutOfMemory errors)

I know it’s been a little while, but think back to that initial list of Important Things … and add into the mix that the component in question was chewing an entire CPU (in normal operation it rarely will go above 25%), and was using 3600MB of virtual memory and 2.8GB of real RAM. That’s a problem. First, because we have 32GB of real memory – there’s no reason the whole component shouldn’t fit in memory (2.8GB is equal to our 2.5GB max plus some jvm overhead). Second, because while it’s chewing an entire CPU, it’s never actually coming up, or, if it does, it’s taking an hour or more (when normally the entire application will start in 12-20 minutes from power on).

What was the problem with this ONE component? The detail is in the list of environmental factors: there was only 1GB of swap space. Uh oh. That means that unless the jvm asks for all 2.5GB up front, it will have to keep re-allocating memory to itself from the system. But with only 1GB of swap space, it has no place to unload itself to while it asks for more and then load itself back into RAM.

What to do? Let’s go back to that obscure Sun bug: “increase swap size on the machine”. We tried going from 1GB to 13GB (had a 12GB partition not being used, so we flipped it to be a swap partition) and rebooting the server.

After increasing swap space, not only does the application start in about the expected amount of time (~15 minutes), but it never pegs the CPU! Woot!

With a newer version of the product, there is an installation prerequisite check to ensure that there is as much swap space as physical RAM installed – but no explanation of why this is now the case.

Whether the above travails are the entire reason, or merely a single example of why it’s important, I won’t be installing onto any machine that doesn’t have enough swap again.

1 without special drivers/kernel modifications

staybridge suites – avoid like the plague

I rarely have good reason to complain about hotels, restaurants, etc.

This is one of those rare occasions. And it was made worse because my wife had to go through it, too 🙁

The following is the message I sent to the general manager of the Fishers IN Staybridge Suites location on Tuesday 19 July 2011 (after sending it to the corporate customer service department):

Our stay at the Staybridge Suites in Fishers IN has been horrible so far.

The facilities have been adequate, but the service has been appalling. Shortly after checking-in on the 18th, my wife was hit in the head by a stray thrown basketball while walking through the parking lot to walk our small dog. While walking our dog, she was also shouted-at by some of the teens staying at the hotel. After reporting this to the front desk, nothing was done – no apology was given, or was any attempt made to control the myriad teens running amok through and around the hotel.

The teen group has been out of control every time we have seen them, except at breakfast: taking-over the pool area, screaming, shouting, horseplaying, throwing basketballs out of the court (and hitting cars), making loud a ruckus after 10p last night in the common area (which could be heard completely down the hall and in every room facing over the great room).

Hotel staff have been unwilling or unable to control the behavior of the teens, and their apparent attempts to contact the chaperons and coaches have gone unheeded: the chaperons have been nowhere to be found (contrary to posted rules), and have done nothing to control the unruly children either.

At 1030p Monday evening, my wife went to the front desk to ask for extra pillows. The front desk lady indicated they would be sent up to our room “shortly”. At midnight, I went down and asked where they were: not only had housekeeping not brought anything up to our room, they had not been contacted even though there is a radio at the desk for such purposes. I waited an addition 12 minutes for someone in housekeeping to find a couple spare pillows and bring them to the desk so I could take them upstairs.

As I mentioned earlier, we are traveling with a small dog (a 2.5 year old Shih Tzu). The pet rules at Staybridge, vs any of the other ICH brands) are excessively stringent. An option is given at check-in as to whether daily housekeeping checks will be done either between 9a and 2p or after 5:30p. We selected the after 5:30p option. At 12:20p today, housekeeping knocked on our door. After being told to “wait just a moment”, they knocked again, and were told again “wait just a moment”. Then they opened the door WITHOUT INDICATING WHO THEY WERE until after the door was partially open. My wife was in the room by herself enjoying a TV show, and this behavior is unacceptable. After being shown that our housekeeping contract shows no one should come before 5:30p, they insisted on attending to the room anyway, and two maids came in, reset the sheets and replaced the towels, then left. If this is the daily housekeeping that is “required” for suites with pets, I shudder to think what “housekeeping” is done typically.

The behavior and attitude of the staff at this location is truly unbelievable for *any* hotel, let alone one in this price range. I travel extensively for business, and stay at hotels 2-4 times per month just for work. This is my first (and I expect my last) experience staying at a Staybridge Suites location. I have stayed at Candlewoods several times in CT and here in IN (just last week down one exit from the Staybridge). Candlewood has always been eagerly attentive to guest needs and requests, has a far more lenient pet policy, and is less expensive than Staybridge. We only selected to stay at a different location this week due to the Candlewood being sold out. We should have stayed the full week on the other side of town at a Candlewood to avoid this horrible experience with Staybridge.

It has been over a year (with hotel stays near weekly for much of that time) since having even a mediocre experience at any hotel – let alone such an egregious example of not caring and not trying on the part of hotel staff – and I am extremely disappointed with the Staybridge in Fishers.

His response to this message was NOT to call me at my cell phone number given on the day of the report (which the manager on duty (a sales coordinator) said he would do prior to my sending of the initial email), but rather to call my room the next morning (Wednesday) – after I had gone to work.

Hello, Mr Miller

My wife called and relayed the conversation you had with her this morning.

We will definitely contact you if anything else comes up and/or we need anything else during our stay.

If you have further questions or need to contact us, please call me at [redacted]

In the intervening days, nothing has been done beyond comp’ing us for the first night’s stay: which is a nice gesture, but doesn’t really do anything for me as this is a business trip (though I am sure my customer will appreciate the discount). The message I sent to Gary Miller today:

Mr Miller

While the noise from the rowdy teens has disappeared (I am guessing from not having seen them the past couple days that they left), housekeeping has been atrocious during our stay. Referring back to my initial email (which you have still not replied to outside of a call to my room on Wednesday morning when I was not available, in contradiction to what Kim Ilagon said you would do by calling my contact number she took down (and that has been on each of the emails I have sent you) or the contact number on my reservation), a “light touch service” is supposed to be mandatory every day in every room, with some form of more extended service if there is a pet in the suite. As you know, Tuesday’s housekeeping visit occurred over 5 hours before they were supposed to come to the room. That service visit was a mere “light touch” – replacing towels, emptying wastebaskets, and straightening the bed.

On Wednesday there was no housekeeping visit at all, and I was forced near the end of the evening to go ask for new towels at the front desk myself. Likewise on Thursday: no housekeeping visit occurred – and we again had to get new towels on our own.

Over the intervening days since Tuesday, the wastebaskets in our room have filled from day-to-day items – with no visit or way of getting housekeeping to take care of the room.

Additional cleanliness problems:

  • There is a cup of beer on the windowsill at the end of the hall by room 201 that has been there since before we checked in
  • The tray we used to bring breakfast up to our room was set outside our door yesterday – by lunchtime it had not been collected, so we moved it to the windowsill as well; it had still not been picked-up by when I left for work this morning
  • Myriad soaked towels were on the floor of the pool last night for over 90 minutes with no one using them or coming to take care of them (my wife and I were swimming by ourselves for that duration)
  • Trash bags have been left in stairwells for over a day before being collected

I appreciate the token you gave of comp’ing us for our first night – but as I am traveling on business, that doesn’t really “do anything” for myself or my wife (though my customer will be happy to have saved a night’s cost).

Last night I went to the front desk to ask if it might be possible to do a late checkout, which the staff member told me was not an issue, and wrote down that we would be checking-out at 2p. Today housekeeping came by at about 1215 to do a service on the room, and were surprised my wife was still in the room.

90 minutes later the front desk called my wife asking if we were going to extend our stay – the woman at the front desk had not bothered to read any of the comments/notes left for her from the night before.

Communication at this hotel amongst the staff is atrocious. I have copied this message back to corporate customer service as well (as I did the first message to you).

Contact information for this hotel:

Front desk: 317.577.9500

General Manager: Gary Miller