Category Archives: insights

why technical intricacies matter

I have been working on a upgrade for one of our customers for nearly a month.

Last week we spent about two hours focused on one specific problem that had been rearing its ugly head on an exceedingly-frequent basis: one of the components of the application was routinely pitching OutOfMemory errors from the Java Virtual Machine (jvm). The errors were actually being returned from WebLogic  (currently an Oracle product; previously from BEA).

Much googling of the error messages returned the following Sun bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4697804, and the workaround:
Disable VM heap resizing by setting -mx and -ms to the same value.
This will prevent us from hitting the most common sources of the vm_exit_out_of_memory exits.
The best thing to do is increase swap size on the machines encountering this error.

[If you want to skip the rest of this, feel free: the short version is we boosted swap space from 1GB to 13GB, and it works like a champ now.]

Important Things You Should Know™

  • The version (1.4) and platform (32-bit) of Java is used for a variety of reasons by this product in this component
  • A 32-bit OS/machine1 can only access ~3GB of RAM (due to OS overhead and bus address mapping strategies)
  • A 64-bit OS/machine can access between 248 and 264 bytes (256TB-16EB) of memory (depending on addressing model used)
  • There are two types of memory a system can use: heap and stack
  • The jvm gets memory for itself from the host OS from the heap
  • If more memory is need by the Java application in question, and it has not yet exceeded the max (-Xmx argument) amount available to the jvm, the jvm will get more memory for itself from the system
  • The 32-bit jvm has a certain amount of overhead itself (I have seen 5-25%, depending on the application)

Environmental issues for the application in question

  • 8 CPUs
  • 32GB physical memory
  • ~9GB RAM in use, the rest unused
  • RHEL 4 64-bit
  • 1GB swap

Go check out this video while you think for a few seconds 🙂

Oh, you’re back? Welcome!

More details about the Sun jvm: when the jvm needs more memory, so long as the system can issue it, it will ask for a multiple of what it really needs (observationally about 40%, or 1.4x the “actual” request). And while it is asking for more memory, it swaps itself out to swap space (virtual memory, or a special location/partition on the drive). After it gets its new allocation, it loads itself back in from swap, and goes on its merry way.

Why does it ask for more than what the application “actually” requested? It’s a best-guess on the part of the jvm – if you have allocated 256M of RAM minimum, and 1G max, when the application asks for 257M, the jvm doesn’t want to ask for more RAM too often from the OS, so it asks for ~360M, with the theory being that if you needed 1M over your initial amount, you will likely need yet more. This continues on until the jvm has asked for as much RAM as it is allowed, or until the application quits – whichever comes first.

Last piece of useful technical data:

  • The specific component in the application I was working with asks for 256MB to start, with a cap of 1280MB (we raised that to 2560MB (2.5GB) as an initial attempt to stave-off OutOfMemory errors)

I know it’s been a little while, but think back to that initial list of Important Things … and add into the mix that the component in question was chewing an entire CPU (in normal operation it rarely will go above 25%), and was using 3600MB of virtual memory and 2.8GB of real RAM. That’s a problem. First, because we have 32GB of real memory – there’s no reason the whole component shouldn’t fit in memory (2.8GB is equal to our 2.5GB max plus some jvm overhead). Second, because while it’s chewing an entire CPU, it’s never actually coming up, or, if it does, it’s taking an hour or more (when normally the entire application will start in 12-20 minutes from power on).

What was the problem with this ONE component? The detail is in the list of environmental factors: there was only 1GB of swap space. Uh oh. That means that unless the jvm asks for all 2.5GB up front, it will have to keep re-allocating memory to itself from the system. But with only 1GB of swap space, it has no place to unload itself to while it asks for more and then load itself back into RAM.

What to do? Let’s go back to that obscure Sun bug: “increase swap size on the machine”. We tried going from 1GB to 13GB (had a 12GB partition not being used, so we flipped it to be a swap partition) and rebooting the server.

After increasing swap space, not only does the application start in about the expected amount of time (~15 minutes), but it never pegs the CPU! Woot!

With a newer version of the product, there is an installation prerequisite check to ensure that there is as much swap space as physical RAM installed – but no explanation of why this is now the case.

Whether the above travails are the entire reason, or merely a single example of why it’s important, I won’t be installing onto any machine that doesn’t have enough swap again.


1 without special drivers/kernel modifications

where google makes its money

Wired has an interesting infographic today from WordStream on where Google makes its money in advertising.

No surprise on some of the top entries: but the last was surprising (both to me, and the folks who did the analysis): Cord Blood. Seems “rich parents” are wanting to store their newborn’s umbilical cord blood for the stem cells contained, in the hopes that they could be used later in life if some health crisis arises.

Fascinating.

kaching

I had been playing with a fun stock market simulator/investing application on Facebook until yesterday. It was called kaching (now defunct). The authors decided to focus their efforts on their for-pay service, kaching.com, and drop the free app on facebook.

That’s all well and good – folks making money does not bother me.

What does bother me is when the maintainers of the application say they are expressly not inviting the 60,000+ users to their new service. A reply I received to a post I made when I found out the app was being removed from one of their admins was incredibly unprofessional and rude. He said they weren’t inviting the facebook users because they were not likely to want to use it, and wouldn’t pay for it.

My request was for the app to be kept up, just have maintenance on it cease and no new features be added. Keeping the app alive would have cost them next to nothing. Removing it has alienated 60,000+ people who [almost] all complained, and have made comments like I am right now warning people away from their “premium” services.

I’m all for folks making money: that is, after all, how bills get paid. I’m all for having a closed platform – if that’s what you want to do (though open platforms seem to last longer and work better overall… but that’s an entire series of posts in its own right). But using 60,000 folks on facebook to effectively beta test your premium services, and then drop them just because you want to refocus, does not bode well for professionalism or future success. No, none of us paid to use the app. But an awful lot of us had a lot of fun playing with it.

Shame they’ve decided to upset 60,000 people in such a way. Even more of a shame is that they used 60,000 people as guinea pigs without telling us.

Active, free alternatives on facebook:

irony

About 2 years ago, I wrote about the problem of holding onto electronic stuff just because storage was cheap.

It wasn’t until I met my fiancee that I realized I did the same thing with “real” stuff – holding onto it just because it was there.

I’m no where near a candidate for Hoarding: Buried Alive (praise the Lord!) .. but I could easily have been in a few years had I not met someone so helpful in keeping priorities about “stuff” straight – if it has a home and can live there relatively neatly.. it’s ok. Otherwise, it needs to be elsewhere.

Why is the self-storage industry doing so well in the US? Why do we own so much stuff we can’t even keep it in our homes? When did “stuff” become more important than people?

want to reduce gas consumption?

Buy better tires.

I referenced Seth Godin earlier today in regards to investment in developing countries.

Why is it, then, that a marketing blogger would talk about wanting to reduce fuel consumption? I think it’s because it’s easier to relate to than streamlining other processes you may have in your business or development cycles. It’s something we can relate to directly.

If we assume that all the cars drive the same number of miles, which would be a better investment:

  • Get new tires for all the Suburbans and increase their mileage a bit to 13 miles per gallon.
  • Replace all the Priuses and rewire them to get 100 miles per gallon (doubling their average!)

That’s right – spend a little bit of money on the Suburbans, and cut fuel usage more than you could by doubling the efficiency of the already-efficient Prius.

Why? It’s because we think in MPG rather than GPM. What does thinking in miles-per-gallon do to our brains that gallons-per-mile would make clearer? Well, here’s the math:

  • Let m be number of miles driven by a car…
  • Let s be the gas consumption (in gallons) for Suburbans (= m/10)
  • Let p be the gas consumption (in gallons) for Priuses (= m/50)
  • Let T be the total consumption (in gallons) (= s + p = m/10 + m/50 = 6m/50 = 0.12/m)

So in Scenario #1, we have T = m/13 + m/50 = 50m+13m/650 = 63m/650 = 0.097m

And in Scenario #2, we have T = m/10 + m/100 = 11m/100 = 0.11m

Scenario #1 reduced consumption by 0.12-0.097 = 0.023; Scenario #2 only by 0.01; Scenario #1 is 2.3x more efficient!

This is due to a power-curve relationship early on in the MPG table, where a minute improvement (1 MPG on 10) is a huge percentage improvement (10%) at the front end whereas later-on it’s minuscule.

I thought it was neat 🙂

Original article – http://sethgodin.typepad.com/seths_blog/2009/08/not-so-good-at-math.html

The math as to why it works – http://www.onpreinit.com/2009/08/mpg-illusion-seth-godin.html or http://charliepark.tumblr.com/post/169016492/in-seth-godins-post-this-morning-he-talks-about

store brands are sometimes better

I shop at various grocery stores, and the cashiers generally look at my purchases a little askance: clementines, milk, ice cream, pot pies, beer, Ensure – they seem to get confused when I checkout with my selections.

I was raised with a thrifty mindset, but am not afraid to spend money for better quality.

For years I’ve preferred store brand cereals – corn flakes, raisin bran, cocoa puffs, cocoa crispies, rice crispies, and chex are all indistinguishable to me when comparing store brand and name brand. Some I can distinguish and just like the store brand more. Cheerios is the only notable difference – fake cheerios are NOT the same as the ones from General Mills.

Trader Joe’s raisin bran, for example, is cheaper than the name brand, has fewer calories, and (I think) tastes better than those from Post or Kellogg.

I don’t go out of my way to buy organic foods to make a statement. Many times I think they taste worse, or the relative percentage change in quality does not match the price percentage shift. Trader Joe’s raisin bran happens to be organic – but the fact that it tastes good and is inexpensive is more important.

I’ve been bitten several times by trying store brand macaroni and cheese. I picked-up a batch from Lowes Foods recently, and am hoping they’re not hideous like the ones from Winn Dixie were. But if they’re decent, then I have a source for less expensive than Kraft mac and cheese. My favorite is Prince brand, but those aren’t purchaseable in NC – and therefore I tend to stock-up periodically when I go home to NY.

Also, I’ve been pleasantly surprised with the high quality of the canned strawberries I can get at my local dollar store. Yes. A dollar store. Most of the food they sell is high enough in sodium to make road ice quiver. But the canned strawberries at my local Dollar Tree near NC55 and NC54 are downright tasty – 90 calories per serving, with only three servings per can. That lines-up with my home-made applesauce for caloric value, and makes a nice shift.

They’re also not those supersized strawberries you find in most produce departments of grocery stores; the ones at Dollar Tree are about 1/2″ in diameter rather than 2″. The smaller size makes for what seems to be a more strawberryish strawberry than the giant ones from the supermarket.

Such experimenting has made me want to do more, and so now when I go shopping I try to compare not merely price or calories – but the taste quality. It leads to a lot of sampling, but being able to shave 10-50% off my grocery bill is a nice [eventual] payoff.