My VMs are running at over 90% RAM, I need MORE RIGHT NOW!!

Haven’t we all had those outrageous claims and demands from application vendors and sometimes managers?! My machine’s running slow, I need 16 vCPUs and 64GB RAM. Yeah, right. NO!!

I recently dealt with one such claim from a vendor – OMG! My machines are repeatedly breaching 90% critical alarms. Increase my RAM allocation or reserve it all right now.

I have had the privilege of designing and implementing a sizable vROps installation and the resultant advantage of using it for such situations. Let’s take a look in vROps for performance statistics for these VMs. Here’s a screenshot of one machine:

Performance1

Quite clearly, the machine’s been demanding about 15GB on average with one spike in the past 30 days. Let’s also take at vCenter performance charts to shore up evidence against the need for more RAM:

Performance2

(You may have noticed this chart is only for the past week, slight issue with vCenter that is irrelevant to the VM performance issue). Anyways, this chart indicates the VM’s been actively consuming about 15GB as shown by the Active metric. Granted is how much physical memory is available to the VM while Consumed is Granted + Shared. So focusing on the Active metric, vCenter (rather the host) believes the machine’s using about 15GB (though this is statistical sampling only and based on the physical memory touched by the VM). Other thing to keep in mind is the hypervisor has no idea of what’s going on in the memory pages inside the guest OS. In times of memory contention it works with the balloon driver to determine which memory pages can be borrowed from the VM so they can be handed out to VMs that need the RAM.

Now that we’ve looked at what the monitoring (vROps) and management (vCenter) stations think, it’s time to turn to what the OS thinks.

Performance3 Performance4

Well now this is way off. ~30GB vs the 15GB vCenter reports. According to the guest OS, almost all of the RAM’s being used, moreso by the java.exe process than anything else. Is it a good thing? Based on face value, yes because it means the VM’s putting most of its configured RAM to good use. But it also means other processes are possibly being starved of memory by the java process. Heap settings! That’s the one thing that springs to mind quickly. Turned out the heap size was set to be just about equal to the RAM configured for the VM, doing so results in high RAM utilization, possible paging to the disk resulting in performance degradation and resultant system, application instability. The vendor adjusted the heap size and since then there have been happy days for the PowerCurve application and the guest OS.

Moral of the story:

  • do not cave into absurd vendor/developer requests especially the ones that cry out – reservations for me please – in big red font!
  • do not ignore them either. Evaluate the requests carefully, sometimes they are legitimate!

Leave a Comment

Your email address will not be published.

1 Trackback

  1. Right sizing VM Memory without using agent - virtual red dot (Pingback)