2

I stumbled upon some interesting problems last week where I noticed that the one of our production servers, which has Apache HTTP Server over Tomcat running, stopped, reporting an HTTP outage.

On further investigating this issue it seemed like it was due to JVM causing memory pages to be swapped out quickly. This resulted in the swap space being completely populated, causing a memory issue the next time a page is moved to swap.

Investigating further, it seems that there is a JVM swappiness factor that's set to 60% by default with some of the our Linux distributions. Based on some research, it seems that this could be a high value for a web service that has high traffic. Our swap space was set to 2 GB.

The swap details are:

 Filename                                Type            Size    Used    Priority
 /dev/sda3                               partition       2096472 1261420 -1

 From /proc/meminfo
 SwapCached:     944668 kB

Our JVM properties are as follows:

-Xmx6g -Xms4g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:PermSize=512M -XX:MaxPermSize=1024M -XX:NewSize=2g -XX:MaxNewSize=2g -XX:ParallelGCThreads=8

The server runs with 12 GB RAM.

Swappiness does not go well with the JVM's GC process. Thus I tried reducing the swappiness to 0, but that did not change anything. We still see cases where the entire swap space is consumed and resulting in an OutOfMemory error.

How can I tune the JVM performance?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Keshi
  • 906
  • 10
  • 23
  • Your heap size is quite high ; how much RAM do you have on the server ? – souser Jun 01 '12 at 15:49
  • 2G ram and 2G swap on a server? Spend the $50 and get 8G – Kevin Jun 01 '12 at 15:57
  • We got 12G of RAM and 2G of swap space on the server. Sorry for typo. – Keshi Jun 01 '12 at 16:38
  • Why have you added those GC-tuning parameters to your JVM? Did you actually need them or did you just add them because you found them on the internet somewhere and they seemed like a good idea? 1/3 of your entire heap for the new generation seems unnecessarily large. The default New space for a heap that large is probably more like 1GiB. – Christopher Schultz Jun 01 '12 at 18:50
  • @Keshi, can you point to some documentation that provides details about the "jvm swapiness factor that's set to 60%"? – Pidster Jun 03 '12 at 14:36
  • http://www.kernel.org/doc/Documentation/sysctl/vm.txt – Keshi Jun 03 '12 at 17:21

2 Answers2

2

The swappiness factor is actually a system-wide setting in Linux, not JVM-specific.

My personal experience with some kinds of applications is that they need swap to be turned off in order to work without hiccups, period. While I can't say for sure if your application belongs to this category, I have seen apps being swapped out despite enough free RAM being available. As you pointed out, GC and swap don't mix well. Swappiness is just an indication to the OS, so its impact on how much gets swapped out may be larger or smaller depending on circumstances. My suggestion would be to try turning swap completely off.

In order to do this, you need to be root or have sudo access. Comment out the line describing swap in /etc/fstab, which will prevent swap from being turned on after a reboot. Then, in order to turn swap off for the current server run, run swapoff -a. This may take up to a few minutes if there's data in swap that needs to be pulled back into RAM. Then, check the last line of the output of free to ensure that the total size of available swap is 0. After this, observe your app in order to determine if turning swap off solved your issue.

Michał Kosmulski
  • 9,855
  • 1
  • 32
  • 51
  • Thanks Michal, yes its a linux specific setting and not anything JVM specific. (wrongly worded in the question). – Keshi Jun 04 '12 at 14:33
  • I had tried setting the swap to 0 and have all on memory. This reduced the frequency of the outages caused but we still see some sporadic outages because of this. – Keshi Jun 04 '12 at 14:38
  • Maybe aside from the swap being used, there is something in the app itself which causes uncontrolled memory consumption at times? Assuming it's Sun/Oracle JVM, you can set the `-XX:+HeapDumpOnOutOfMemoryError` command-line option to get a memory dump when you run out of memory. You can then analyze the dump with `jhat` or `jvisualvm` (see [here](http://marxsoftware.blogspot.com/2009/06/heap-dump-and-analysis-with-visualvm.html) ) and perhaps you can find the cause. – Michał Kosmulski Jun 04 '12 at 21:18
  • Thanks Michal. Yes its a sun JVM. The memory dump logs would help. – Keshi Jun 04 '12 at 22:27
0

If your physical RAM is 12 GB and your heap is set to only 6 GB (max), then you have plenty of RAM. Is this server dedicated to the Tomcat instance?

Take a look at the verbose GC log files to see what the memory usage is over time. Check your access log files to confirm whether the access requests have increased over time.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
souser
  • 5,868
  • 5
  • 35
  • 50
  • Yes, there is plenty of RAM but maybe the GC and the swap is not syncing well. Will take a look and post the results here. Thanks – Keshi Jun 02 '12 at 17:42