1

I am using liferay 6.0.6 with tomcat 6 and using terracotta 3.5.4. The terracotta installation in on another server and works fine.

The debian and redhat systems are virtual machines and use the exact same virtual hardware. 1 cpu , 4gb ram, both 64bit OS.

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.3) (rhel-1.48.1.11.3.el6_2-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

If im using a debian system and boot the application then it everything works as expected.

If im using the redhat system then i get an error :

java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:657)
    at java.util.Timer.<init>(Timer.java:176)
    at com.tc.object.locks.ClientLockManagerImpl.<init>(ClientLockManagerImpl.java:39)
    at com.tc.object.StandardDSOClientBuilder.createLockManager(StandardDSOClientBuilder.java:190)
    at com.tc.object.DistributedObjectClient.start(DistributedObjectClient.java:639)
    at com.tc.object.bytecode.ManagerImpl$2.execute(ManagerImpl.java:263)
    at com.tc.lang.StartupHelper.startUp(StartupHelper.java:39)
    at com.tc.object.bytecode.ManagerImpl.startClient(ManagerImpl.java:281)
    at com.tc.object.bytecode.ManagerImpl.init(ManagerImpl.java:202)
    at com.tc.object.bytecode.ManagerImpl.init(ManagerImpl.java:190)
    at com.tc.object.bytecode.hook.impl.DSOContextImpl.createStandaloneContext(DSOContextImpl.java:179)
    at org.terracotta.express.StandaloneL1Boot.call(StandaloneL1Boot.java:190)
    at org.terracotta.express.ClientImpl.<init>(ClientImpl.java:309)
    at org.terracotta.express.ClientFactoryImpl.newClient(ClientFactoryImpl.java:232)
    at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:225)
    at org.terracotta.express.ClientFactoryImpl.createClient(ClientFactoryImpl.java:212)
    at org.terracotta.express.ClientFactoryImpl.getOrCreateClient(ClientFactoryImpl.java:190)
    at org.terracotta.express.ClientFactory.getOrCreateClient(ClientFactory.java:28)
    at net.sf.ehcache.terracotta.StandaloneTerracottaClusteredInstanceFactory.<init>(StandaloneTerracottaClusteredInstanceFactory.java:37)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

The weirdest part of this error is that memory is not used. I have set the Xmx at 3500m but the memory usage never climbs above 2.3gb.

I've been searching for a possible cause and found this : Low Java single process thread limit in Red Hat Linux

So i changed my ulimit -n and ulimit -u to 80000 but still nothing changes.

I then edited my /etc/security/limits.conf

*       soft    nproc           81920
*       hard    nproc           81920
*       soft    nofile          81920
*       hard    nofile          81920

I also edited /etc/sysctl.conf

fs.file-max = 100000

then i rebooted.

I also added -Xms=2g to my setenv.sh

I increased -Xss to -Xss5000k

I run echo 200000 > /proc/sys/kernel/threads-max

But the same error still comes up. Any ideas ?

free -m at begining of booting

             total       used       free     shared    buffers     cached
Mem:          3963       1027       2935          0         25        748
-/+ buffers/cache:        253       3709
Swap:         2047         18       2029

free -m just before crash

             total       used       free     shared    buffers     cached
Mem:          3963       2897       1065          0         31        823
-/+ buffers/cache:       2043       1920
Swap:         2047         18       2029

Update : i also tried with a machine with 8gb ram and the same result appears.

Community
  • 1
  • 1
SpikerTom
  • 125
  • 3
  • 16

1 Answers1

2

Java process contains several memory pool. java memory structure

In your case OutOfMemory error say: no enough space in your Java Process for stack allocation.

  • So if you increase -Xss pool you are able to start less threads count in your application. Try to decrease -Xss parameters. Thread with big stack you are able to start with special constructor.
  • Also you are able to decrease memory for heap, because the more memory is allocated for the heap (not necessarily used by the heap), the less memory available in the stack.
  • Or even better try to review your application structure and decrease thread count in your application, using ThreadPool or e.t.c.
Taky
  • 5,284
  • 1
  • 20
  • 29
  • i tried as you described. JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8 -Duser.timezone=Europe/Paris -Xss512k -Xmx2g -Xms1g -XX:MaxPermSize=512m -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv4Stack=true" but this gives the save error as before. Please note that this boots fine in ubuntu but not in redhat. – SpikerTom Aug 08 '12 at 13:12
  • So i decreased the Xss value even further to 256k. (128kb gives me a overflow error ). This gives me the exact same error again. To thing is it gives me the error at exactly the same point then before. – SpikerTom Aug 08 '12 at 13:53
  • How many threads in your application? Do you able to edit source code in your case? – Taky Aug 09 '12 at 06:19
  • I belive there exist stack size configuration in Ubuntu, so it may override java configuration. – Taky Aug 09 '12 at 06:33
  • i am not able to overide the source code. I will have a look at the stack size configuration. – SpikerTom Aug 13 '12 at 07:44
  • ubuntu has ulimit -s of 8096 by default while red hat has 6000. So i changed it to 10000 but the same error appears. – SpikerTom Aug 13 '12 at 07:48