0

We have a problem where our OpenNMS instance is sending false alarms at certain times and the logs shows this:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) ~[?:1.8.0_45]
        at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_45]
        at org.postgresql.Driver.connect(Driver.java:283) ~[postgresql-9.3-1100-jdbc4.jar:?]
        at org.opennms.netmgt.collectd.jdbc.JdbcAgentState.getJdbcConnection(JdbcAgentState.java:120) ~[org.opennms.features.jdbc-collector-17.1.1.jar:?]
        at org.opennms.netmgt.collectd.JdbcCollector.collect(JdbcCollector.java:220) ~[org.opennms.features.jdbc-collector-17.1.1.jar:?]
        at org.opennms.netmgt.collectd.CollectionSpecification.collect(CollectionSpecification.java:274) ~[opennms-services-17.1.1.jar:?]
        at org.opennms.netmgt.collectd.CollectableService.doCollection(CollectableService.java:395) ~[opennms-services-17.1.1.jar:?]

When i researched online it could be that we're hitting max limits set by ulimit. However, by issuing ulimit -a we can see that there should be no problem:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 257405
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 257405
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The system is allocated 64 GB RAM and still have to swap. Is this simply resolved by allocating more RAM to the machine and JVM?

EDIT: ps -ef | grep java

root     10055     1 99 Jul04 ?        27-06:47:17 /usr/bin/java -Djava.endorsed.dirs=/usr/share/opennms/lib/endorsed -Dopennms.home=/usr/share/opennms -Xmx50248m -XX:+HeapDumpOnOutOfMemoryError -d64 -Xms50248m -Duser.language=sv -XX:MaxPermSize=6144m -XX:+UseStringDeduplication -XX:+UseG1GC -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.login.config=opennms -Dcom.sun.management.jmxremote.access.file=/usr/share/opennms/etc/jmxremote.access -DisThreadContextMapInheritable=true -XX:MaxMetaspaceSize=256m -Djava.io.tmpdir=/usr/share/opennms/data/tmp -jar /usr/share/opennms/lib/opennms_bootstrap.jar start

In /etc/opennms/opennms.conf i've got the following:

JAVA_HEAP_SIZE=50248
ADDITIONAL_MANAGER_OPTIONS="-d64 -Xms"$JAVA_HEAP_SIZE"m -Duser.language=sv -XX:MaxPermSize=6144m -XX:+UseStringDeduplication -XX:+UseG1GC"
#MAXIMUM_FILE_DESCRIPTORS=204800 #2017-03-23
MAXIMUM_FILE_DESCRIPTORS=1024000
MAXIMUM_SIZE_STACK_SEGMENT=16384
nillenilsson
  • 153
  • 1
  • 8
  • Isn't there some memory limit set in JVM options when starting your application? Could you edit with results of top or free command and ' ps -ef|grep java ' to see how JVM is started ? – tonioc Jul 09 '18 at 11:15
  • @tonioc Sure! I've added it to the main post. – nillenilsson Jul 09 '18 at 11:42
  • So OpenNMS is limited to 50Gb is heap size. You could: 1- increase JAVA_HEAP_SIZE value to see if fixes your issue, 2- monitor memory usage of opennms process using ps command (i.e. 'ps -e -o pid,vsz,rsz,comm |grep java'). If it increases with time going, this may result from a memory leak. – tonioc Jul 09 '18 at 12:43
  • @tonioc We are able to allocate up to 128GB of RAM to the machine, how much would you recommend allocating to JAVA itself? – nillenilsson Jul 11 '18 at 07:08
  • If you have 128Gb available, you could set -Xmx to 110G for example, but remove the -Xms parameter - initial heap size - or set it to a much lower value (let's say 10Gb). This will permit to see how memory usage is actually increasing with time going (ps command above). – tonioc Jul 12 '18 at 07:35

0 Answers0