eucalyptus-cloud.service 4.4.4 constantly crashing with JVM memory errors

Question

I'm trying to work through the manual installation guide. When I get as far as registering an admin account for the console (euare-accountcreate) or registering services (euserv-register-service) the eucalyptus-cloud service inconsistently crashes with the same JVM memory error. As an example a few times I've been able to register the UFS or create the admin account but when I move on to perform another step in the install it will fail and checking the service status shows that it has crashed. I don't have experience with Java memory errors and could really use some help understanding whats going on and how to investigate this type of error in general:

[root@cloud ~]# euserv-describe-services ufs-10.0.0.2

euserv-describe-services: error: connection error (('Connection 
aborted.', BadStatusLine("''",)))
[root@cloud ~]# systemctl status eucalyptus-cloud -l
● eucalyptus-cloud.service - Eucalyptus cloud web services
Loaded: loaded (/usr/lib/systemd/system/eucalyptus-cloud.service; 
enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2018-10-10 08:45:09 EDT; 
4s ago
Process: 45951 ExecStart=/usr/sbin/eucalyptus-cloud $CLOUD_OPTS 
(code=exited, status=1/FAILURE)
Main PID: 45951 (code=exited, status=1/FAILURE)
CGroup: /system.slice/eucalyptus-cloud.service
       ├─46206 /usr/bin/postgres -D /var/lib/eucalyptus/db/data - 
h0.0.0.0 -p8777
       ├─46207 postgres: logger process                                         
       ├─46209 postgres: checkpointer process                                   
       ├─46210 postgres: writer process                                         
       ├─46211 postgres: wal writer process                                     
       ├─46212 postgres: autovacuum launcher process                            
       └─46213 postgres: stats collector process                                

Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: OpenJDK 64-Bit Server VM 
warning: INFO: os::commit_memory(0x00007f496534b000, 12288, 0) failed; 
error='Cannot allocate memory' (errno=12)
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: #
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: # There is insufficient 
memory for the Java Runtime Environment to continue.
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: # Native memory 
allocation (mmap) failed to map 12288 bytes for committing reserved 
memory.
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: # An error report file 
with more information is saved as:
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: # 
/tmp/hs_err_pid45954.log
Oct 10 08:45:09 cloud eucalyptus-cloud[45951]: 2018-10-10 08:45:09 
45951 ERROR 0574 Service exit with a return value of 1.
Oct 10 08:45:09 cloud systemd[1]: eucalyptus-cloud.service: main 
 process exited, code=exited, status=1/FAILURE
 Oct 10 08:45:09 cloud systemd[1]: Unit eucalyptus-cloud.service 
 entered failed state.
 Oct 10 08:45:09 cloud systemd[1]: eucalyptus-cloud.service failed.

Edit to include some version info:

[root@cloud ~]# cat /etc/centos-release

CentOS Linux release 7.5.1804 (Core) 

[root@cloud ~]# java -version

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

EDIT to note my point of confusion: This system only has 25G of RAM in use and 223G free. So I need help understanding how java is running out of memory.

What is unclear on this line? `There is insufficient memory for the Java Runtime Environment to continue` — rkosegi, Oct 10 '18 at 13:42
[root@cloud tmp]# free -h indicates 25GB used and 223GB free on the system. The install documents for the eucalyptus application don't indicate a need to modify any java environment variables etc... I'm not very familiar with java and applications built on java so I guess what is unclear to me is how to go about understanding and troubleshooting the error that on its face indicates insufficient memory when the operating system indicates 223GB of RAM free. — D N, Oct 10 '18 at 15:05

score 0 · Accepted Answer · answered Oct 13 '18 at 07:42

0

try this fix:

    echo 1999999 > /proc/sys/vm/max_map_count

answered Oct 13 '18 at 07:42

Dee

74
6

This worked. Thank you!! The JVM errors were a red-herring and one of many scenarios crashing my services. I read up a little on the max_map_count value and have permanently added an appropriately high value to my /etc/sysctl.conf . My Eucalyptus services are now stable and I can move on to configuring the rest of the installation. – D N Oct 15 '18 at 18:04
Cheers! Let me know how you install windows image when you get there – Dee Oct 16 '18 at 21:17
I will. I have no experience working with AWS API or Eucalyptus but I intend to include that as one of my first use cases once I've conquered these installation concepts. – D N Oct 16 '18 at 21:44

eucalyptus-cloud.service 4.4.4 constantly crashing with JVM memory errors

1 Answers1