Only getting quarter of the memory allocated via app.yaml

Question

We are running a java8/jetty9 setup on appengine flex. In the app.yaml we have allocated 30G of RAM. However in the server, when we call Runtime.getRuntime().maxMemory(), the number returned is ~7G. If I increase the RAM to 60G, this number increases to ~15G. Seems like the JVM only gets a quarter of the RAM we have allocated in app.yaml.

I also tried running the server with -Xmx30g

However the server upload failed with the error

There is insufficient memory for the Java Runtime Environment to continue.

Native memory allocation (mmap) failed to map 7158235136 bytes for committing reserved memory.

The error confirming what we were seeing with Runtime.getRuntime().maxMemory, that there is a limit of ~7G when we have allocated 30G of RAM. Where is the rest of the 75% allocation going?

Here is the Dockerfile


    FROM gcr.io/google-appengine/jetty9
    RUN apt-get -q update && \
        apt-get -y -q --no-install-recommends -t jessie-backports install 
    openjdk-8-jdk && \
        apt-get -y -q --no-install-recommends install ssh sshpass && \
        apt-get clean && \
        rm /var/lib/apt/lists/*_*
    ADD backend.war $JETTY_BASE/webapps/root.war
    WORKDIR $JETTY_BASE
    RUN java -jar -Xmx30g -Xms20g $JETTY_HOME/start.jar --approve-all-licenses \
       --add-to-startd=jmx,stats,hawtio,requestlog \
       -Djava.util.logging.config.file=src/main/appengine/logging.properties \
       && chown -R jetty:jetty $JETTY_BASE

Here is the app.yaml


    runtime: custom
    threadsafe: true
    env: flex

    handlers:
    - url: /.*
      script: this field is required, but ignored
      secure: always

    health_check:
      enable_health_check: True
      check_interval_sec: 5
      timeout_sec: 4
      unhealthy_threshold: 2
      healthy_threshold: 2

    resources:
      cpu: 6
      memory_gb: 30
      disk_size_gb: 20
    automatic_scaling:
      min_num_instances: 1
      max_num_instances: 3
      cool_down_period_sec: 120
      cpu_utilization:
        target_utilization: 0.5

You may have old instances running with the previous config. Make sure you stop all of them from the admin console and that the default version is set to your latest one. Also, what’s the output of `totalMemory()`? — Federico Panunzio, Jun 26 '18 at 10:09
Free memory (bytes): 1256519656 Maximum memory (bytes): 7692877824 Total memory (bytes): 7692877824 I have double checked that there is only one version running. These Runtime stats were added only recently. I also looked at the config from the "Versions" page and it shows the app.yaml that I have mentioned in the post. — Vinay Chitlangia, Jun 26 '18 at 15:50
In the [documentation](https://cloud.google.com/appengine/docs/flexible/java/reference/app-yaml#resource-settings) it's stated that each CPU core requires a total memory between 0.9 and 6.5 GB. So the math is the following: memory_gb = cpu * [0.9 - 6.5] - 0.4. What's the output of the environment variable GAE_MEMORY_MB? — Federico Panunzio, Jun 27 '18 at 13:36
I just made a test and out of 30gb I'm getting 24gb of JVM total memory. Could you try running the [quickstart](https://cloud.google.com/appengine/docs/flexible/java/quickstart) without a dockerfile, but specifying 30gb in the app.yaml file? — Federico Panunzio, Jun 27 '18 at 15:21
Many thanks Federico for your help. GAE_MEMORY_MB shows 30GB in our servers. I was able to see 30GB as maxMemory when running the quickstart. So something was amiss in the Dockerfile. I am installing openjdk8 in it, and it by default sets the max memory to 1/4 of available. So now I have changed the app.yaml to specify -Xmx in JAVA_OPTS environement variable. That did the trick. Many thanks again... — Vinay Chitlangia, Jun 28 '18 at 03:55
Great to hear it's working now! Could you please post the solution as an answer and after two days accept it? This way the question gets resolved. Thanks! — Federico Panunzio, Jun 28 '18 at 07:39

Vinay Chitlangia · Accepted Answer · 2018-06-29T02:09:26.290

1


 env_variables:
  JAVA_OPTS: "-Xmx16g"

Adding the above to app.yaml did the trick...

edited Jun 29 '18 at 02:09

answered Jun 28 '18 at 14:26

Vinay Chitlangia

31
5

Only getting quarter of the memory allocated via app.yaml

1 Answers1