3

After deploying a new version of our Java/Spring Boot software to the Swisscom Developer Cloud, running on CloudFoundry, the startup suddenly failed with the following error: OutOfMemoryError: Compressed class space. So, we decided to deploy a previous version of the software, the version which was running actually just before: The same error occurred. We did not switch from Java7 to Java8, nor did we change any configuration. This leads to the question: Is this really an error on our side or rather on the server's side?

We then tried to increase the MaxMetaspaceSize by setting the variable JBP_CONFIG_OPEN_JDK_JRE to one of the following lines:

[jre: {version: 1.8.0_+}, memory_calculator: {memory_sizes: {metaspace: 128m}}]
{memory_calculator: {memory_sizes: {metaspace: 128m}}}
{memory_sizes: {metaspace: 128m}}

The application always warned, that the value of memory_sizes was invalid. What is the correct format of this YAML variable?

[ConfigurationUtils] WARN User config value for 'memory_sizes' is not valid, existing property not present

We then deleted the Java app and the database service on the Swisscom Developer Console and recreated it. It had no effect, the same error occured.

And finally, do you know why this error does suddenly occur, even with a version which was just running fine a few minutes ago?

EDIT:

This is the manifest ([database-service-name] and [application-name] were replaced):

---
path: .
instances: 1
buildpack: https://github.com/cloudfoundry/java-buildpack
services:
- [database-service-name]
applications:
- name: [application-name]
  domain: scapp.io
  host: [application-name]
  memory: 1024M
  disk_quota: 1024M
  env:
    SPRING_PROFILES_ACTIVE: stage, cloudfoundry

The Java buildpack version is (according to the logs):

2017-03-03 11:47:02 [STG/0] OUT -----> Java Buildpack Version: b08a692 | https://github.com/cloudfoundry/java-buildpack#b08a692

This command seems to be executed (in the logs after the crash):

2017-03-03 11:46:25 [APP/PROC/WEB/0] OUT vcap 8 0 99 10:46 ? 00:01:09 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.io.tmpdir=/home/vcap/tmp -XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=68540K -XX:ReservedCodeCacheSize=240M -XX:CompressedClassSpaceSize=8731K -Xmx408104K -Djavax.net.ssl.trustStore=/home/vcap/app/.java-buildpack/container_certificate_trust_store/truststore.jks -Djavax.net.ssl.trustStorePassword=java-buildpack-trust-store-password -cp /home/vcap/app/. org.springframework.boot.loader.WarLauncher
Sybil
  • 2,503
  • 3
  • 25
  • 36
ssc-hrep3
  • 15,024
  • 7
  • 48
  • 87
  • @FyodorGlebov Thanks, I've updated the question accordingly. – ssc-hrep3 Mar 03 '17 at 11:33
  • you are useing the buildpack released two days ago, see https://github.com/cloudfoundry/java-buildpack/releases. Can you please try with old buildpack to narrow the issue? `cf push -b https://github.com/cloudfoundry/java-buildpack.git\#v3.12` I wish to see if issue is related to new buildpack. – Sybil Mar 03 '17 at 11:46
  • can you please watch and or update this GitHub issue? It seems the problem you are facing https://github.com/cloudfoundry/java-buildpack/issues/390 `cf set-env $APPNAME JAVA_OPTS '-XX:CompressedClassSpaceSize=20m'` – Sybil Mar 03 '17 at 12:11
  • @FyodorGlebov Thank you very much! Using the older version of the java buildpack actually solved this problem. I'll be definitely watching this issue on github. Thanks, again :) – ssc-hrep3 Mar 03 '17 at 12:23
  • Would you suggest to permanently set this environment variable or just wait for an update of the default configuration of the SwisscomDev cloud? – ssc-hrep3 Mar 03 '17 at 12:24
  • Sorry I don't know. I would wait 1-2 nights for updates on GitHub and or Stackoverflow. This issue is not specific to Swisscom App Cloud, it's not related to our configuration. – Sybil Mar 03 '17 at 12:30
  • Okay, thanks anyway! – ssc-hrep3 Mar 03 '17 at 12:31

2 Answers2

5

The OutOfMemory error occurred because the Java buildpack was changed to use version 3.x of the memory calculator. Similar problems arising from this change are under discussion in GitHub issue 390. Please refer to this issue for details.

In general, v3.x of the memory calculator chooses values for various JVM memory settings based on the number of class files in the application and some default values which depend on the version of Java. It then sets the maximum heap size to the remaining amount of memory.

The previous version of the memory calculator was configured by setting JBP_CONFIG_OPEN_JDK_JRE. However, v3.x can be configured simply by setting the corresponding Java memory settings in JAVA_OPTS. For example, you could set the maximum metaspace size to 100 Mb as follows:

cf set-env app-name JAVA_OPTS '-XX:MaxMetaspaceSize=100m'

If you simply want a workaround, you can use the version of the Java buildpack released before the memory calculator change:

cf push -b https://github.com/cloudfoundry/java-buildpack.git\#v3.14 ...
glyn
  • 1,180
  • 8
  • 19
2

comment from Swisscom Java developer:

For sure they will be reverting the memory heuristics to what they were in 3.13 or at least refine the calculations. The current recommendation to all customers is either

  • to use 3.13 or
  • to use some explicit options with the environment variable JAVA_OPTS.
Sybil
  • 2,503
  • 3
  • 25
  • 36