It seems like we ran into a OutOfMemoryError: Metaspace before actually running out of available memory for that pool.
More specifically, we appeared to hit that error as soon as the committed amount for that pool reached the maximum, instead of when the used amount did.
Here's the setup:
We have a Jenkins server running on Oracle Java 8 update 121, and have the following metaspace arguments -XX:MetaspaceSize=10G -XX:MaxMetaspaceSize=10G. We also have Datadog monitoring heap and non-heap pools.
We hit an issue where the Jenkins log indicated that some thread threw an OutOfMemoryError: Metaspace. However, in Datadog at the time of incident, the amount of non-heap used is shown to be very low (graph below).
At first I thought Datadog might be measuring it wrong, but using jconsole I get matching results for current usage (I didn't have jconsole open at the time of incident).
My only other conclusion was that the error originated from trying to allocate more committed metaspace, even though there's still plenty of gap between that and the used amount.
Am I missing something about how these memory pools are supposed to work?
Note: I'm perfectly aware that this is a pretty large metaspace to have to begin with, and that we likely have a classloader leak somewhere. This is something we hit while trying to investigate that leak.
