4

We are seeing some of our services (Java <> SpringBoot) getting OOM. On checking heap dump, we found the micrometer library is taking 113MB (around 54% of total heap memory).

io.micrometer.statsd.internal.LogbackMetricsSuppressingUnicastProcessor
Jar: io.micrometer:micrometer-core

I did some research online and found creating a lot of distinct tags can lead to this issue. However, this is not the case with our services. We are pushing a lot of metrics to data dog but reusing tags all the time.

Neeraj Kukreti
  • 141
  • 14
  • Can you add details about how long it takes for an OOM to occur? Also how many metrics are you sending? 1 meter with 2 distinct tags would take about the same as 2 meters. (Internally they are the same thing) – checketts Sep 19 '21 at 22:45

1 Answers1

10

Memory issues like this are often caused by using high cardinality tags.

High cardinality here means that the tag can have a large number of unique values. For example, if you want to attach the gender of your users to your metrics, that tag potentially has low cardinality because you might not have a lot of values to choose from. The userID on the other hand is high cardinality since every single user will have a unique one.

Having high cardinality tags can result in a huge amount of meters since you have a meter for every unique value which will result in high memory consumption and your metrics backend will experience similar issues too.

Examples for (potentially) high cardinality tags:

  • ID of the resource, e.g.: userID
  • Email address (basically any user input)
  • Request path instead of template (e.g.: it can contain resourceID)

See this article for more details: https://develotters.com/posts/high-cardinality/ also see the Micrometer docs: https://micrometer.io/docs/concepts#_tag_values

Do you have high cardinality tags? If so, you can normalize or remove them.

Jonatan Ivanov
  • 4,895
  • 2
  • 15
  • 30