Hazelcast client metrics have no value (are always 0.0)

Question

Background / Project Setup: We are developing a (micro-) service in Springboot (`2.0.4.RELEASE`) with JCache (`javax.cache:cache-api:1.1.0`). We recently switched from Ehcache to Hazelcast (`3.10.4`) to have a central cache cluster for our distributed microservices. We furthermore use Prometheus (`io.micrometer:micrometer-registry-prometheus:1.0.6`) to export important metrics. After switching, the exported cache metrics do not have any value other than 0.0. Details: I use the following Spring configuration for Hazelcast (deleted non-relevant imports)

    import org.springframework.cache.CacheManager;
    import com.hazelcast.client.HazelcastClient;
    import com.hazelcast.client.config.ClientConfig;
    import com.hazelcast.core.HazelcastInstance;
    import com.hazelcast.spring.cache.HazelcastCacheManager;
    
    @Configuration
    public class HazelcastCacheConfig {
    
        @Bean
        public ClientConfig config() {
            ClientConfig config = new ClientConfig();
            // set group and network config
            return config;
        }
    
        @Bean
        @DependsOn("config")
        public HazelcastInstance hazelcastInstance() {
            return HazelcastClient.newHazelcastClient(config());
        }
    
        @Bean
        @DependsOn("hazelcastInstance")
        public CacheManager cacheManager() {
            return new HazelcastCacheManager(hazelcastInstance());
        }
    }

Our project requires to create caches dynamically on the fly. So I implemented a custom CacheResolver to create and register these caches and their corresponding metrics:

    import org.springframework.boot.actuate.metrics.cache.CacheMetricsRegistrar;
    import org.springframework.cache.Cache;
    import org.springframework.cache.CacheManager;
    import io.micrometer.core.instrument.binder.cache.HazelcastCacheMetrics;
    import io.micrometer.prometheus.PrometheusMeterRegistry;
    
    
    @Component
    public class CacheManagement implements CacheResolver {
    
        @Autowired
        CacheManager cacheManager;

        @Autowired
        CacheMetricsRegistrar cacheMetricsRegistrar;

        @Autowired
        PrometheusMeterRegistry meterRegistry;
    
        @Override
        public Collection<? extends Cache> resolveCaches(CacheOperationInvocationContext<?> context) {
            String cacheName = context.getMethod().getAnnotation(Cacheable.class).cacheNames()[0];
            Cache cache = cacheManager.getCache("some Name");

            // checks if cache already exists in io.micrometer.prometheus.PrometheusMeterRegistry
            if (!cacheRegistered(cache)) {
                if (cache.getNativeCache() instanceof IMap<?, ?>)
                    HazelcastCacheMetrics.monitor(meterRegistry, (IMap<?, ?>) cache.getNativeCache(), /*some tags*/);
                // same result with this
                // cacheMetricsRegistrar.bindCacheToRegistry(cache, /*some tags*/)
            }
        }
        return cache;
    }

Finally I annotate the chacheable methods with

    @Cacheable(
        cacheNames = "someGeneratedName",
        cacheResolver = "cacheManagement",
        keyGenerator = "cacheKeyGenerator",
        unless = /*..*/,
        condition = /*..*/
    )
    public Object someCacheableMethod(Object... someParameters) {
        // logic
    }

Now caching works great. The caches are generated at runtime and through debugging I could verify that the caching mechanism works as expected. The metrics are also exported through Prometheus. The only problem is that all caching related metrics always have a value of 0.0.

With debugging I discovered, that the setHits(long hits) method in com.hazelcast.monitor.impl.LocalMapStatsImpl is never called. So when Prometheus scraping leads to getHits() being called, it always returns 0.

What else I tried:

• Let Spring generate the CacheManager bean: same result, Spring wraps a HazelcastClientCacheManager in the generated CacheManager bean.
• Inject a JCacheCacheManager bean in CacheManagement. Spring still wraps a HazelcastClientCacheManager bean in the JCacheCacheManager but now only JCache cache metrics are exported, none of the Hazelcast specific ones (like cache_partition_gets_total which I thought has to be exported as an alternative to the cache_gets_total{result="miss"} metric according to micrometer issue #586). All values are still 0.0

One last thought/idea that I have is that caching metrics need to enabled on the Hazelcast members somehow but I could not find any information on this.

Answer 1

@steve-mcgarrett, JCache stats in Hazelcast is disabled by default. You need to enable it, either programmatically or adding below config to hazelcast.xml file:

<cache name="default">
  <statistics-enabled>false</statistics-enabled>
</cache>

Please see: http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#jcache-declarative-configuration

Answer 2

Better very late than never they say...

So turns out that I was completely wrong. If you connect to a Hazelcast server (members run separately and are not embedded in the Spring application), you cannot retrieve metrics via the Hazelcast client (which is part of the Spring application).

You have to retrieve the metrics from the Hazelcast server members instead. This is done via a JMX agent. The official docs give a good starting point.