0

I'm trying to configure monitoring of infinispan with prometheus. By default infinispan doesn't expose metrics. Tried to configure it like this in standalone.xml and domain.xml (for domain mode) infinispan's configuration:

    <subsystem xmlns="urn:infinispan:server:core:9.3" default-cache-container="local">
        <cache-container name="local" default-cache="default" statistics="true">
            <metrics gauges="true" histograms="true" />
            <global-state/>
            <local-cache name="default"/>
            <local-cache name="namedCache"/>
        </cache-container>
        <metrics enabled="true">
            <metrics-infinispan default-enabled="true">
                <jmx enabled="true" domain="infinispan" expose-jmx="true"/>
                <prometheus enabled="true" prefix="infinispan"/>
            </metrics-infinispan>
        </metrics>
    </subsystem>

But when I verify locally that statistics are exported to the metrics endpoint getting "Connection refused":

curl -v http://localhost:11222/metrics

Where am I doing wrong?

Garamoff
  • 84
  • 1
  • 6

2 Answers2

0
<infinispan>
  <cache-container statistics="true">
    <metrics gauges="true"
             histograms="true" />
  </cache-container>
</infinispan>

curl -v http://localhost:11222/metrics --digest -u username:password
Diego
  • 413
  • 3
  • 14
0

Look at the 9.4 documentation, which should be the closest: https://infinispan.org/docs/9.4.x/server_guide/server_guide.html#exposing_jmx_beans_to_prometheus I would also really recommend upgrading to a more recent version.

Tristan Tarrant
  • 1,299
  • 6
  • 6
  • Thanks for answer. We can't upgrade infinispan now, too many services uses it. But I still can't understand how to enable exposing jmx metrics for prometheus monitoring. Or I should use something like jmx exporter? And what about this? I can't understand what does it means in documentation? What is it my-config.yaml and cloud.xml? We use infinispan in domain mode. $ ./standalone.sh -c cloud.xml --jmx 8180:my-config.yaml – Garamoff Jun 07 '23 at 11:18
  • And here https://usermanual.wiki/Document/userguide.1330147408/html#pfd1 on page 197 there is info about Monitoring cluster health. But I can't understand how to enable it from this documentation. It's not full. – Garamoff Jun 07 '23 at 11:25
  • Apologies, 9.3.x includes the old fabric8 agent_bond which fails to work properly with JBoss Logmanager when enabling the prometheus JMX exporter. Infinispan 9.4 includes the Prometheus JMX exporter directly. You could grab the `modules/system/add-ons/ispn/io/prometheus/jmx/ispn-9.4/` directory and the relevant changes in the `bin` directory from a 9.4 installation and apply them to your 9.3 installation. Although I really recommend you upgrade to 9.4. – Tristan Tarrant Jun 08 '23 at 09:35
  • If I understood correctly, anyway I should install version 9.4 to apply these changes to version 9.3? I hope these changes won't crash my 9.3 installation. – Garamoff Jun 08 '23 at 11:02
  • In installation 9.3 I catched error while running java -javaagent:/path/to/jmx_prometheus_javaagent-0.18.0.jar -Djava.util.logging.config.file=/path/to/config.yml -jar /opt/infinispan-server/jboss-modules.jar – Garamoff Jun 08 '23 at 11:13
  • "Installing" 9.4 is not necessary, but you should download it, extract the files you need and apply them to your 9.3 installation. Upgrading to 9.4 is a better and safer approach, though. – Tristan Tarrant Jun 08 '23 at 11:40
  • I think it's best if you come over to https://infinispan.zulipchat.com to discuss this interactively – Tristan Tarrant Jun 08 '23 at 11:41