3

I'm trying to monitor Apache Ignite with Prometheus' JMX exporter, but I'm seeing only default JVM metrics plus metrics only for "Thread Pools" Beans. JMX exporter run as agent:

/usr/bin/java -XX:+AggressiveOpts -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.13.0.jar=8080:/etc/prometheus/prometheus_config.yml -Xms1g -Xmx1g -server -XX:MaxMetaspaceSize=256m -Dfile.encoding=UTF-8 -Dcom.sun.management.jmxremote.rmi.port=49112 -Djava.rmi.server.hostname=127.0.0.1 -DIGNITE_QUIET=true -DIGNITE_SUCCESS_FILE=/usr/share/apache-ignite/work/ignite_success_ed3b2798-4d48-4188-94ac-1728fa8628dc -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=49112 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -DIGNITE_HOME=/usr/share/apache-ignite -DIGNITE_PROG_NAME=/usr/share/apache-ignite/bin/ignite.sh -cp /usr/share/apache-ignite/libs/*:/usr/share/apache-ignite/libs/ignite-indexing/*:/usr/share/apache-ignite/libs/ignite-spring/*:/usr/share/apache-ignite/libs/licenses/* org.apache.ignite.startup.cmdline.CommandLineStartup /etc/apache-ignite/default-config.xml

Ignite config has enables metrics:

<property name="metricExporterSpi">
    <list>
        <bean class="org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi"/>
    </list>
</property>

I can see a lot of ignite metrics from jconsole connected to 49112 port. jconsole

Tried different jmx-exporter options, nothing helped.

---
hostPort: 127.0.0.1:49112
lowercaseOutputLabelNames: true
lowercaseOutputName: true

---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:
- pattern: "^org.apache<clsLdr=(.+), name=sys"
  name: ignite_sys_stats
  help: Ignite cluster amount of heap memory in bytes
  labels:
    attr: $3
  type: GAUGE

and even empty config, that should mean "gather everything as is", still see just standard JVM + "Thread Pools". Can you suggest what's wrong here ?

Olaf Kock
  • 46,930
  • 8
  • 59
  • 90
Alex Saraseka
  • 81
  • 6
  • Since JConsole can consume all the metrics from the exporter, it sounds like the issue is on the JMX Prometheus exporter end. It might be that the Prometheus exporter can't parse messages of the Ignite exporter SPI. Do you use this exporter? https://github.com/prometheus/jmx_exporter – dmagda Jul 22 '20 at 15:32
  • It seems I faces with https://github.com/prometheus/jmx_exporter/issues/483 `Exception in thread "main" java.lang.ClassCastException: javax.management.openmbean.TabularDataSupport cannot be cast to javax.management.Attribute at io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:160) at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117) at io.prometheus.jmx.JmxScraper.main(JmxScraper.java:329)` when I run io.prometheus.jmx.JmxScraper – Alex Saraseka Jul 23 '20 at 16:09
  • Alternatively, you can try to use Prometheus OpenCensus export. Ignite can export metrics in that format: https://apacheignite.readme.io/docs/new-metrics#opencensus – dmagda Jul 25 '20 at 01:37

1 Answers1

2

  1. Create an empty prometheus_config.yml file. Make sure the file is properly read. Trace here: https://github.com/prometheus/jmx_exporter/blob/ce04b7dca8615d724d8f447fa25c44ae1c29238b/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L75 to make sure you are using the correct file.

  2. remove the metricExporterSpi property

  3. connect to the port specified, 8080 in your case, to see all the results.

Your pattern matching rules are not allowing Apache Ignite results to show properly.

You can use debug/tracing instructions here: https://github.com/prometheus/jmx_exporter to see what is happening.

pattern rule processing is here: https://github.com/prometheus/jmx_exporter/blob/ce04b7dca8615d724d8f447fa25c44ae1c29238b/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L357

Alex K
  • 841
  • 4
  • 5
  • I added catching ClassCastException for JmxScrapper.javs now java -cp collector/target/collector*.jar io.prometheus.jmx.JmxScraper service:jmx:rmi:///jndi/rmi://127.0.0.1:49112/jmxrmi returns more then 800 items, but when I run javaagent - it gives me only 64 items – Alex Saraseka Jul 30 '20 at 11:13
  • The javaAgent most likely returns only the JVM properties. It might be an error in the javaagent. – Alex K Aug 03 '20 at 14:35
  • Yep, jmx was not catching ClassCastException. I added catch and now I see metrics (some skipped due to "TabularDataSupport cannot be cast to javax.management.Attribute", but now it's just ignoring these metrics instead of failing. – Alex Saraseka Aug 04 '20 at 15:15