I am trying to start an Apache Ignite Docker container (2.15.0 or 2.15.0-jdk11). We need to get the metrics from the container with OpenCensus.
Now the documentation says there is an OpenCensus exporter: https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics-system
And here we have the corresponding class: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/metric/opencensus/OpenCensusMetricExporterSpi.html
I did what I read in the documentation and added a few things to the default-config.xml which i can successfully use when starting my docker container with this docker-compose.yml:
services:
ignite:
image: apacheignite/ignite:2.15.0-jdk11
environment:
- IGNITE_QUIET=false
- IGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
- IGNITE_NO_ASCII=true
- JAVA_OPTS="Djava.net.preferIPv4Stack=true"
volumes:
- ./config:/opt/ignite/apache-ignite/config/
ports:
- 11211:11211
- 47100-47110:47100-47110
- 47500-47509:47500-47509
- 49112:49112
- 10800:10800
This is in the configuration file (default-config.xml):
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true"/>
<property name="metricsLogFrequency" value="5000"/>
<property name="metricExporterSpi">
<list>
<bean class="org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.log.LogExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi"/>
</list>
</property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_NODE_SEGMENTED"/>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="mycache"/>
<!-- Enable statistics for the cache. -->
<property name="statisticsEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
</beans>
So the problem now is: Basically the metrics/monitoring works, as long as I don't have that line in the default-config.xml:
<bean class="org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi"/>
Then I get this error from spring:
WARNING: Unknown module: jdk.internal.jvmstat specified to --add-exports
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.ignite.internal.util.GridUnsafe$2 (file:/opt/ignite/apache-ignite/libs/ignite-core-2.15.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.ignite.internal.util.GridUnsafe$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Ignite Command Line Startup, ver. 2.15.0#20230425-sha1:f98f7f35
2023 Copyright(C) Apache Software Foundation
class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/opt/ignite/apache-ignite/config/default-config.xml]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1150)
at org.apache.ignite.Ignition.start(Ignition.java:328)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:365)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/opt/ignite/apache-ignite/config/default-config.xml]
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:381)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:103)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:97)
at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:698)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:883)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:808)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:647)
at org.apache.ignite.Ignition.start(Ignition.java:325)
... 1 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ignite.cfg' defined in URL [file:/opt/ignite/apache-ignite/config/default-config.xml]: Cannot create inner bean 'org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi#71e9ddb4' of type [org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi] while setting bean property 'metricExporterSpi' with key [2]; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi] for bean with name 'org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi#71e9ddb4' defined in URL [file:/opt/ignite/apache-ignite/config/default-config.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:389)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:127)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:428)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:173)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1702)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1447)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:375)
... 9 more
Caused by: org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi] for bean with name 'org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi#71e9ddb4' defined in URL [file:/opt/ignite/apache-ignite/config/default-config.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1486)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:488)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:374)
... 24 more
Caused by: java.lang.ClassNotFoundException: org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:284)
at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:469)
at org.springframework.beans.factory.support.AbstractBeanFactory.doResolveBeanClass(AbstractBeanFactory.java:1551)
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1478)
... 26 more
Failed to start grid: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/opt/ignite/apache-ignite/config/default-config.xml]
Note! You may use 'USER_LIBS' environment variable to specify your classpath.
So finally my question: Am I missing a step? Do I have to manually provide the SPI class for the container? Or is there something else that I can do?
I have tried to adjust the configuration file in many ways, but this seems to be the correct structure.
Update:
I have adjusted the docker-compose.yml, so now it looks like this:
services:
ignite:
image: apacheignite/ignite:2.15.0-jdk11
environment:
- IGNITE_QUIET=false
- IGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
- IGNITE_NO_ASCII=true
- OPTION_LIBS=ignite-opencensus,ignite-web
- JAVA_OPTS="Djava.net.preferIPv4Stack=true"
#- EXTERNAL_LIBS = "C:/git/ignite-prometheus/target/ignite-prometheus-1.0.0-jar-with-dependencies.jar"
volumes:
- ./config:/opt/ignite/apache-ignite/config/
- ./libs:/opt/ignite/apache-ignite/libs/user_libs
ports:
- 11211:11211
- 47100-47110:47100-47110
- 47500-47509:47500-47509
- 49112:49112
- 10800:10800
- 9000:9000
(Note: I'm testing locally, so I have Windows paths. This will ofc not be the final version.)
Additionally I have provided a Jar file with the needed dependencies for my new beans in the configuration. (Taken from the blog post mentioned in the answer by Stephen).
So now my default-config.xml looks like this: (I excluded the beans tag for this.)
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true"/>
<property name="metricsLogFrequency" value="60000"/>
<property name="metricExporterSpi">
<list>
<bean class="org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.log.LogExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi">
<property name="period" value="60000" />
</bean>
</list>
</property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_NODE_SEGMENTED"/>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="mycache"/>
<property name="statisticsEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
<bean id="opencensusWrapper" class="org.springframework.beans.factory.config.MethodInvokingBean">
<property name="staticMethod" value="io.opencensus.exporter.stats.prometheus.PrometheusStatsCollector.createAndRegister"/>
</bean>
<bean id="httpServer" class="io.prometheus.client.exporter.HTTPServer">
<constructor-arg type="java.lang.String" value="localhost"/>
<constructor-arg type="int" value="9000"/>
<constructor-arg type="boolean" value="true"/>
</bean>
The Ignites spring framework does something with it. Because without the Jar file I would get corresponding ClassNotFoundExceptions.
Now maybe the structure of the configuration could be messed up, but I don't have any idea what I could do to reach my goal.
Note that in the logs of the Ignite container I don't see any sign of a web server being started. And no sign of port 9000 being used.
But I also don't get any errors in the container log.
