Is it possible to configure JMXRemote both on localhost and remote IP?

Question

I have a java app running within Kubernetes. I sometimes need to debug it, so I used to connect to the jmx port via kubectl port-forward pods/my-java-app 9010:9010 -n my-java-app.

Our monitoring team aksed to be able to query the jmx port remotely, so I reconfigured my docker-entrypoint.sh accordingly:

    #!/bin/bash
    set -ex
    
    echo "Start docker-entrypoint.sh"
    
    JMXARGS="-Dsun.management.jmxremote.level=FINEST \
          -Dsun.management.jmxremote.handlers=java.util.logging.ConsoleHandler \
          -Djava.util.logging.ConsoleHandler.level=FINEST \
          -Dcom.sun.management.jmxremote.local.only=false \
          -Dcom.sun.management.jmxremote.ssl=false \
          -Dcom.sun.management.jmxremote.authenticate=false \
          -Dcom.sun.management.jmxremote.port=$JMX_PORT \
          -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT \
          -Djava.net.preferIPv4Stack=true" 
    
    /usr/local/openjdk-8/bin/java ${JAVA_OPTS} ${JMXARGS} \
        -XX:+HeapDumpOnOutOfMemoryError \
        -XX:+UseContainerSupport \
        -XX:+UseG1GC -DLOG_PATH=${LOG_PATH}  \
        -DAPP_LOG_FILENAME=${APP_LOG_FILENAME} \
        -jar /opt/${JARFILENAME} \
        -conf /opt/${CONFFILENAME} \
        -server -XX:CompileThreshold=5 \
        -Djava.net.preferIPv4Stack=true \
        -XX:-OmitStackTraceInFastThrow \
        -Dserver.port="${LISTENING_PORT}" \
        run myjavaapp

Since then, I am unable to use kubectl port-foward to query the jmx from my computer. Monitoring team has their monitoring server on the same local network as the Kubernetes cluster, so it's working for them, querying JMX remotley.

How may I get the best of both worlds? Meaning:

Having the port remotley available for monitoring team needs
Having the port available through the kubectl port-forward pods/my-java-app 9010:9010 -n my-java-app command for my debuging purposes

EDIT:

To go a little further, here are some tests run from the container:

root@my-java-app-864d5d487f-99qwl:/usr/local/openjdk-8# java -cp /jmxquery.jar org.nagios.JMXQuery -U service:jmx:rmi://172.29.4.202/jndi/rmi://172.29.4.202:9010/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -w 800000000 -c 900000000
JMX OK 53811560
root@my-java-app-864d5d487f-99qwl:/usr/local/openjdk-8# java -cp /jmxquery.jar org.nagios.JMXQuery -U service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:9010/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -w 800000000 -c 900000000
JMX OK 55908712

And from my computer, after a port-forward:

PS C:\dev> kubectl port-forward pods/my-java-app-864d5d487f-99qwl 9010:9010 -n my-java-app
Forwarding from 127.0.0.1:9010 -> 9010
Forwarding from [::1]:9010 -> 9010
Handling connection for 9010
Handling connection for 9010

...

(in another term:)
PS C:\dev> java -cp jmxquery.jar org.nagios.JMXQuery -U service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:9010/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -w 800000000 -c 900000000                                                     
JMX CRITICAL Connection timed out: connect

Best Regards,

score 0 · Answer 1 · answered Apr 12 '21 at 12:06

Your second connection is timing out; its not being refused. Thus, I would conclude that the JMX RMI interface (yuck by the way) is singly threaded and blocking; potentially the JMX client is staying connected between polling.

JMX RMI is also really really slow, depending if you're wanting to scrape a lot of data and which API you're client is using (the slow-and-steady method of jconsole) or the fast-and-efficient mechanism that will blow up with a serialisation exception as soon as you hit a mbean that lacks the serialisation interface.

A potentially better way (although this may require changes in your monitoring) is to use a JMX Agent based mechanism. This is the mechanism that most monitoring providers (eg. the likes of Data Dog, Prometheus) use to collect data. Also a much better security boundary too potentially.

I've only had experience with Prometheus's jmx_exporter, but which probably won't be suitable for Nagios. I would fully expect that there will be other JMX Agents that would not have the limitations that JMX RMI has... in particular I think Jolokia is popular for exposing JMX over HTTP

Another possibility: if using jconsole (which can inject itself as an agent using the Attach API (?), which is why you don't need to set up JMX RMI to use jconsole) you may find that works around your contention issue with the JMX RMI port.

Is it possible to configure JMXRemote both on localhost and remote IP?

1 Answers1