1

I'm trying to use JMX with activeMQ for monitoring so far I've been using this and this as a reference but so far I'm unable to connect to jmx remotely and also I don't see any mention of jmx url in activemq logs. I'm wondering if there is another way to make sure jmx is working? is it supposed to be indicated in activemq logs? PS I'm using jdk1.7 and activeMQ 5.14.2.

Thanks in advance!

EDIT

I set useJmx="true" in my activemq.xml file:

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="primary" useJmx="true" dataDirectory="${activemq.data}">

I tried two steps:

FIRST

I tried changing management context from createConnector="false" to :

<managementContext>
   <managementContext createConnector="true" connectorPort="1099"/>
</managementContext>

FOR FIRST TIME THE PORT IS OPEN AND ACTIVEMQ RUNS FINE AND JMX URL GETS REPORTED IN LOGS ALTHOUGH I CAN NOT CONNECT IT TO IT REMOTLEY BUT IM ASSUMING ITS WORKING

SECOND

I reverted back the changes I made for managmentContext and I tried setting:

ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/jmx.password -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/jmx.access"

in bin/activemq script and I set a username in conf/jmx.access file as:

admin readwrite

And also have set a password in conf/jmx.password:

admin activemq

NOW ACTIVEMQ IS NOT RUNNINT AT ALL BUT IT WILL RUN IF I SET AUTHENTICATE=FALSE AND DELETE JMX.ACCESS AND JMX.PASSWORD CONFIGURATION IN BIN/ACTIVEMQ FILE BUT I NEED USER NAME AND PASSWORD FOR SECURITY REASONS I found this post which has the exact same issue as mine. any ideas?

Community
  • 1
  • 1
tkyass
  • 2,968
  • 8
  • 38
  • 57
  • can you post your activemq.xml .ActiveMQ logs the JMX url at INFO level during startup. – Hassen Bennour Dec 14 '16 at 07:03
  • Hi @HassenBennour thanks for your comment, I updated my question. – tkyass Dec 14 '16 at 18:03
  • With the first step, What was the error message in jconsole and which hostname did you used. Can you try a telnet activemqhost 8161. Have you introduced user/pwd ? Can you confirm that your ActiveMQ is running in another different host and what is the hostname. – Hassen Bennour Dec 14 '16 at 18:23
  • connection failed or connection did not succeed. I used the ip address with the port number of jmx and also tried using full url `service:jmx:rmi:///jndi/rmi://localhost:8161/jmxrmi` with specifying ip address and port number. I have jmx.access and jmx.password files in my conf directory but I didn't point to them anywhere in the config files so I tried connecting without username and password. Also tried with and got same error. – tkyass Dec 14 '16 at 18:38
  • I have a question.. do I have to do step FIRST and SECOND together? or doing one is enough at a time? – tkyass Dec 14 '16 at 18:39
  • If you use them together you will have 2 mbeans servers, not a good idea. When you said "used the full url with localhost " this means that your AMQ is running in localhost ? – Hassen Bennour Dec 14 '16 at 18:46
  • so the exact url is service:jmx:rmi:///jndi/rmi://10.10.10.16:12333/jmxrmi which points to the vm's ip and port number opened for jmx. I'm trying to connect to from my local machine to the jmx running in vm. activeMQ is using the same ip address. I hope this answers ur question – tkyass Dec 14 '16 at 18:55
  • Port 12333 ? Not 8161 ?? Default jmx port is 1099 – Hassen Bennour Dec 14 '16 at 18:57
  • yeah I used 12333 because of security issues. we don't have port 1099 exposed for remote access. – tkyass Dec 14 '16 at 19:00
  • Ok so you have updated your config to use this port ? connectorPort="12333" – Hassen Bennour Dec 14 '16 at 19:03
  • correct I updated connectorPort="12333" – tkyass Dec 14 '16 at 19:05
  • Can you try in terminal : telnet 10.10.10.16 12333 . Or nmap -p12333 10.10.10.16 and post the result – Hassen Bennour Dec 14 '16 at 19:06
  • I tried telneting and screen turns to all black which I assume it means that it was able to connect. because before that I had some issues with configuration when I tried telnet it used to give me connection error message but now it doesn't – tkyass Dec 14 '16 at 19:13
  • Ok the port is open. Can you try with VisualVM – Hassen Bennour Dec 14 '16 at 19:16
  • its giving me error message .. can not connect to 10.10.10.16 12333 using service:jmx:rmi:///jndi/rmi://10.10.10.16 12333/jmxrmi – tkyass Dec 14 '16 at 19:44
  • I have a question if `JMX consoles can connect to service:jmx:rmi:///jndi/rmi://10.10.10.16 12333/jmxrmi | org.apache.activemq.broker.jmx.ManagementContext | JMX connector` is showing up in the logs that means jmx is working fine right? I'm asking because it might be port accessibility issue so I might need to change the permissions – tkyass Dec 14 '16 at 19:45
  • Yes but since telnet works fine the problem is somewhere else i think, can you try with jconsole and add user/pwd from the file conf/jmx.password – Hassen Bennour Dec 14 '16 at 20:32
  • still no luck .. I noticed also that I have a warn that storeopenwireversion is set to work with old version 6 instead of 11 so it reverted back to older version and this might cause losing some new feature .. I know this is related to KahaDB but do you think it might impact what I'm trying to do? I tried setting storeOpenWireVersion="11" in my activemq.xml file but it gave me error that its not allowed to appear in KahaDB – tkyass Dec 20 '16 at 16:05
  • As i see on github AMQ 5.14.x uses storeOpenWireVersion="11" by default. i think there is no relation with jmx and what you try to do. Have you tried with parameters of my answer and still not working ? – Hassen Bennour Dec 20 '16 at 18:34
  • yes, I tried setting a user in jmx.access file and setting a pwd in jmx.password both under /conf path. when I set remote jmx in env file I see the port in listen status but I dont see any indication that jmx has started in log files. I will update my question to show the env file and activemq main script how it looks – tkyass Dec 20 '16 at 18:53
  • I think that you have set createConnector="false" for not seeing jmx logs. I added 2 jvm parameters try with adding them to env file with older jmx opts. I think it is network or server config problem – Hassen Bennour Dec 20 '16 at 20:35

2 Answers2

4

Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the JVM: -Dcom.sun.management.jmxremote.authenticate=false like you done in second test but you need to add system property -Dcom.sun.management.jmxremote

Try to add these jvm param to env file and update host ip

-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=X.X.X.X

UPDATE

SO, to resume, i think that the FIRST step you tried is the best, for making it working these are the steps :

  1. revert all jmx env file changes, like this :

        # ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
        # ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
        # ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
        # ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
        ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
    
  2. <broker useJmx="true" ...

<managementContext>
  <managementContext createConnector="true"  connectorPort="1099" />
</managementContext>

verify that in AMQ logs you have

INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi | org.apache.activemq.broker.jmx.ManagementContext | JMX connector

NOTE : Assuming that 10.10.10.16 is the IP of AMQ host.

  1. try to connect with jconsole from another machine than AMQ host with url "service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi" without user/pwd.

  2. if you cannot connect, try like this :

    <managementContext>
      <managementContext createConnector="true"  connectorPort="1099"  connectorHost="10.10.10.16" />
    </managementContext>
    

verify that in AMQ logs you have

INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi | org.apache.activemq.broker.jmx.ManagementContext | JMX connector

  1. retry to connect, step 4

  2. at this step normally you can connect with jconsole.

  3. if you want to add security and authorizations, use this :
<managementContext>
  <managementContext  createConnector="true"  connectorPort="1099"  connectorHost="10.10.10.16" >
      <property xmlns="http://www.springframework.org/schema/beans" name="environment">
          <map xmlns="http://www.springframework.org/schema/beans">
              <entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.password.file"
                     value="${activemq.conf}/jmx.password"/>
              <entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.access.file"
                     value="${activemq.conf}/jmx.access"/>
          </map>
      </property>
  </managementContext>
</managementContext>

Please try these steps and let me know in which one you fails to connect and provide error message from jconsole.

Hassen Bennour
  • 3,885
  • 2
  • 12
  • 20
  • Hi Hassen, I have edited my question and this is the latest issue I'm getting could you please check and let me know if you have any thoughts. thanks! – tkyass Dec 22 '16 at 18:19
  • Can you connect with jconsole IF you SET AUTHENTICATE=FALSE AND DELETE JMX.ACCESS AND JMX.PASSWORD CONFIGURATION IN BIN/ACTIVEMQ FILE ?? – Hassen Bennour Dec 22 '16 at 20:24
  • i updated my answer with a step by step config to try – Hassen Bennour Dec 23 '16 at 11:54
  • thanks a lot Hassen!!! it finally worked .. really appreciate your help and continues replies! – tkyass Jan 11 '17 at 22:19
1

A couple troubleshooting steps:

  1. Start jconsole or visualvm on the same system and connect using the "pid" attach method. Browse the MBeans and confirm org.apache.activemq beans are present

  2. Run netstat -na and confirm ports 1099 (and 44444) are in LISTEN

  3. Look at logs and confirm you do not have any "java.net.BindException: Address already in use.." messages that indicate a port conflict with an already running Java process.

Edit bin/env to configure JMX (this disables requiring SSL, sets the port to 1099 and disables requiring username and password.

ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"`
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
Matt Pavlovich
  • 4,087
  • 1
  • 9
  • 17
  • Thanks Matt for your inout .. I tried your steps and for the first step I'm unable to connect using jconsole remotely so I dont see the MBeans. for the second I have port 1099 listening but couldnt fine 44444, and why checking for port 44444? for final step I dont have any conflict in the log. please see my updated question. thanks. – tkyass Dec 14 '16 at 18:05
  • You have a conflict in configurations. Either use the " – Matt Pavlovich Dec 14 '16 at 18:32
  • if I use the defaults then how can I specify the jmx port? – tkyass Dec 14 '16 at 18:56
  • You specify it in the bin/env. Updated answer to include editing the bin/env. The advantage of the bin/env is you can set no ssl and the username+password files as well – Matt Pavlovich Dec 14 '16 at 20:22
  • I tried your settings and I'm not even sure if they're working or not .. could you please take a look at my updated question – tkyass Dec 20 '16 at 19:17
  • The log may not show anything unless you turn on DEBUG. JMX is core to Java piece, and not ActiveMQ specific. If 1099 is LISTEN than it should be running. Try running jconsole and do the pid attach to the process. If you see the org.apache.activemq then you are good. – Matt Pavlovich Dec 20 '16 at 19:26
  • do you mean I should be running jconsole in my linux machine where AMQ is running or the remote machine (my local) which I'm trying to use for jconsole connection? I tried running jconsole Pid# in my linux and connecting to it from my local using jconsole. It doesnt establish any connections. Is Jconsole record logs somewhere? – tkyass Dec 20 '16 at 19:31
  • For remote access try jconsole on local linux machine and use the url format: hostname:1099 – Matt Pavlovich Dec 20 '16 at 19:33
  • Hi Matt, I have updated my question with the current issue I'm facing in more details could you take a look and let me know you your thoughts. thanks! – tkyass Dec 22 '16 at 18:20
  • The jmx.access and jmx.password files need to have specific permissions. chmod 600 jmx.password See: http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html – Matt Pavlovich Dec 22 '16 at 19:03
  • 777 is the exact opposite of what's needed. It needs to be restricted, not wide open. Need more logs or debugging to provide other guidance – Matt Pavlovich Dec 22 '16 at 19:15