Zabbix JMX Tomcat monitoring

Question

I have been trying to setup Zabbix to monitor my 2 tomcat servers on 2 different Amazon EC2 machines, but in vain.

The Z on the host is green, however te JMX is red with these errors
- ZBX_TCP_READ() failed: [4] Interrupted system call
- Someother error [111] connection refused and many such errors, one after another, in the sense I resolve an error to see one more new error popping up.

These are some assumptions
All the machines run Ubuntu 12.10 and later
Server's IP address: 66.55.12.120 (Runs Zabbix server v2.2.4 (revision 46772) (23 June 2014) )
Agent's IP address: 87.52.45.198 ( Runs Zabbix agent v2.2.2 (revision 42525) (12 February 2014) )
My local machine's IP address: 76.89.54.111

Here is what I've done so far.

On Server Side:
1) Installed Zabbix_server using sudo apt-get install zabbix-server-mysql.
2) The GUI, mysql database all have been installed and configured.
3) The following are the only 3 changes that I've made in the file
/etc/zabbix/zabbix_server.conf

  ... 
  JavaGateway=localhost
  JavaGatewayPort=10052
  StartJavaPollers=5
  ...

4) The Zabbix Java gateway was installed using sudo apt-get install zabbix-java-gateway.
5) The following are the only 3 changes that I've made in the file
/etc/zabbix/zabbix_java_gateway.conf

  ... 
  LISTEN_IP="127.0.0.1"
  LISTEN_PORT=10052
  START_POLLERS=5
  ...

On Client Side: 1) Installed Zabbix Client using sudo apt-get install zabbix-agent
2) The following are the only 3 changes that I've made in the file

  /etc/zabbix/zabbix_agentd.conf 
  ... 
  Server=66.55.12.120
  StartAgents=5
  ServerActive=66.55.12.120:10051
  Hostname=Security-test-JMX-EC2
  ... <br />

3) The Hostname is the same as the one that is mentioned while creating the Host on the GUI.

I believe that there are some issues with the IP and ports. So, here are the outbound rules for both the machines as obtained from Amazon EC2 Security Groups for the machines

OUTBOUND RULES for SERVER SECURITY GROUP:
Type      Protocol  Port   Source           Reasoning
Custom-   TCP       8080   0.0.0.0/0
TCP Rule    

All ICMP  All       N/A    0.0.0.0/0    

Custom-   TCP       10052  27.52.52.128/32   For access from Agent
TCP Rule

Custom-   TCP       8081   76.84.120.130/32  To access Zabbix GUI from-
TCP Rule                                     -my local machine's web browser

Custom-   TCP       10051  27.52.52.128/32   As the agent responds to- 
TCP Rule                                     -the server on Port 10051TCP Rule-
                                             -Must allow inbound communications-
                                             - from the agent.

Custom-   TCP       11000  27.52.52.128/32   The agent's JMX reporting-
TCP Rule                                     -happens on port 11000(not on 12345).

OUTBOUND RULES for CLIENT SECURITY GROUP:
Type    Protocol Port   Source
HTTPS     TCP   443     0.0.0.0/0

Custom-   TCP   10050   66.55.12.120/32
TCP Rule

Custom-   TCP   10052   66.55.12.120/32
TCP Rule

Custom-   TCP   11000   66.55.12.120/32
TCP Rule

HTTP      TCP   80  76.89.54.111/32

Custom-   TCP   8080    76.89.54.111/32
TCP Rule

Custom-   TCP   8443    76.89.54.111/32
TCP Rule

What am I missing? Please guide me. Any help is appreciated.

Thanks
Goutham

score 7 · Answer 1 · answered Sep 30 '14 at 14:41

If you can, then run VisualVm (probably using a tunneled X session) on the zabbix host, and see if you can connect to the target JVM with that. If you can't connect from that, you won't be able to connect from Zabbix.

Try with the following CATALINA_OPTS, replacing with the IP on the target that you want JMX to listen on:

export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=falseom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<LOCAL_IP>"

This will disable all JMX security so be aware!

Once you hopefully get it to connect, the "Tomcat JMX" items in Zabbix are also all incorrect! e.g.

Incorrect Zabbix default:

jmx["Catalina:type=GlobalRequestProcessor,name=http-8080",bytesReceived]

Correct entry:

jmx["Catalina:type=ThreadPool,name=\"http-bio-8080\"", bytesReceived]

Note the escaped quotes and incorrect thread name. Add the Mbeans plugin to VisualVM, and use that to browse the MBeans on the target VM, and check the Zabbix names.

It does work eventually, but is a real pain to setup. Zabbix is however one of the few open source monitoring tools that supports JMX at all!

Instead of installing VisualVm, I was able to connect using `jconsole`, which was already installed on my system. `jconsole` appears to a standard part of the JDK (source:https://en.wikipedia.org/wiki/JConsole). — G Mawr, Dec 04 '14 at 11:08

score 0 · Answer 2 · answered Aug 05 '14 at 10:11

0

By default, JMX does not work very well with firewalls. You might find related bug reports on Zabbix tracker useful: ZBX-5326 and ZBX-6815. The first one contains a workaround for Tomcat which might work for you.

answered Aug 05 '14 at 10:11

asaveljevs

2,220
1
16
13

score 0 · Answer 3 · answered Jan 28 '15 at 23:48

@gvatreya wrote:

Server: (Runs Zabbix server)

Agent: (Runs Zabbix agent)

It looks like you have to start Zabbix Java gateway as well on host where it is installed (it is a daemon/service).

I configured as follows:

Server: (Runs Zabbix server, Zabbix Java gateway)

Agent: (Runs Zabbix agent)

I think it is possible to install it on a dedicated host.

score 0 · Answer 4 · answered Mar 17 '16 at 17:54

0

Have you tried adding -Djava.net.preferIPv4Stack=true to the VM options?

answered Mar 17 '16 at 17:54

Sam

71
5

How would this help? What does it do? Any reference? – dakab Mar 17 '16 at 18:13
If you're using Java 8 on a RHEL/CentOS 7.x, port 10051/10052 will be listening on tcp6 stack. By forcing tcp(4) and having the port open in firewalld, you'll have your agent (e.g zapcat) properly connected to the server. – Sam Mar 17 '16 at 21:59

score 0 · Answer 5 · answered Jun 26 '17 at 20:27

to make it work add next java_opts to your tomcat startup script

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.port=2345 
-Dcom.sun.management.jmxremote.rmi.port=12345 
-Djava.rmi.server.hostname=<tomcat_hostname>

Zabbix JMX Tomcat monitoring

5 Answers5

Linked