I have checkmk running in our lan. I want to monitor a linux vm in the dmz. So i installed the agent on the linux vm, enabled ipv4 for it, but checkmk can not find any services since the agent was not found running.
All lan-hosts can reach hosts in the dmz over a router with a firewall. I added a static route on the linux vm as well as a firewall rule so that the linux vm should be able to reach checkmk on the monitoring port, what IMHO shouldn't be necessary. It does not work anyway.
So i allowed all ports and all protocols from the linux vm in the dmz to checkmk in the firewall - but it does not work anyway.
I can ping from the linux vm to checkmk, i can use ssh from the linux vm to checkmk, but TCP Port 6556 seems to be filtered (nmap says that). I also can't access the port with telnet.
I monitored the traffic with tcpdump while trying to find the agent with checkmk with the following result:
checkmk (lan) 192.168.200.93
11:52:10.161596 IP 192.168.200.93.54568 > 192.168.250.5.6556: Flags [S], seq 1804524389, win 64240, options [mss 1460,sackOK,TS val 929690233 ecr 0,nop,wscale 7], length 0
11:52:11.181705 IP 192.168.200.93.54568 > 192.168.250.5.6556: Flags [S], seq 1804524389, win 64240, options [mss 1460,sackOK,TS val 929691253 ecr 0,nop,wscale 7], length 0
11:52:13.197715 IP 192.168.200.93.54568 > 192.168.250.5.6556: Flags [S], seq 1804524389, win 64240, options [mss 1460,sackOK,TS val 929693269 ecr 0,nop,wscale 7], length 0
11:52:27.171005 IP 192.168.200.93 > 192.168.250.5: ICMP echo request, id 2259, seq 0, length 76
11:52:27.171923 IP 192.168.250.5 > 192.168.200.93: ICMP echo reply, id 2259, seq 0, length 76
11:52:27.171994 IP 192.168.200.93 > 192.168.250.5: ICMP echo request, id 2259, seq 1, length 76
11:52:27.172407 IP 192.168.250.5 > 192.168.200.93: ICMP echo reply, id 2259, seq 1, length 76
11:52:27.172439 IP 192.168.200.93 > 192.168.250.5: ICMP echo request, id 2259, seq 2, length 76
11:52:27.172875 IP 192.168.250.5 > 192.168.200.93: ICMP echo reply, id 2259, seq 2, length 76
11:52:27.172904 IP 192.168.200.93 > 192.168.250.5: ICMP echo request, id 2259, seq 3, length 76
11:52:27.173326 IP 192.168.250.5 > 192.168.200.93: ICMP echo reply, id 2259, seq 3, length 76
11:52:27.173363 IP 192.168.200.93 > 192.168.250.5: ICMP echo request, id 2259, seq 4, length 76
11:52:27.173784 IP 192.168.250.5 > 192.168.200.93: ICMP echo reply, id 2259, seq 4, length 76
linux vm (dmz) 192.168.250.5
11:52:23.672563 ARP, Request who-has 192.168.250.5 tell 192.168.250.2, length 46
11:52:23.672614 ARP, Reply 192.168.250.5 is-at 02:6a:ef:43:81:37 (oui Unknown), length 28
11:52:27.174672 IP 192.168.250.2 > 192.168.250.5: ICMP echo request, id 2259, seq 0, length 76
11:52:27.174742 IP 192.168.250.5 > 192.168.250.2: ICMP echo reply, id 2259, seq 0, length 76
11:52:27.175350 IP 192.168.250.2 > 192.168.250.5: ICMP echo request, id 2259, seq 1, length 76
11:52:27.175362 IP 192.168.250.5 > 192.168.250.2: ICMP echo reply, id 2259, seq 1, length 76
11:52:27.175803 IP 192.168.250.2 > 192.168.250.5: ICMP echo request, id 2259, seq 2, length 76
11:52:27.175815 IP 192.168.250.5 > 192.168.250.2: ICMP echo reply, id 2259, seq 2, length 76
11:52:27.176264 IP 192.168.250.2 > 192.168.250.5: ICMP echo request, id 2259, seq 3, length 76
11:52:27.176275 IP 192.168.250.5 > 192.168.250.2: ICMP echo reply, id 2259, seq 3, length 76
11:52:27.176728 IP 192.168.250.2 > 192.168.250.5: ICMP echo request, id 2259, seq 4, length 76
11:52:27.176739 IP 192.168.250.5 > 192.168.250.2: ICMP echo reply, id 2259, seq 4, length 76
The router has the ip 192.168.250.2 in the dmz. So as you can see, checkmk sends a SYN to 192.168.250.5:6556 to find the agent - but the packages don't reach the linux vm. After that, checkmk send a ping/ICMP echo request to the linux vm, what worked since the linux vm answered with a ICMP echo reply. The ICMP packages are correctly forwarded by the router, but the TCP SYN Packages over Port 6556 are not.
What could be the problem here? The router is a turris omnia which is based on OpenWRT. So there is the possibility to add iptables-rules.
Any ideas?
