AJP Connection Failure

Question

The other night a Apache load balanced cluster of Tomcat instances because to fail when attempting to communicate over AJP.

I can not figure out why this occurred nor can I get apache to comminucate when I can find no obvious issues.

EDIT

The only part that I'm uncertain about is that port 8010 with port scan shows XMPP service. Is it supposed to appear this way? I guess I've never noticed.

My environment:

• 2 CentOs servers
• 2 tomcat instances each
• 1 tomcat AJP is 8009 and the other is 8010

All tomcats will respond to their respective HTTP port.

Nmap results:

$ ansible integra -a "nmap -p8009-8010 hip-integratc01 hip-integratc02"
hip-integratc02 | SUCCESS | rc=0 >>

Starting Nmap 6.40 ( http://nmap.org ) at 2016-05-13 11:26 EDT
Nmap scan report for hip-integratc01 (10.56.8.226)
Host is up (0.00028s latency).
rDNS record for 10.56.8.226: hip-integratc01.cat.local
PORT     STATE SERVICE
8009/tcp open  ajp13
8010/tcp open  xmpp

Nmap scan report for hip-integratc02 (10.56.8.227)
Host is up (0.00016s latency).
rDNS record for 10.56.8.227: hip-integratc02.cat.local
PORT     STATE SERVICE
8009/tcp open  ajp13
8010/tcp open  xmpp

Nmap done: 2 IP addresses (2 hosts up) scanned in 0.05 seconds

hip-integratc01 | SUCCESS | rc=0 >>

Starting Nmap 6.40 ( http://nmap.org ) at 2016-05-13 11:26 EDT
Nmap scan report for hip-integratc01 (10.56.8.226)
Host is up (0.00016s latency).
rDNS record for 10.56.8.226: hip-integratc01.cat.local
PORT     STATE SERVICE
8009/tcp open  ajp13
8010/tcp open  xmpp

Nmap scan report for hip-integratc02 (10.56.8.227)
Host is up (0.00030s latency).
rDNS record for 10.56.8.227: hip-integratc02.cat.local
PORT     STATE SERVICE
8009/tcp open  ajp13
8010/tcp open  xmpp

Nmap done: 2 IP addresses (2 hosts up) scanned in 6.54 seconds

Netstat results:

$ ansible integra -a "netstat -npl" --become
hip-integratc01 | SUCCESS | rc=0 >>
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
...       
tcp6       0      0 :::8009                 :::*                    LISTEN      7954/java           
tcp6       0      0 :::8010                 :::*                    LISTEN      11190/java   

hip-integratc02 | SUCCESS | rc=0 >>
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
...        
tcp6       0      0 :::8009                 :::*                    LISTEN      17514/java          
tcp6       0      0 :::8010                 :::*                    LISTEN      21084/java  

Gist of the Apache error_log

[Fri May 13 10:24:44.177672 2016] [proxy:error] [pid 11077] (111)Connection refused: AH00957: AJP: attempt to connect to 10.56.8.227:8010 (hip-integratc02.cat.local) failed
[Fri May 13 10:24:44.177733 2016] [proxy:error] [pid 11077] AH00959: ap_proxy_connect_backend disabling worker for (hip-integratc02.cat.local) for 60s
[Fri May 13 10:24:44.177743 2016] [proxy_ajp:error] [pid 11077] [client 10.56.8.138:6422] AH00896: failed to make connection to backend: hip-integratc02.cat.local
[Fri May 13 10:24:44.177818 2016] [proxy:error] [pid 11077] (111)Connection refused: AH00957: AJP: attempt to connect to 10.56.8.226:8010 (hip-integratc01) failed
[Fri May 13 10:24:44.177830 2016] [proxy:error] [pid 11077] AH00959: ap_proxy_connect_backend disabling worker for (hip-integratc01) for 60s
[Fri May 13 10:24:44.177835 2016] [proxy_ajp:error] [pid 11077] [client 10.56.8.138:6422] AH00896: failed to make connection to backend: hip-integratc01