RHEL6: httpd can't bind to port

Question

We have RHEL 6.5 server which serves as local YUM server and uses httpd for the purpose. It's running on port 6809. And until yesterday it was running fine. One of my colleague was doing some additional deployment on different server which is supposed to use the YUM server, but the yum couldn't connect. He admits, that he was also doing some work on the YUM server, but he's not aware of doing something related to httpd configuration.

So we realized that the httpd on the YUM server isn't running and refuses to start:

# service httpd start
Starting httpd: (98)Address already in use: make_sock: could not bind to address 172.29.84.41:6809
no listening sockets available, shutting down
Unable to open logs
                                                           [FAILED]

We've already checked that the port 6809 isn't occupied with lsof and netstat:

lsof -iTCP -sTCP:LISTEN -P -n

and

netstat -tupan | grep -Ei LISTEN

All clear, no other process.

We've also greped for Listen in /etc/httpd directory and there is only single directive with 6809 port:

httpd# grep -Ri listen *
conf/httpd.conf:# Listen: Allows you to bind Apache to specific IP addresses and/or
conf/httpd.conf:# Change this to Listen on specific IP addresses as shown below to
conf/httpd.conf:#Listen 12.34.56.78:80
conf/httpd.conf:#Listen 80
conf.d/ssl.conf:# When we also provide SSL we have to listen to the
conf.d/ssl.conf:#Listen 443
conf.d/yum.conf:Listen yum-server:6809
logs/test.log:read(4, "Listen yum-server:6809\nServe"..., 4096) = 496
logs/test.log:listen(3, 511)                          = 0
logs/test.log:write(2, "no listening sockets available, "..., 46no listening sockets available, shutting down
Binary file modules/mod_info.so matches
Binary file modules/mod_wsgi.so matches
Binary file modules/mod_proxy_ftp.so matches
Binary file modules/mod_cgid.so matches

So it isn't a duplicate.

The IP is local to the server.

No httpd process is running either.

We've also reinstalled the httpd RPM.

SELinux isn't running:

# sestatus
SELinux status:                 disabled

Please, what else could be wrong?

Thank you very much for your advices.

Add 2:

To @André Fernandes:

# lsof -n -P -i :6809
# find /etc/httpd -type f | xargs grep Listen
/etc/httpd/conf.d/ssl.conf:#Listen 443
/etc/httpd/conf.d/yum.conf:Listen yum-server:6809
/etc/httpd/conf/httpd.conf:# Listen: Allows you to bind Apache to specific IP addresses and/or
/etc/httpd/conf/httpd.conf:# Change this to Listen on specific IP addresses as shown below to
/etc/httpd/conf/httpd.conf:#Listen 12.34.56.78:80
/etc/httpd/conf/httpd.conf:#Listen 80

To @MadHatter I've shortened the output and removed real names. But the alias yum_server is there:

# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.somewhe.re localhost trap-host
172.29.84.41    <FQDN and name> yum-server <other aliases>
...

Add 3:

The ip addr show output:

# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether 3c:4a:92:ed:c0:28 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether 3c:4a:92:ed:c0:28 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP qlen 1000
    link/ether 68:b5:99:c6:42:4c brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP qlen 1000
    link/ether 68:b5:99:c6:42:4c brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond2 state UP qlen 1000
    link/ether 68:b5:99:c6:42:4e brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond2 state UP qlen 1000
    link/ether 68:b5:99:c6:42:4e brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 3c:4a:92:ed:c0:28 brd ff:ff:ff:ff:ff:ff
    inet 172.29.84.41/26 brd 172.29.84.63 scope global bond0
9: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 68:b5:99:c6:42:4c brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global bond1
10: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 68:b5:99:c6:42:4e brd ff:ff:ff:ff:ff:ff
    inet 10.1.212.71/24 brd 10.1.212.255 scope global bond2

Possible duplicate of https://serverfault.com/questions/875500/unable-to-stop-an-invisible-socket-on-port-80 — Gerald Schneider, Oct 06 '17 at 10:33
Similar, but not duplicate. Even if we tried to change the port to different, httpd still reports that it can't bind. — Jaroslav Kucera, Oct 06 '17 at 10:37
Please read the answer on the linked question. The problem was not the port, but multiple bind directives on the apache config. — Gerald Schneider, Oct 06 '17 at 10:46
Yes, that's what we've checked as well. It's in the original question: "We've also greped for Listen in /etc/httpd directory and there is only single directive with 6809 port. So it isn't duplicit" — Jaroslav Kucera, Oct 06 '17 at 10:49
Firstly, you mean duplicate, not duplicit (which is something completely different). Secondly, what you report doesn't rule out overlapping Listen directives; we need to *see what you see*, not merely be told what you think you know. Editing into your question the output of `grep -i listen /etc/httpd/conf/httpd.conf /etc/httpd/conf.d/*` would be more dispositive. Thirdly, the same is true for other things you think you know. I'm personally very suspicious of SELinux, so I'd like to see *how* you think you've established that it's not the cause. Show your evidence. — MadHatter, Oct 06 '17 at 11:09
If you stop your yum httpd instance and do `lsof -n -P -i :6809`, do you get any output? also search recursively through the configs with `find /etc/httpd -type f | xargs grep Listen`. — André Fernandes, Oct 06 '17 at 11:10
@MadHatter I understand, we could have missed something. I've added grep and sestatus outputs into the question. — Jaroslav Kucera, Oct 06 '17 at 11:19
Thanks, that deals with my SELinux suspicions. Could you show how you know that the server has an ip address to which the name `yum-server` resolves? — MadHatter, Oct 06 '17 at 11:34
Thanks, that shows what address the name is likely (though not guaranteed, absent `nsswitch.conf`) to resolve to, but it doesn't prove the server has that address. Could we get the output of `ip addr show`? — MadHatter, Oct 06 '17 at 12:22
@MadHatter The ip addr show output has been added. And in the /etc/nsswitch.conf, there is "hosts: files dns" — Jaroslav Kucera, Oct 06 '17 at 12:56
Thanks, I agree that box is ticked. I'm sorry to have to go through all the basics again, but that's the nature of the enquiry. Could we see that when the web server is down (ie, now) and you, from the server itself, do `telnet 172.29.84.41 6809`, you get `Connection refused`? — MadHatter, Oct 06 '17 at 14:03

score 0 · Accepted Answer · answered Oct 09 '17 at 09:00

We've found the reason. It was hidden in the /etc/hosts. The IP and all the line content was there listed twice. Among other entries required by our products, there was the entry added again. When we commented out the second entry, httpd started to behave as expected and stopped double-bind attempts.

RHEL6: httpd can't bind to port

1 Answers1