0

I'm hosting a clients wordpress site on a CentOS 7 server, with apache 2.4.6, hosted behind a cloudflare proxy. I set everything up about 8 months ago, and it has been running fine ever since.

At arround 5:30 this morning the website went down, showing timeouts. I checked the logs, in /var/log/httpd/error_log I found the following entries, around the time the site stopped working:

[Thu Sep 19 04:35:57.343495 2019] [mpm_prefork:notice] [pid 13447] AH00170: caught SIGWINCH, shutting down gracefully
[Thu Sep 19 04:35:58.598479 2019] [core:notice] [pid 22161] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Thu Sep 19 04:35:58.602722 2019] [suexec:notice] [pid 22161] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
[Thu Sep 19 04:35:58.638160 2019] [lbmethod_heartbeat:notice] [pid 22161] AH02282: No slotmem from mod_heartmonitor
[Thu Sep 19 04:35:58.701474 2019] [mpm_prefork:notice] [pid 22161] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 configured -- resuming normal operations
[Thu Sep 19 04:35:58.701545 2019] [core:notice] [pid 22161] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Since then, the only entries in the access_log have been from making local requests.

In order to debug, I've currently disabled the cloudflare proxy (so the DNS points directly to my server now) and I only have an index.html file in the web root. if I run wget mydomain.com locally on the machine, I get the contents of my test index.html just fine, and an entry appears in the access_log. If I try the same thing on an external server, I get the following error and no entry in the access_log:

(mydomain.com)|<the server's ip>|:80... failed: Resource temporarily unavailable

I've double checked the iptables config, but it hasn't been modified since May, so I'm pretty sure it's not a firewall issue.

I've also restarted httpd, mariadb and rebooted the server, to no avail.

Is there anything else that can be causing these issues with apache?

If it helps, the server is hosted in rackspace cloud, and the configs are as close to default as possible; I've not set up VirtualHosts or anything like that.

Here's the output of netstat -l -n -t -p

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1459/master
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      1383/mysqld
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1500/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1459/master
tcp6       0      0 :::443                  :::*                    LISTEN      1110/httpd
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::80                   :::*                    LISTEN      1110/httpd
tcp6       0      0 :::22                   :::*                    LISTEN      1500/sshd

nmap output

Not shown: 999 filtered ports

PORT   STATE SERVICE VERSION

22/tcp open  ssh     OpenSSH 7.4 (protocol 2.0)
Sam
  • 121
  • 4
  • Can you edit your question and add the output of `nmap -Pn ` (run from another server, and you may need to install it first) and `netstat -l -n -t -p` (run on the server in question). – Halfgaar Sep 19 '19 at 10:23
  • Certainly! I've just added the netstat output. I had to install nmap, and it gave a lot of `socket_bindtodevice: Protocol not available Problem binding to interface , errno: 92` errors when I ran it. Is this an issue with my local machine, or to be expected? – Sam Sep 19 '19 at 10:31
  • That's a local issue. Can you run it from somewhere else? Or as root? – Halfgaar Sep 19 '19 at 10:33
  • Ah, I was running it in the windows ubuntu, which was probably causing the issue. Just tried zenmap instead, and added the output to the question – Sam Sep 19 '19 at 10:41
  • I think it might have highlighted an issue too, as only port 22 appeared. I'm assuming ports 80 and 443 should be on the list too for http access to work? – Sam Sep 19 '19 at 10:42
  • @Halfgaar It's fixed! the http service was missing from the firewall, so I ran `firewall-cmd --add-service=http --permanent` and now everything works again. I guess I forgot to add the `--permanent` when I was setting up the server, and something triggered a reboot this morning and I lost all of my rules. So I take it iptables and firewal are two different systems then? – Sam Sep 19 '19 at 10:59
  • Also, apologies, I'm unsure of stackexchange ettiquette in this situation. Should I post the answer myself? I probably wouldn't have solved it without your input though, so you deserve the credit. – Sam Sep 19 '19 at 11:02
  • probably `firewall` uses iptables. I always just do iptables manually. As for your answer: you can post it yourself and I'll upvote. You deserve some credit yourself, and (whisper-mode on) I think low repuration on SO sites can be somewhat of a hole you can't get out of. – Halfgaar Sep 19 '19 at 11:48

1 Answers1

2

After looking at the output of nmap, it becomes apparent that the http(s) ports aren't being exposed. This is further confirmed by running firewall-cmd --list-all on the server:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0 eth1
  sources:
  services: dhcpv6-client ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

to remedy this, run: firewall-cmd --add-service=https --permanent. You can confirm it works, by running firewall-cmd --list-all again, and seeing if http appears in the services: section.

After this, apache chan see all external requests again.

Sam
  • 121
  • 4