Apache's PidFile directory is removed every boot

Question

Every time the server is rebooted, the directory /run/httpd is removed. /run is a tmpfs filesystem, therefore, mounted in RAM.

/run/httpd is created only during installation. When I start httpd after reboot, the directory isn't recreated.

Server has installed CentOS 7 and the oficial repository Apache package (v. 2.4.6-18).

After Apache install and start, the directory is as bellow:

# ls -alR /run/httpd/
/run/httpd/:
total 8
drwx--x---.  3 root   apache  120 Sep 30 08:39 .
drwxr-xr-x. 28 root   root   1020 Sep 30 08:37 ..
-rw-r--r--.  1 root   root      8 Sep 30 08:39 authdigest_shm.2953
drwx------.  2 apache apache   40 Jul 23 10:48 htcacheclean
-rw-r--r--.  1 root   root      5 Sep 30 08:39 httpd.pid
srwx------.  1 apache root      0 Sep 30 08:39 wsgi.2953.0.1.sock

/run/httpd/htcacheclean:
total 0
drwx------. 2 apache apache  40 Jul 23 10:48 .
drwx--x---. 3 root   apache 120 Sep 30 08:39 ..

But after reboot:

# ls -l /run/httpd
ls: cannot access /run/httpd: No such file or directory

And here are the logs when trying to restart Apache:

/var/log/httpd/error_log

[Tue Sep 30 09:30:32.310825 2014] [core:notice] [pid 3370] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Sep 30 09:30:32.312072 2014] [suexec:notice] [pid 3370] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Sep 30 09:30:32.330380 2014] [auth_digest:notice] [pid 3370] AH01757: generating secret for digest authentication ...
[Tue Sep 30 09:30:32.330421 2014] [auth_digest:error] [pid 3370] (2)No such file or directory: AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.3370
[Tue Sep 30 09:30:32.330440 2014] [auth_digest:error] [pid 3370] (2)No such file or directory: AH01760: failed to initialize shm - all nonce-count checking, one-time nonces, and MD5-sess algorithm disabled
[Tue Sep 30 09:30:32.330445 2014] [:emerg] [pid 3370] AH00020: Configuration Failed, exiting
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.

/var/log/message

Sep 30 08:56:09 brejetuba2 systemd: Starting The Apache HTTP Server...
Sep 30 08:56:09 brejetuba2 systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Sep 30 08:56:09 brejetuba2 systemd: Failed to start The Apache HTTP Server.
Sep 30 08:56:09 brejetuba2 systemd: Unit httpd.service entered failed state.

/var/log/audit/audit.log

Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
type=SERVICE_START msg=audit(1412083740.602:469): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="httpd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

When I create the directory manually, Apache starts:

# mkdir /run/httpd
# systemctl restart httpd
# ls -lRa /run/httpd/
/run/httpd/:
total 8
drwxr-xr-x.  2 root   root  100 Sep 30 09:36 .
drwxr-xr-x. 28 root   root 1020 Sep 30 09:36 ..
-rw-r--r--.  1 root   root    8 Sep 30 09:36 authdigest_shm.3452
-rw-r--r--.  1 root   root    5 Sep 30 09:36 httpd.pid
srwx------.  1 apache root    0 Sep 30 09:36 wsgi.3452.0.1.sock

And after reboot, it's gone again.

Any thoughts on why this is happening?

I ran into an issue very similar to this and the bug that was reported as fixed in 2017 here https://bz.apache.org/bugzilla/show_bug.cgi?id=54622. I was able to manually delete the _shm.* files that apache was erroring on and get past the issue. — Shyam Habarakada, Dec 22 '20 at 18:43

score 9 · Accepted Answer · answered Sep 30 '14 at 17:25

The issue was that, when installing Apache, group apache wasn't being created.

# systemctl status systemd-tmpfiles-setup.service
systemd-tmpfiles-setup.service - Create Volatile Files and Directories
  Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static)
  Active: failed (Result: exit-code) since Tue 2014-09-30 09:40:30 EDT; 3h 24min ago
Docs: man:tmpfiles.d(5)
      man:systemd-tmpfiles(8)
  Process: 724 ExecStart=/usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev (code=exited, status=1/FAILURE)
Main PID: 724 (code=exited, status=1/FAILURE)

Sep 30 09:40:30 servername systemd-tmpfiles[724]: [/usr/lib/tmpfiles.d/httpd.conf:1] Unknown group 'apache'.
Sep 30 09:40:30 servername systemd-tmpfiles[724]: [/usr/lib/tmpfiles.d/httpd.conf:2] Unknown user 'apache'.
Sep 30 09:40:30 servername systemd[1]: systemd-tmpfiles-setup.service: main process exited, code=exited, status=1/FAILURE
Sep 30 09:40:30 servername systemd[1]: Failed to start Create Volatile Files and Directories.
Sep 30 09:40:30 servername systemd[1]: Unit systemd-tmpfiles-setup.service entered failed state.

That's because I've a NIS server configured, whith a NIS apache user. As it has an apache user, Apache installation doesn't create apache group. But apache group also exists on NIS! Well, NIS is messing up things.

Bottom line is: I have to stop ypbind, install Apache and then restart ypbind (or just create manually a apache group in /etc/group).

score 2 · Answer 2 · edited Apr 13 '17 at 12:14

I had this same problem but with a slightly different solution.

Using @joaoolavo's solution, I tried systemctl status systemd-tmpfiles-setup.service:

[root@server ~]# systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)

Note that Active: inactive (dead).

Restarting systemd-tmpfiles-setup created the files I needed in /run/ , and the status changed to Active: active (exited), although, obviously, httpd (and in my case, postgresql) weren't loaded:

[root@server ~]# systemctl start systemd-tmpfiles-setup.service
[root@server ~]# systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled)
   Active: active (exited) since Fri 2016-03-18 13:35:36 AEDT; 8s ago
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
  Process: 2551 ExecStart=/usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev (code=exited, status=0/SUCCESS)
 Main PID: 2551 (code=exited, status=0/SUCCESS)

Mar 18 13:35:36 server.org systemd[1]: Starting Create Volatile Files and Directories...
Mar 18 13:35:36 server.org systemd[1]: Started Create Volatile Files and Directories.

Would it survive a reboot?

Yes it does. In fact, reboots now come up as we would expect - all tmpfiles created, httpd and postgresql are also started.

It would seem that systemd-tmpfiles-setup.service needs to be in Active state active (exited) rather than inactive (dead) to work properly after boot.

score 1 · Answer 3 · edited Apr 13 '17 at 12:14

I had the same solution for my problem as @datakid with the difference that after a reboot the systemd-tmpfiles-setup.service was dead again.

For my solution you first need to know that I mounted my /var directory on a different disk. And there was the problem. My /etc/fstab for the /var looked like this:

/dev/xvdb1 /var ext4 defaults,noatime,_netdev,nofail 0 2

So the problem was the _netdev. Because this might be useful for a NFS, where you need a network but not for my case with the /vardirectory

Here's the explanation for _netdev:

The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).

After i removed the _netdev everything was working again, also after reboot

/run on RHEL systems should be a tmpfs therefore "fixes" which reference /var are a red-herring: — DaveG, Sep 15 '22 at 10:13

Apache's PidFile directory is removed every boot

3 Answers3

Linked