Apache2.4 returns 403/Forbidden error on symlinks after Ubuntu upgrade

Question

I just upgraded my development workstation from Kubuntu 18.04 to Kubuntu 22.04.2 LTS.

For context, I keep /home on a separate disk from /. The previous / is still present at /mnt/sdc5.

Kubuntu 18.04 was running:

$ /mnt/sdc5/usr/sbin/apache2 -v
Server version: Apache/2.4.56 (Ubuntu)
Server built:   2023-03-09T07:33:5

Whereas 22.04 is running:

$ apache2 -v
Server version: Apache/2.4.52 (Ubuntu)
Server built:   2023-03-08T17:32:01

(Yes, as of this past weekend, Bionic apparently had a newer apache2 than Jammy).

I have copied all the apache/ssl configs from /mnt/sdc5, as well as all the web roots (I have a several vhosts set up) from /mnt/sdc5/var/www/ to their functional location in /var/www/. As in the previous environment, symlinks under /var/www/*/ cross physical disks from sda5 to my ~ in sdb5 (/home/dracos/).

I keep pretty much everything under /var/www/* owned by dracos:www-data, and I have put myself in the www-data group:

$ groups dracos
dracos : dracos adm cdrom sudo dip www-data plugdev lpadmin lxd sambashare

Apache runs fine, except browsing any symlink returns a 403 error (ie, http://ref.harad/knotwork):

[Wed Mar 29 17:54:02.090919 2023] [core:error] [pid 439254] [client 192.168.0.21:51496] AH00037: Symbolic link not allowed or link target not accessible: /var/www/ref/htdocs/knotwork

In this case, htdocs is a directory, while knotwork is the symlink:

$ ll /var/www/ref/htdocs/|grep knotwork lrwxrwxrwx 1 dracos www-data 30 Jun 18 2021 knotwork -> /home/dracos/graphics/knotwork

And the full path to it:

$ stat /var/
  File: /var/
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 805h/2053d      Inode: 2883585     Links: 15
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-29 11:03:25.026834422 -0600
Modify: 2023-03-27 22:03:04.511844496 -0600
Change: 2023-03-27 22:03:04.511844496 -0600
 Birth: 2023-03-25 22:09:41.157329268 -0600

$ stat /var/www/
  File: /var/www/
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 805h/2053d      Inode: 3277054     Links: 24
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-29 15:47:21.907537823 -0600
Modify: 2023-03-29 15:46:51.232613614 -0600
Change: 2023-03-29 15:46:51.232613614 -0600
 Birth: 2023-03-27 22:03:04.511844496 -0600

$ stat /var/www/ref
  File: /var/www/ref
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 805h/2053d      Inode: 3285520     Links: 6
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-29 15:47:22.487517480 -0600
Modify: 2021-05-01 11:12:49.857126723 -0600
Change: 2023-03-29 15:47:09.575970401 -0600
 Birth: 2023-03-29 15:46:51.232613614 -0600

$ stat /var/www/ref/htdocs
  File: /var/www/ref/htdocs
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 805h/2053d      Inode: 3287655     Links: 41
Access: (0775/drwxrwxr-x)  Uid: ( 1000/  dracos)   Gid: (   33/www-data)
Access: 2023-03-29 17:54:02.380602649 -0600
Modify: 2023-01-13 15:20:42.794065309 -0700
Change: 2023-03-29 17:53:56.380789344 -0600
 Birth: 2023-03-29 15:46:51.724596365 -0600

$ stat /var/www/ref/htdocs/knotwork
  File: /var/www/ref/htdocs/knotwork -> /home/dracos/graphics/knotwork
  Size: 30              Blocks: 0          IO Block: 4096   symbolic link
Device: 805h/2053d      Inode: 3287656     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/  dracos)   Gid: (   33/www-data)
Access: 2023-03-29 17:54:02.084611865 -0600
Modify: 2021-06-18 16:02:41.686238932 -0600
Change: 2023-03-29 17:53:56.308791583 -0600
 Birth: 2023-03-29 15:46:51.724596365 -0600

In apache2.conf:

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Require all denied
</Directory>

<Directory /usr/share>
    AllowOverride None
    Require all granted
</Directory>

<Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

And the full config for this vhost (001-ref.conf):

<VirtualHost ref.harad:80>
    ServerName ref.harad
    ServerAdmin webmaster@localhost

    DocumentRoot /var/www/ref/htdocs
    <Directory />
        Options FollowSymLinks Includes
        AllowOverride None
    </Directory>
    <Directory /var/www/ref/htdocs/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        #Order allow,deny
        #allow from all
        Require all granted
    </Directory>
    <Directory /var/www/ref/htdocs/_flyspray/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
        #Order allow,deny
        #allow from all
        Require all granted
    </Directory>

    ErrorLog /var/log/apache2/error_ref.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access_ref.log combined
</VirtualHost>

In this file I commented out the Order and allow lines in favor of the Require line below, which made no difference.

There are no configuration errors:

$ apachectl configtest
Syntax OK

Apache is running:

$ systemctl status apache2.service
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-03-29 17:44:29 MDT; 49min ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 439245 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
   Main PID: 439250 (/usr/sbin/apach)
      Tasks: 7 (limit: 38327)
     Memory: 42.0M
        CPU: 917ms
     CGroup: /system.slice/apache2.service
             ├─439250 /usr/sbin/apache2 -k start
             ├─439251 /usr/sbin/apache2 -k start
             ├─439252 /usr/sbin/apache2 -k start
             ├─439253 /usr/sbin/apache2 -k start
             ├─439254 /usr/sbin/apache2 -k start
             ├─439255 /usr/sbin/apache2 -k start
             └─439896 /usr/sbin/apache2 -k start

Mar 29 17:44:29 harad systemd[1]: Starting The Apache HTTP Server...
Mar 29 17:44:29 harad systemd[1]: Started The Apache HTTP Server.

Permissions leading to knotwork in /home:

$ stat /home
  File: /home
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 812h/2066d      Inode: 2           Links: 8
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-29 10:33:09.874123424 -0600
Modify: 2023-03-25 22:13:38.754784783 -0600
Change: 2023-03-25 22:13:38.754784783 -0600
 Birth: 2019-03-03 08:52:01.000000000 -0700

$ stat /home/dracos
  File: /home/dracos
  Size: 24576           Blocks: 48         IO Block: 4096   directory
Device: 812h/2066d      Inode: 103153665   Links: 116
Access: (0750/drwxr-x---)  Uid: ( 1000/  dracos)   Gid: ( 1000/  dracos)
Access: 2023-03-29 11:03:19.023044685 -0600
Modify: 2023-03-29 11:03:17.423100718 -0600
Change: 2023-03-29 11:03:17.423100718 -0600
 Birth: 2023-03-25 22:13:38.750781079 -0600

$ stat /home/dracos/graphics
  File: /home/dracos/graphics
  Size: 36864           Blocks: 72         IO Block: 4096   directory
Device: 812h/2066d      Inode: 94372596    Links: 16
Access: (0755/drwxr-xr-x)  Uid: ( 1000/  dracos)   Gid: ( 1000/  dracos)
Access: 2023-03-29 15:48:58.536144219 -0600
Modify: 2023-02-25 15:55:31.878748109 -0700
Change: 2023-03-26 02:15:19.817837898 -0600
 Birth: 2019-03-03 17:58:31.009967662 -0700
harad /etc/apache2

$ stat /home/dracos/graphics/knotwork
  File: /home/dracos/graphics/knotwork
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 812h/2066d      Inode: 124257516   Links: 2
Access: (0775/drwxrwxr-x)  Uid: ( 1000/  dracos)   Gid: ( 1000/  dracos)
Access: 2023-03-29 10:32:38.795209595 -0600
Modify: 2022-04-09 19:03:00.488673788 -0600
Change: 2023-03-10 20:49:52.217436682 -0700
 Birth: 2021-01-17 12:55:57.612873620 -0700

I've spent over 3 hours trying to figure this out. The config is clean. Permissions are the same as before. I don't understand why I can't browse symlinks. Once I solve this, all my other vhosts (which all have symlinks) will be functional again.

I've been able to solve this myself several times in the past. What obscure detail have I overlooked after reading through two dozen threads looking for the answer?

score 0 · Accepted Answer · answered Mar 30 '23 at 19:00

SOLVED:

The solution was right there in the question:

$ stat /home/dracos
  File: /home/dracos
  Size: 24576           Blocks: 48         IO Block: 4096   directory
Device: 812h/2066d      Inode: 103153665   Links: 116
Access: (0750/drwxr-x---)  Uid: ( 1000/  dracos)   Gid: ( 1000/  dracos)
Access: 2023-03-29 11:03:19.023044685 -0600
Modify: 2023-03-29 11:03:17.423100718 -0600
Change: 2023-03-29 11:03:17.423100718 -0600
 Birth: 2023-03-25 22:13:38.750781079 -0600

/home/dracos was set to permissions 750. The fix was:

$ chmod o+rx /home/dracos

Which results in:

$ stat /home/dracos
  File: /home/dracos
  Size: 24576           Blocks: 48         IO Block: 4096   directory
Device: 812h/2066d      Inode: 103153665   Links: 116
Access: (0755/drwxr-xr-x)  Uid: ( 1000/  dracos)   Gid: ( 1000/  dracos)
Access: 2023-03-30 11:24:15.992545084 -0600
Modify: 2023-03-30 11:24:12.112622510 -0600
Change: 2023-03-30 12:47:43.904426309 -0600
 Birth: 2023-03-25 22:13:38.750781079 -0600

I finally noticed this by closely examining /home/:

$ ll /home/
total 96K
drwxr-xr-x   8 root   root   4.0K Mar 25 22:13 .
drwxr-xr-x  20 root   root   4.0K Mar 25 22:11 ..
drwx------   4 dracos dracos 4.0K Mar 27 12:54 bak_18.4
drwxr-x--- 116 dracos dracos  24K Mar 30 11:24 dracos
drwxr-xr-x  77 dracos dracos  32K Mar 27 18:02 dracos_18.04
drwxr-xr-x   7   1001   1001 4.0K Mar  4  2019 git
drwx------   2 root   root    16K Mar  3  2019 lost+found
drwxr-xr-x  25 root   root   4.0K Mar 10 20:12 root_18.4

Where /home/dracos_18.04 is my ~ from the previous install which I renamed prior to running the 22.04 installer in the live boot disk.

Apache2.4 returns 403/Forbidden error on symlinks after Ubuntu upgrade

1 Answers1