Restrict location to just one IP in vhosts not working

Question

I am trying to restrict the access to a specific location for just one IP address in the vhosts file but for some reason it is not working. This for apache 2.4.29.

Here is my vhosts file:

<IfModule mod_ssl.c>
        <VirtualHost *:80>
                ServerName subdomain.example.com
                Redirect / https://subdomain.example.com/
        </VirtualHost>

        <VirtualHost *:80>
                ServerName subdomain.example.com
                ServerAlias example.com
                <Location />
                        <RequireAll>
                                Require all granted
                        </RequireAll>

                        AllowMethods DELETE GET POST PUT HEAD
                </Location>

                <Location /admin>
                        Require ip XX.XX.XX.XX
                </Location>

        </VirtualHost>

        <VirtualHost *:443>
                ServerName subdomain.example.com
                ServerAlias example.com
                <Location />
                        <RequireAll>
                                Require all granted
                        </RequireAll>
                        AllowMethods DELETE GET POST PUT HEAD
                </Location>

                <Location /admin>
                        Require ip XX.XX.XX.XX
                </Location>
        </VirtualHost>
</IfModule>

And this is my apache.conf

<Directory />
        Order Deny,Allow
        Deny from all
        Options None
        AllowOverride None
        Require all denied
</Directory>

Feel free to suggest alternatives - I thought this could also be done via mod_security

.htaccess is not an option in the case.

Thank you!