nginx radicale reverse proxy: 403 Error

Question

Made some attempts to configure a reverse proxy for radicale. Site is up and running meaning I can connect to the webui but im getting a 403 error when I attempt to connect it to my client (Calcurse-caldav).

Here is the error in question error: The server at cal.xxx.re replied with HTTP status code 403 (Forbidden) error: while trying to access https://cal.xxx.re/var/lib/radicale/collections/.

Here are the configs for the setup

1- /etc/radicale/config

[server]
hosts = 0.0.0.0:5232, [::]:5232

[auth]
type = htpasswd
htpasswd_filename = /etc/radicale/users
htpasswd_encryption = bcrypt

2- /etc/nginx/site-enabled/cal

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name cal.xxx.re;
    location / {
        proxy_pass http://localhost:5232/; # The / is important!
        proxy_set_header X-Script-Name /;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Remote-User $remote_user;
        proxy_set_header Host $http_host;
        proxy_pass_header Authorization;

    }

    ssl_certificate /etc/letsencrypt/live/cal.afu.re/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/cal.afu.re/privkey.pem; # managed by Certbot
}

3- config/calcurse/caldav/config

[General]
Binary = /usr/bin/calcurse
Hostname = cal.xxx.re
Path = /var/lib/radicale/collections/
SyncFilter = cal,todo

I made sure bcrypt was installed. What should I investigate to solve this 403 error ? Any suggestion will be greatly appreciated !