Bad Gateway 502 Connection Refused Nginx, Docker Compose and Keycloak

Question

I'm having trouble configuring Nginx and Keycloak together with docker-compose. I keep receiving 502 Bad Gateway error when trying to access the Keycloak dashboard behind Nginx reverse proxy.

Here is my docker-compose.yaml file

# Docker Compose file Reference (https://docs.docker.com/compose/compose-file/)

version: '3.8'

services:

  nginx:
    image: my-nginx-image
    ports:
      - "80:80"
    depends_on:
      - db-keycloak
      - keycloak
    restart: 
      always
    networks: # join the backend and frontend network
      - backend
      - frontend

  # Keycloak Service (Auth Server)
  keycloak:
    image: jboss/keycloak:15.0.0
    restart: always
    depends_on:
      - db-keycloak
    environment:
      DB_VENDOR: postgres
      DB_ADDR: db-keycloak
      DB_DATABASE: keycloak
      DB_USER: ${KEYCLOAK_DB_USER}
      DB_PASSWORD: ${KEYCLOAK_DB_PASSWORD}
      KEYCLOAK_USER: ${KEYCLOAK_ADMIN_USER}
      KEYCLOAK_PASSWORD: ${KEYCLOAK_ADMIN_PASSWORD}
      PROXY_ADDRESS_FORWARDING: "true"
    command: ["-Djboss.http.port=8100"]
    ports:
      - 8100:8100
    networks: # join the backend and frontend network
      - backend
      - frontend # commenting out this line somehow resolves my issue
    
  # Keycloak Database Service
  db-keycloak:
    image: postgres:latest
    ports:
      - "5432:5432"
    restart: always
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: ${KEYCLOAK_DB_USER}
      POSTGRES_PASSWORD: ${KEYCLOAK_DB_PASSWORD}
    networks:
      - backend # join the backend network only
    volumes:
      - db-keycloak-data:/var/lib/postgres # persist keycloak db data
    
  
# Volumes
volumes:
  db-keycloak-data:

# Networks for the backend and frontend
networks:
  backend:
  frontend:

I'm using a custom Nginx image built from the following Dockerfile:

FROM nginx

expose 80

COPY ./default.conf /etc/nginx/conf.d/default.conf

CMD ["nginx", "-g", "daemon off;"]

The default.conf file is:

upstream keycloak {
    server keycloak:8100;
}

server {
    listen 80;

    # keycloak
    location /auth {
        proxy_pass http://keycloak/auth;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
    }

    location /auth/admin {
        proxy_pass http://keycloak/auth/admin;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
    }
 
}

I've disabled SSL in Keycloak, and I'm running keycloak on port 8100 because another container is using 8080 (I've excluded some of the irrelevant config for my other images, just note that I have other services running on the backend and frontend networks). The problem I'm having is that when I try to access the Keycloak dashboard at /auth I am greeted with a 502 Bad Gateway page. However, if I remove the keycloak service from the frontend network, I can access the dashboard just fine (like the following):

networks: # only join the backend network
   - backend

This is the output from the Nginx when I try to navigate to the page:

2022/01/07 17:27:52 [error] 31#31: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 94.10.13.254, server: , request: "GET /auth HTTP/1.1", upstream: "http://172.18.0.4:8100/auth", host: "my-ec2-instance.eu-west-2.compute.amazonaws.com"

Running docker container inspect on the Keycloak container I can see that the IP 172.18.0.4 does match so it seems to be forwarding the request to the correct container address, and Nginx and Keycloak are both on the same network. Could this be an issue with my docker compose file configuration or maybe with Keycloak refusing the connection for another reason? Is there something I'm missing. Let me know if there is any other info I should include.