Set nginx fallback when proxy fails because docker container is not running

Question

I am trying to get nginx to proxy a subdomain to another docker container or return an error if the container is not available.

My issue is that nginx seems to check if the server is available and refuses to start if it is not. I tried using error_page like described here but it didn't change anything about the issue.

The following configuration works if both containers are up and running:

server {
    server_name localhost;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        return 501;
    }
}

server {
    server_name *.ref.localhost ref.localhost;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_intercept_errors  on;

        proxy_pass http://reference:5000;
    }
}

When I kill the reference container after nginx is running, I get a 502 error, but when I try to start the nginx container without the reference container running, nginx crashes with the following error message:

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Recreating nginx ... done
Attaching to nginx
nginx        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx        | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx        | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx        | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx        | 2021/05/15 12:55:51 [emerg] 1#1: host not found in upstream "reference" in /etc/nginx/conf.d/default.conf:25
nginx        | nginx: [emerg] host not found in upstream "reference" in /etc/nginx/conf.d/default.conf:25
nginx        | 2021/05/15 12:55:53 [emerg] 1#1: host not found in upstream "reference" in /etc/nginx/conf.d/default.conf:25
nginx        | nginx: [emerg] host not found in upstream "reference" in /etc/nginx/conf.d/default.conf:25
nginx exited with code 1

Both nginx and reference are running with docker-compose:

version: "3.8"
services:
  nginx:
    container_name: nginx
    restart: always
    build:
      context: ./nginx
    ports:
      - "80:80"
  reference:
    container_name: reference
    build:
      context: ./app

nginx/Dockerfile

# pull the Node.js Docker image
FROM node:alpine

# create the directory inside the container
WORKDIR /usr/src/app

# copy the package.json files from local machine to the workdir in container
COPY package*.json ./

# run npm install in our local machine
RUN npm install

# copy the generated modules and all other files to the container
COPY . .

# our app is running on port 5000 within the container, so need to expose it
EXPOSE 5000

# the command that starts our app
CMD ["node", "index.js"]

reference/Dockerfile:

# pull the Node.js Docker image
FROM node:alpine

# create the directory inside the container
WORKDIR /usr/src/app

# copy the package.json files from local machine to the workdir in container
COPY package*.json ./

# run npm install in our local machine
RUN npm install

# copy the generated modules and all other files to the container
COPY . .

# our app is running on port 5000 within the container, so need to expose it
EXPOSE 5000

# the command that starts our app
CMD ["node", "index.js"]