Why doesn't my docker-compose volume get updated with local file additions?

Question

Here's my docker-compose.yml, it contains a web service, which mounts a folder locally so that upon any code change watchmedo restarts the webapp django server. It's a reverse proxy with nginx.

version: '3'
services:
  webapp:
    build:
      context: .
      dockerfile: ./web/Dockerfile
    volumes:
      - ./web/:/usr/src/app/
      - staticfile_volume:/usr/src/app/public
    entrypoint: >
      watchmedo auto-restart --recursive 
        --pattern="*.py" --directory="." 
          gunicorn myapp.wsgi:application -- --bind 0.0.0.0:9000
  nginx:
    build:
      context: .
      dockerfile: ./nginx/Dockerfile
    depends_on:
      - webapp
    volumes:
      - staticfile_volume:/usr/src/app/public
volumes:
  staticfile_volume:

My local file setup is like:

$ tree -L 2
.
├── docker-compose.yml
├── nginx
│   ├── Dockerfile
│   └── nginx.conf
└── web
   ├── Dockerfile
   ├── manage.py
   ├── myapp
   ├── public
   └── static

But when I create a new file in web/public (the same folder mounted as a shared volume between webapp and nginx services), I don't see it from inside the running webapp container.

Yet, if I create a new file anywhere else in the web/ folder (which is also mounted as a separate volume), I see the change from inside the running webapp container.

What is causing this? And how do I change this behavior?

(I need to be able to run python manage.py collectstatic from inside the running container but output into my local hard drive's web/public so I can build the production Docker images for deploy.)

The question is a little confusing, and like Chris's answer, I could also see it working - just looking at the last paragraph of the question, keep in mind that your `public` folder is no longer synced with your host, since you mounted it to a volume. I believe what you are looking for is just one simple folder mount (entire app in the webapp container, and only public in the nginx container) — DannyB, Jun 02 '19 at 09:59
Ah yes - if you're creating the file on your host inside web/public, it won't appear in either of the container volumes. This is the most likely scenario, good spotting. Feel free to post an answer below :-) — Chris McKinnel, Jun 02 '19 at 10:09

Chris McKinnel · Accepted Answer · 2019-06-02T10:30:44.757

I can't see a problem with what you've got here. I copied your docker-compose.yml and removed only the entrypoint: part of it and it worked for me.

docker-compose.yml

version: '3'
services:
  webapp:
    build:
      context: .
      dockerfile: ./web/Dockerfile
    volumes:
      - ./web/:/usr/src/app/
      - staticfile_volume:/usr/src/app/public
  nginx:
    build:
      context: .
      dockerfile: ./nginx/Dockerfile
    depends_on:
      - webapp
    volumes:
      - staticfile_volume:/usr/src/app/public
volumes:
  staticfile_volume:

web/Dockerfile

FROM ubuntu:18.04

CMD tail -f /dev/null

nginx/Dockerfile

FROM nginx:latest

CMD tail -f /dev/null

Demo:

To elaborate on DannyB's comment above, make sure you're not creating the file on the host in web/public as this folder gets mounted over when the containers start. A docker volume works in a similar way to a standard linux mount - docker will just mount the new volume over the top of the existing directory.

If you want to run a command inside a Docker container that changes files on your host filesystem, don't use a docker volume - instead use a bind mount like you've done here: - ./web/:/usr/src/app/.

The difference between a bind mount and a docker volume is that a bind mount will mount files from your host inside your container, and will rely on those folders / files being on your host filesystem, and a docker volume will be completely managed by docker and can only be shared between containers, not with the host as well (although these files do live on the host somewhere, it's not practical to track them down and attempt to use them).

You can actually just remove your docker volume from your docker-compose file and add a bind mount for nginx and you'll start to see the behaviour you're after:

docker-compose.yml

version: '3'
services:
  webapp:
    build:
      context: .
      dockerfile: ./web/Dockerfile
    volumes:
      - ./web/:/usr/src/app/
  nginx:
    build:
      context: .
      dockerfile: ./nginx/Dockerfile
    depends_on:
      - webapp
    volumes:
      - ./web/public:/usr/src/app/public

Demo:

great point. I was able to get it to work removing the docker volume and having two bind mounts, as suggested. then a separate Dockerfile for nginx on production simply copies the `web/public` into the nginx image when building. this is great! the only con seems to be doubling the space on prod instance for staticfiles (not a big deal). I suppose if I want to do file uploads and serve through nginx (media) i'd need to create a docker volume on the prod instance & mount, which would still be fine. Is my thinking correct here? — lollercoaster, Jun 02 '19 at 18:36
I would strongly consider trying to architect production in such a way that offloaded serving your user uploads to somewhere else. What you're describing here is a stateful instance, which limits you in what you can do - you're already seeing that you need to care about data in your containers and it's a bit of a headache. What happens if the containers get replaced? What do you do if you need to scale and have multiple nginx instances? That being said, if it's not possible to offload user uploads, then having a bind mount on your host might be the best option. — Chris McKinnel, Jun 02 '19 at 22:22
If you do have to use a bind mount, what I would also look at doing is running `collectstatic` on container run, not in the `docker build`. This means your nginx container can remain vanilla and just mount a directory to serve static / media files, and your web container can mount the same directory and use it as an output directory for `collectstatic` and user uploads. You still have the problem of what to do if your production host goes away for any reason, though. If you were using AWS you could offload static + media to S3, and remove the need for nginx altogether. — Chris McKinnel, Jun 02 '19 at 22:24

score 4 · Answer 2 · answered Jun 02 '19 at 10:10

4

You've told Docker that staticfile_volume contains critical application data that must be preserved across runs of the container. The first time the container starts up, and the first time only, Docker will populate it from the image. If you update the image later, since the volume contains critical application data, Docker won't change it.

The easiest short-term workaround is to delete the volume. Try docker-compose down -v; docker-compose up --build. You'll need to do this whenever you change the static content.

It'd be a little easier long-term to configure the back-end application to serve its own files. Django has a django.contrib.staticfiles module to do this. Then your nginx proxy can unconditionally redirect to the backend container, and you don't have to worry about the file sharing issue.

(To see this better, take @ChrisMcKinnel's reproduction recipe and run it once. Then take the web/Dockerfile from there and COPY a file into /usr/src/app/public and re-run docker-compose up --build. You won't see the file appear unless you docker-compose down -v.)

answered Jun 02 '19 at 10:10

David Maze

130,717
29
175
215

ah, I see. so in other words, the docker volume is taking precedence over the bind mount for `/usr/src/app/public`? is this just due to the ordering within the `volumes` list in the docker-compose.yml file itself? – lollercoaster Jun 02 '19 at 18:29
Docker sorts the mounts in order, so any mount on `/usr/src/app/public` will always hide content from any other mount on `/usr/src/app`. – David Maze Jun 02 '19 at 18:29
that's clever. can you expand on what you meant with static files? in my nginx.conf now, I have a context+directive `location /static/ { alias /usr/src/app/public/; }` which, for all files I specify in templates as `{% static 'img/test.png' %}`, should be served directly by nginx with no need for django to do any lifting. Isn't this the "right" way to serve static files? maybe I'm not understanding the reverse proxy correctly. – lollercoaster Jun 02 '19 at 18:40
There's some minor technical advantages to using nginx for this, but the sorts of volume mounts you're showing in this question really don't match what you'd do in a production environment at all, so just using the Django development server for everything (including static files) won't be a big difference. – David Maze Jun 02 '19 at 19:06
oh. what would I do in a production environment ? – lollercoaster Jun 02 '19 at 19:22
In a production environment you'd `docker build` your code into an image and run only the image; you would not build an image and then also separately distribute your code. – David Maze Jun 02 '19 at 19:28

Why doesn't my docker-compose volume get updated with local file additions?

2 Answers2