Multiple docker compose files and sharing volumes

Question

I am trying to setup some docker compose files with one compose file per project. Each project has its own folder with its own docker compose file. But then I want a convenient way to start up all of these docker compose files (possibly with dependencies between them?). I have found a nice way to do this using the extends keyword (see https://docs.docker.com/compose/extends/). But I am facing a minor issue with sharing volumes.

This question is similar to Share volumes between separate docker compose files, but I think that question and answer were possibly written before name could be include in volumes defined in the top level volumes section, so I think answers may be different now. The questions are also a little bit different. This one focusing on starting multiple compose files from a top level compose file.

I am using named volumes so that the volumes are managed by docker. Also, I don't know if the top level or lower level compose files will be called first so I would like both files to be able to create the volume if it does not yet exist. This means I don't want to set external: true for the volumes in either file.

My code is roughly like this. The directory structure is:

Servers
|-- docker-compose.yaml
|-- project_1
|   |-- p1-docker-compose.yaml
|-- project_2
|   |-- p2-docker-compose.yaml

and the docker compose file contents are

# Servers/project_1/p1-docker-compose.yaml
version: '3.8'

services:
  service_1:
    image: image_1
    volumes: service_1_data

volumes:
  service_1_data:

# Servers/project_2/p2-docker-compose.yaml
version: '3.8'

services:
  service_2:
    image: image_2
    volumes: service_2_data

volumes:
  service_2_data:

# Servers/docker-compose.yaml
version: '3.8'

services:
  service_1:
    extends:
      file: project_1/p1-docker-compose.yaml
      service: service_1
  service_2:
    extends:
      file: project_2/p2-docker-compose.yaml
      service: service_2

volumes:
  service_1_data:
    name: project_1_service_1_data
  service_2_data:
    name: project_2_service_2_data

In the top-level docker compose file the project prefixes are needed because these prefixes get added if the volume is created using the lower level docker compose files. Alternatively, I could explicitly specify names without the prefixes in all of the lower and upper level docker compose files and this would have a similar result.

This all actually works. I can start the containers using either the lower or upper level docker compose files and in all cases the same volumes are used.

But there is one minor problem. Suppose I first run docker compose up -d in the Servers folder, using the top level docker compose to spin up the containers. This will also create the volumes named project_1_service_1_data and project_2_service_2_data. However, if I now bring the containers down and then go into project_1 and do docker compose up -d the container will spin up fine and use the right volume but I get the following warning:

WARN[0000] volume "project_1_service_1_data" already exists but was not created for project "servers". Use `external: true` to use an existing volume

This message is confusing because the volume was in fact created for project "servers". It was not created for project "project_1". Indeed if I do docker volume inspect project_1_service_1_data I see

"com.docker.compose.project": "servers"

Essentially the questions are:

Is there any way to configure these compose files to avoid getting the warning shown above?
Is there generally a better way using docker compose to accomplish what I am trying to do here?

Multiple docker compose files and sharing volumes

0 Answers0