-1

I have an existing Docker Compose project that is currently used to orchestrate several services. However, I now want to integrate an additional service, in this case, Mayan EDMS, which has its own Docker Compose file.

---
version: '3'

x-airflow-common:
  &airflow-common
  # In order to add custom dependencies or upgrade provider packages you can use your extended image.
  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.3.3}
  # build: .
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    # For backward compatibility, with Airflow <2.3
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:${POSTGRES_PASSWORD}@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
    ENVIRONMENT: '${ENVIRONMENT}'
    TENANT_ID: '${TENANT_ID}'
    APPLICATION_ID: '${APPLICATION_ID}'
    SECRET_VALUE: '${SECRET_VALUE}'
    # Modificación 
    # Pip packages
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- azure-storage-file-datalake openpyxl azure.keyvault azure.identity python-dotenv}
  volumes:
    - ./airflow_dags:/opt/airflow/dags
    - ./airflow_logs:/opt/airflow/logs
    - ./airflow_plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    &airflow-common-depends-on
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy

services:
  postgres:
    container_name: postgresql_airflow
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: airflow
    expose:
      - "5432"
    ports:
      - "5433:5432"
    volumes:
      - airflow-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    restart: always

  redis:
    container_name: redis_airflow
    image: redis:latest
    expose:
      - 6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    container_name: airflow_webserver
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-scheduler:
    container_name: airflow_scheduler
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-worker:
    container_name: airflow_worker
    <<: *airflow-common
    command: celery worker
    healthcheck:
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 10s
      timeout: 10s
      retries: 5
    environment:
      <<: *airflow-common-env
      # Required to handle warm shutdown of the celery workers properly
      # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
      DUMB_INIT_SETSID: "0"
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-triggerer:
    container_name: airflow_triggerer
    <<: *airflow-common
    command: triggerer
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-init:
    container_name: airflow_init
    <<: *airflow-common
    entrypoint: /bin/bash
    # yamllint disable rule:line-length
    command:
      - -c
      - |
        function ver() {
          printf "%04d%04d%04d%04d" $${1//./ }
        }
        airflow_version=$$(AIRFLOW__LOGGING__LOGGING_LEVEL=INFO && gosu airflow airflow version)
        airflow_version_comparable=$$(ver $${airflow_version})
        min_airflow_version=2.2.0
        min_airflow_version_comparable=$$(ver $${min_airflow_version})
        if (( airflow_version_comparable < min_airflow_version_comparable )); then
          echo
          echo -e "\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m"
          echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
          echo
          exit 1
        fi
        if [[ -z "${AIRFLOW_UID}" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
          echo "If you are on Linux, you SHOULD follow the instructions below to set "
          echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
          echo "For other operating systems you can get rid of the warning with manually created .env file:"
          echo "    See: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user"
          echo
        fi
        one_meg=1048576
        mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
        cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
        disk_available=$$(df / | tail -1 | awk '{print $$4}')
        warning_resources="false"
        if (( mem_available < 4000 )) ; then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
          echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
          echo
          warning_resources="true"
        fi
        if (( cpus_available < 2 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
          echo "At least 2 CPUs recommended. You have $${cpus_available}"
          echo
          warning_resources="true"
        fi
        if (( disk_available < one_meg * 10 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
          echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
          echo
          warning_resources="true"
        fi
        if [[ $${warning_resources} == "true" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
          echo "Please follow the instructions to increase amount of resources available:"
          echo "   https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin"
          echo
        fi
        mkdir -p /sources/logs /sources/dags /sources/plugins
        chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
        exec /entrypoint airflow version
    # yamllint enable rule:line-length
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-belero-admin}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-${AIRFLOW_PASSWORD}}
      _PIP_ADDITIONAL_REQUIREMENTS: ''
    user: "0:0"
    volumes:
      - .:/sources

  airflow-cli:
    container_name: airflow_cli
    <<: *airflow-common
    profiles:
      - debug
    environment:
      <<: *airflow-common-env
      CONNECTION_CHECK_MAX_COUNT: "0"
    # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
    command:
      - bash
      - -c
      - airflow

  # You can enable flower by adding "--profile flower" option e.g. docker-compose --profile flower up
  # or by explicitly targeted on the command line e.g. docker-compose up flower.
  # See: https://docs.docker.com/compose/profiles/
  flower:
    container_name: airflow_flower
    <<: *airflow-common
    command: celery flower
    profiles:
      - flower
    ports:
      - 5555:5555
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  djangoappdev:
    container_name: django_app_dev
    build: .
    environment:
      ENVIRONMENT: '${ENVIRONMENT}'
      TENANT_ID: '${TENANT_ID}'
      APPLICATION_ID: '${APPLICATION_ID}'
      SECRET_VALUE: '${SECRET_VALUE}'
    ports:
      # Where the django app runs 
      - "8001:8001"
    volumes:
      - ./beleroforms:/code/beleroforms
      - ./beleroforms_app:/code/beleroforms_app

  djangodb:
    container_name: django_app_db_dev
    # image: postgres:latest
    image: postgres:14.7
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_USER=django"
      - "POSTGRES_PASSWORD=${POSTGRES_PASSWORD_DJANGO}"
      - "POSTGRES_DB=django"
    expose:
      - "5432"
    ports:
      - "5434:5432" 

volumes:
  airflow-db-volume:
  airflow_dags:
  airflow_logs:
  airflow_plugins:
  postgres_data:
  # beleroforms:
  # beleroforms_app:

What is the best way to add Mayan's Docker Compose file to my existing project, how can I ensure that both files run consistently and communicate with each other if necessary?

I would very much appreciate any guidance or practical examples on how to achieve this integration without causing conflicts between existing and new services.

Thanks in advance for your help!

opeonikute
  • 494
  • 1
  • 4
  • 15
Julian Buxton
  • 31
  • 5

1 Answers1

0

You can use the -f flag to combine multiple compose files

To use multiple override files, or an override file with a different name, you can use the -f option to specify the list of files. Compose merges files in the order they're specified on the command line. See the docker compose command reference for more information about using -f.

A practical example for your use-case:

docker-compose -f compose.yml -f mayan-compose.yml up -d

The documentation however lists a bunch of rules for merging, which you should be aware of. I recommend reading the full list but some key points:

  1. For single-value options like image, command or mem_limit, the new value replaces the old value.

  2. For the multi-value options ports, expose, external_links, dns, dns_search, and tmpfs, Compose concatenates both sets of values.

  3. In the case of environment, labels, volumes, and devices, Compose "merges" entries together with locally defined values taking precedence. For environment and labels, the environment variable or label name determines which value is used.

  4. Entries for volumes and devices are merged using the mount path in the container.

opeonikute
  • 494
  • 1
  • 4
  • 15
  • but I don't want to have two docker compose documents... that's the thing... – Julian Buxton Sep 01 '23 at 08:23
  • Is there a particular reason for that? It sounds like the other compose file is not within your control, and it would be wise to get updates when it is changed etc. Other than that, you'll have to copy the entire thing and place it in your own compose file. – opeonikute Sep 01 '23 at 08:26
  • I do it for the simple fact that I am new to docker, and I don't want to have problems working with docker compose. – Julian Buxton Sep 01 '23 at 08:30
  • Well when you're new to a piece of software, my advice is to learn its limitations and how to work with it effectively from the docs. You will only get problems by trying to circumvent its design and design decisions. – opeonikute Sep 01 '23 at 08:50