Not able to connect between Celery and Minio containers?

Question

In my Django app, I'm offloading resizing of uploaded videos to a celery worker. I'm also storing all media files in Minio S3 buckets.

The resizing goes well without any errors, however Celery can't connect to the Minio container. This is the logs from Celery:

celery-worker_1  | [2023-07-19 20:51:46,076: INFO/MainProcess] Task common.tasks.resize_video[c38a90eb-793e-4190-a681-94b69eafadc8] received
celery-worker_1  | ffmpeg version 4.3.6-0+deb11u1 Copyright (c) 2000-2023 the FFmpeg developers
celery-worker_1  |   built with gcc 10 (Debian 10.2.1-6)
celery-worker_1  | ffmpeg version 4.3.6-0+deb11u1[tcp @ 0x55c7500727c0]  Copyright (c) 2000-2023 the FFmpeg developers Connection to tcp://localhost:9000 failed: Cannot assign requested address
celery-worker_1  | http://localhost:9000/event_media/test_video.mp4: Cannot assign requested address

Question is: how do I make the Celery container to connect to http://minio:9000/ rather than to localhost?

This is the docker-compose.yml file I'm using:

  postgres:
    image: "postgres:14"
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
    volumes:
      - .:/opt/myapp
    ports:
      - "5432:5432"

  redis:
    image: redis:6
    ports:
      - "6379:6379"

  minio:
    image: minio/minio
    command: minio server /export
    ports:
      - "9000:9000"
    environment:
      - MINIO_ACCESS_KEY=dev_access
      - MINIO_SECRET_KEY=dev_secret

  celery-worker:
    build:
      context: .
      dockerfile: ./Dockerfile.celery
    env_file:
      - ./docker.dev.env
    environment:
      - CELERY_BROKER_URL=redis://redis:6379
      - CELERY_RESULT_BACKEND=redis://redis:6379
    depends_on:
      - redis
      - postgres
    command: celery -A myapp worker --loglevel INFO

This is contents of the docker.dev.env file:

MINIO_STORAGE_ENDPOINT=minio:9000
MINIO_STORAGE_ACCESS_KEY=dev_access
MINIO_STORAGE_SECRET_KEY=dev_secret
MINIO_STORAGE_USE_HTTPS=False
MINIO_STORAGE_MEDIA_BUCKET_NAME=media-dev
MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET=True
MINIO_STORAGE_MEDIA_URL=http://localhost:9000/
DEFAULT_FILE_STORAGE=minio_storage.storage.MinioMediaStorage

Edit: Tried to change

MINIO_STORAGE_MEDIA_URL=http://minio:9000/

Which instead gave me this following error:

celery-worker_1  | [http @ 0x55698dd429c0] HTTP error 403 Forbidden
celery-worker_1  | http://minio:9000/event_media/test_video.mp4: Server returned 403 Forbidden (access denied)

Still scratching my head

score 0 · Accepted Answer · answered Jul 21 '23 at 10:59

After pouring over the documentation at https://django-minio-storage.readthedocs.io/en/latest/usage/#django-settings-configuration, and specifically the setting key MINIO_STORAGE_MEDIA_URL, I updated docker.dev.env:

From

MINIO_STORAGE_MEDIA_URL=http://localhost:9000/

To

MINIO_STORAGE_MEDIA_URL=http://minio:9000/media-dev

What I learned was:

that localhost points on its own container, thus need to use the container name minio instead (as defined in docker-compose.yml).
The MINIO_STORAGE_MEDIA_URL should also contain the name of the minio bucket, in this case "media-dev" as defined by MINIO_STORAGE_MEDIA_BUCKET_NAME.

Not able to connect between Celery and Minio containers?

1 Answers1