Docker-compose: bind Celery to Postgres database

Question

My Docker app runs with Flask as backend and Celery as an asyncronous task manager. Task results are then dumped in a SQLalchemy database owned by Postgres.

However, I am not being able to make Celery interact with Postgres.

Configuration:

docker-compose-dev.yml

  web:
    build:
      context: ./services/web
      dockerfile: Dockerfile-dev
    volumes:
      - './services/web:/usr/src/app' 
    ports:
      - 5001:5000
    environment:
      - FLASK_ENV=development
      - APP_SETTINGS=project.config.DevelopmentConfig
      - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev 
      - DATABASE_TEST_URL=postgres://postgres:postgres@web-db:5432/web_test
      - SECRET_KEY=my_precious  
    depends_on:  
      - web-db
      - redis

  web-db:  
    build:
      context: ./services/web/project/db
      dockerfile: Dockerfile
    ports:
      - 5435:5432
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres

  celery:
    image: dev3_web
    restart: always
    volumes:
     - ./services/web:/usr/src/app
     - ./services/web/celery_logs:/usr/src/app/celery_logs

    command: celery worker -A celery_worker.celery --loglevel=DEBUG --logfile=celery_logs/celery.log -Q cache
    environment:
     - CELERY_BROKER=redis://redis:6379/0
     - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
     - web
     - redis
    links:
     - redis:redis
     - web-db

  redis:
    image: redis:5.0.3-alpine
    restart: always
    expose:
      - '6379'
    ports:
      - '6379:6379'

  monitor:
    image: dev3_web
    ports:
      - 5555:5555
    command:  flower -A celery_worker.celery --port=5555 --broker=redis://redis:6379/0
    depends_on:
      - web
      - redis

Log:

celery_1| Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".

How do I bind Celery tasks to my Postgres database?

Hi data_garden. I wonder if we can edit this again? I edited this because the usages of the quote block here are not quotes, and some of the usages of the inline code formatting are not code. These format types have a specific semantic meaning just like HTML, and it is preferable not to misuse them as general highlighters. This is particularly the case if you have a long tail of posts using the same approach. — halfer, Apr 21 '19 at 00:04
(Things like `Dockerfile` and `postgres` are OK in code formatting, as file names and Unix users are frequently rendered to the console. It is also fine to use code formatting for console I/O. However Celery, Docker, Flask, etc are proper nouns, and so should either be rendered with an initial capital (or with special caps rules where appropriate, such as camel-case or all-caps, depending on how each name is normally written)). — halfer, Apr 21 '19 at 00:07
The mis-spelling of "database" has also been re-added to the title. — halfer, Apr 21 '19 at 00:11
Thank you, data_garden! If you could also abide by those conventions, volunteer editors would appreciate it - they may be disinclined to improve the readability of your earlier posts if new ones would be created in the future `:-)`. — halfer, Apr 22 '19 at 12:23

Allen · Accepted Answer · 2019-04-29T13:08:37.060

celery_1   | Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".

suggests that the error from your celery container, which missing the environment valuable setup, you might need DATABASE_URL setup for it which created based on dev3_web.

changes need apply to your docker-compose-dev.yml:

  celery:
    image: dev3_web
    restart: always
    volumes:
     - ./services/web:/usr/src/app
     - ./services/web/celery_logs:/usr/src/app/celery_logs

    command: celery worker -A celery_worker.celery --loglevel=DEBUG --logfile=celery_logs/celery.log -Q cache
    environment:
     - CELERY_BROKER=redis://redis:6379/0
     - CELERY_RESULT_BACKEND=redis://redis:6379/0
     - APP_SETTINGS=project.config.DevelopmentConfig
     - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev 
    depends_on:
     - web
     - redis
    links:
     - redis:redis
     - web-db

add `- DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev ` below your `- CELERY_RESULT_BACKEND=redis://redis:6379/0` ? — Allen, Apr 28 '19 at 21:18
it worked only when I added `DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev ` AND `APP_SETTINGS=project.config.DevelopmentConfig` below, as well. could you please add both to your answer for the sake of completeness? — 8-Bit Borges, Apr 29 '19 at 02:29

Docker-compose: bind Celery to Postgres database

1 Answers1