Docker Cloud autotest cant find service

Question

I am currently trying to dockerize one of my Django API projects. It uses postgres as the database. I am using Docker Cloud as a CI so that I can build, lint and run tests.

I started with the following DockerFile

# Start with a python 3.6 image
FROM python:3.6

ENV PYTHONUNBUFFERED 1
ENV POSTGRES_USER postgres 
ENV POSTGRES_PASSWORD xxx 
ENV DB_HOST db 

RUN mkdir /code

ADD . /code/

WORKDIR /code

RUN pip install -r requirements.txt

RUN pylint **/*.py

# First tried running tests from here.

RUN python3 src/manage.py test

But this DockerFile always fails as Django cant connect to any database when running the unit tests and justs fails with the following error as no postgres instance is running in this Dockerfile

django.db.utils.OperationalError: could not translate host name "db" 
to address: Name or service not known

Then I discovered something called "Autotest" in Docker Cloud that allows you to use a docker-compose.text.yml file to describe a stack and then run some commands with each build. This seemed like what I needed to run the tests, as it would allow me to build my Django image, reference an already existing postgres image and run the tests.

I removed the

 RUN python3 src/manage.py test

from the DockerFile and created the following docker-compose.test.yml file.

 version: '3.2'

 services:
   db:
     image: postgres:9.6.3
     environment:
       - POSTGRES_USER=$POSTGRES_USER
       - POSTGRES_PASSWORD=$POSTGRES_PASSWORD

   sut:
     build: .
     command: python src/manage.py test
     environment:
       - POSTGRES_USER=$POSTGRES_USER
       - POSTGRES_PASSWORD=$POSTGRES_PASSWORD
       - DB_HOST=db
     depends_on:
       - db

Then when I run

  docker-compose -f docker-compose.test.yml build

and

  docker-compose -f docker-compose.test.yml run sut

locally, the tests all run and all pass.

Then I push my changes to Github and Docker cloud builds it. The build itself succeeds but the autotest, using the docker-compose.test.yml file fails with the following error:

  django.db.utils.OperationalError: could not connect to server: 
  Connection refused
  Is the server running on host "db" (172.18.0.2) and accepting
  TCP/IP connections on port 5432?

So it seems like the db service isnt being started or is too slow to start on Docker Cloud compared to my local machine?

After Google-ing around a bit I found this https://docs.docker.com/compose/startup-order/ where it says that the containers dont really wait for each other to be a 100% ready. Then they recommend writing a wrapper script to wait for postgres if that is really needed.

I followed their instructions and used the wait-for-postgres.sh script.

Juicy part:

  until psql -h "$host" -U "postgres" -c '\l'; do
    >&2 echo "Postgres is unavailable - sleeping"
    sleep 1
  done

and replaced the command in my docker-compose.test.yml from

 command: python src/manage.py test

to

 command: ["./wait-for-postgres.sh", "db", "python", "src/manage.py", 
 "test"]

I then pushed to Github and Docker Cloud starts building. Building the image works but now the Autotest just waits for postgres forever (I waited for 10 minutes before manually shutting down the build process in Docker Cloud)

I have Google-d a fair bit around today and it seems like most "Dockerize Django" tutorials dont really mention unit testing at all.

Am I running Django unit tests completely wrong using Docker?

Seems strange to me that it runs perfectly fine locally but when Docker Cloud runs it, it fails!

score 0 · Answer 1 · answered Jun 25 '17 at 14:32

I seem to have fixed it by downgrading the docker-compose version in the file from 3.2 to 2.1 and using healthcheck.

The healthcheck option gives me a syntax error in depends_on clause as you have to pass an array into it. No idea why this is not supported in version 3.2

But here is my new docker-compose.test.yml that works

version: '2.1'

services:
  db:
    image: postgres:9.6.3
    environment:
      - POSTGRES_USER=$POSTGRES_USER
      - POSTGRES_PASSWORD=$POSTGRES_PASSWORD
    healthcheck:
      test: ["CMD-SHELL", "psql -h 'localhost' -U 'postgres' -c 
            '\\l'"]
      interval: 30s
      timeout: 30s
      retries: 3

 sut:
   build: .
   command: python3 src/manage.py test
   environment:
     - POSTGRES_USER=$POSTGRES_USER
     - POSTGRES_PASSWORD=$POSTGRES_PASSWORD
     - DB_HOST=db
   depends_on:
   // Does not work in 3.2
     db:
       condition: service_healthy

Docker Cloud autotest cant find service

1 Answers1