Docker: container exits, but it still looks like running

Question

I have a Docker service (specifically MongoDB). After some kind of failure (specifically No space left on device), it gets weird:

Command docker-compose logs --follow --tail=10 -t service-name exits and writes service-name exited with code 14. When I run docker-compose up service-name, it attempts to start the container.
However, docker ps shows the service running and "Up 2 days".
When the underlying issue is resolved (i.e., some space is freed), the service doesn't restart automatically, although docker-compose.yaml contains restart: unless-stopped. It however starts after I tell explicitly Docker to do so (i.e., docker-compose up some-service).

Why does this happen? Is there some better solution than just running docker-compose up -d <list of services> every minute?

Excerpt from docker-compose.yaml:

  mongodb:
    image: mongo:5
    container_name: mongodb
    environment:
    - PUID=1000
    - PGID=1000
    ports:
    - 127.0.0.1:…:27017
    volumes:
    - …:/data/db
    - "…:/docker-entrypoint-initdb.d/:ro"
    restart: unless-stopped

Excerpt from docker ps:

144a…   mongo:5   "docker-entrypoint.s…"   3 days ago     Up 5 hours       127.0.0.1:…->27017/tcp    mongodb

Failed attempts to start the container:

$ sudo docker-compose up mongodb
Starting mongodb ... error

ERROR: for mongodb  Cannot start service mongodb: mkdir /var/lib/docker/overlay2/…/merged: no space left on device

ERROR: for mongodb  Cannot start service mongodb: mkdir /var/lib/docker/overlay2/…/merged: no space left on device
ERROR: Encountered errors while bringing up the project.

Sounds like the primary process in the container itself did not terminate (this is what determines whether the container restarts). Is this a custom image? — Turing85, Jul 22 '23 at 07:58
@Turing85 It is a vanilla `mongo:5` container. I see, it partially looks like running (docker ps), but partially like terminated (`mongodb exited with code 14` in docker-compose logs, docker-compose up service-name tries to start it). — v6ak, Jul 22 '23 at 08:09
What do you mean by "tries to start it"? What happens if docker tries to start that container? Could it be that you have - in fact - two containers running (one suck, one trying to start)? Can you post the `docker-compose.yml`, as well as the output of `docker ps` and `docker ps -a`? — Turing85, Jul 22 '23 at 08:34
@Turing85 Added excerpts from `docker-compose.yaml` and `docker ps`, as well as what happens when I try to start the service before adding free space. Note that it is not a surprise that it fails, it is a surprise that it attempts to start it, although `docker ps` shows it running. — v6ak, Jul 22 '23 at 12:25
Hm... have you tried destroy the whole deployment (via `docker compose down`) and re-deploy (via `docker compose up`)? — Turing85, Jul 22 '23 at 15:45

Docker: container exits, but it still looks like running

0 Answers0