How do I debug crashing application from inside docker and prevent it from terminating?

Question

I need to debug an application that is crashing inside a docker, how do I achieve that the docker does not stop when the app crashes, allowing me to -it inside and try starting the app manually?

Thats…. Not how You should be doing it. Log your errors to stderr and review them when the container crashes. — GregL, Jun 12 '21 at 04:28
@GregL I log the errors, but since I can't reproduce the error outside of container, I need to step over the code inside the docker. I may also need to modify files in the docker on the fly to understand what's wrong and how to solve that. — Qwerty, Jun 14 '21 at 00:23

score 1 · Answer 1 · answered Jul 21 '22 at 21:01

You can, i needed to add single package to existing docker for my Python app to work and didnt want to wait for build time.

docker run container_name tail -f /dev/null

Then get in via

docker exec -it container_name /bin/sh

Ref:https://levelup.gitconnected.com/keep-docker-container-running-for-debugging-fc2dfa39472c

score 0 · Accepted Answer · answered Jul 22 '22 at 19:46

So in the end, this is what I did:

First, know that it is possible to pause the container before it runs any command, such as npm start, etc. We can then enter the docker container with terminal and run any command by hand. Running the app like this won't exit and shut down the container on a critical error. We can even mount local filesystem inside the docker and run the app from there with the docker picking up changes live - perfect for debugging!

Project layout

In my particular case, it was a NestJS app with two important folders

/src - the app source code (development, nest start --watch)
/dist - the output folder for built app (production)
(npm run build(nest build), npm run start:prod(node dist/src/main))

The docker was then configured to copy the built /dist folder and relevant files to an /app folder which was used as WORKDIR from where the app was then launched.

Excerpt from Dockerfile (simplified a lot)

WORKDIR /app
RUN npm run build

COPY /app/tsconfig.json ./tsconfig.json
COPY /app/package.json ./package.json
COPY /app/dist ./dist

CMD ["npm", "run", "start:prod"]

With this clear, let's now look at package.json "scripts"

  "scripts": {
    "dev": "npm run start:dev",
    "dev:docker": "npm run docker:debug:start:local",
    "build": "nest build",
    "start:dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/src/main",
    "docker:build": "docker build -t my-nest-service --platform linux/amd64 .",
    "docker:start": "docker run --env-file ./.env --rm --name my-nest-service -p 9002:9002 my-nest-service",
    "docker:stop": "docker stop my-nest-service",
    "docker:debug:start": "docker run --env-file ./.env --rm --name my-nest-service -p 9002:9002 my-nest-service sleep infinity",
    "docker:debug:start:local": "docker run -v %cd%:/app --env-file ./.env --rm --name my-nest-service -p 9002:9002 my-nest-service sleep infinity",
    "docker:debug:enter": "docker exec -it my-nest-service sh"
  }

Explanation

So, what is happening above is that we have commands to build the app build, start development or prod start:dev start:prod, start and stop docker docker:start docker:stop, start docker paused docker:debug:start, enter-in docker docker:debug:enter with terminal, ..

And then a SPECIAL command dev:docker or docker:debug:start:local, which allows us to mount local repository inside Docker's /app folder, which (as mentioned above) is the WORKDIR folder where our app code is copied..

dev:docker                 # alias for docker:debug:start:local
docker:build
docker:start               # Starts the image and app from container's `dist` folder
docker:stop
docker:debug:start         # Starts image without running the app
docker:debug:start:local   # Same as above but mounts local filesystem
docker:debug:enter         # Allows us to connect terminal inside running container

Debugging inside Docker

dev:docker - Start docker container in paused state (pause infinity) and mounted local repo
docker:debug:enter - This allows us to get inside the container with a terminal
npm run dev We can then run the app manually from inside docker
Use debugging tools, connect remotely with VSCode, etc. etc.

Follows actual excerpt from our README.md:

Development inside Docker

Useful for debugging docker but also for platform agnostic development. When on deployed environment, the image itself is set to run npm run start:prod (see Dockerfile CMD). All commands prefixed with docker: in package.json are never used and are there to ease control of docker in local environment.

dev:docker                 # alias for docker:debug:start:local
docker:build
docker:start               # Starts the image and app from container's `dist` folder
docker:stop
docker:debug:start         # Starts image without running the app
docker:debug:start:local   # Same as above but mounts local filesystem
docker:debug:enter         # Allows us to connect terminal inside running container

PS: For local development inside docker use trio dev:docker, docker:debug:enter, npm run dev. Read below.

Build the image
```
npm run docker:build
```
Start the container with sleep
sleep pauses the container before running the application, which allows us to inspect it even in case of a terminating error.
There are two options:

a) Run the complete package built within the container. This runs the code from dist folder (see Dockerfile CMD command).
```
npm run docker:debug:start
npm run start:prod # then inside docker (see step 3a)
```
b) Or run with mounted local repo. This will completely overwrite the container's /app directory with your local repo's content, every change to your repo will reflect inside docker.
*Note %var% is windows syntax, use $var on unix.
```
npm run docker:debug:start:local
npm run dev # then inside docker (see step 3b)
```
Connect into the container
Launch this command from a second terminal.
```
docker:debug:enter
```
Then, inside the container you can
a) start the built production code from dist. This is the same command docker uses in Dockerfile CMD.
```
npm run start:prod
```
b) start from src. (When using docker:debug:start:local.)
```
npm run dev
```
At this point, you can run any command, since you have your full local repository linked.
Open app in browser http://localhost:9002/api/v1/ and do your thing.
Stop the container
```
docker:stop
```
Tip: You can check running containers.
```
docker ps
```
Start the app in normal way to verify that everything works and test in browser on http://localhost:9002/api/v1/.
PS: You might need to rebuild your image 1).
```
# run prod in docker
npm run docker:start

# runs prod without docker
npm run start:prod
```

How do I debug crashing application from inside docker and prevent it from terminating?

2 Answers2

Project layout

Explanation

Debugging inside Docker

Development inside Docker