How to set breakpoint in Dockerfile itself?

Question

Searching up the above shows many results about how to set breakpoints for apps running in docker containers, yet I'm interested in setting a breakpoint in the Dockerfile itself, such that the docker build is paused at the breakpoint. For an example Dockerfile:

FROM ubuntu:20.04

RUN echo "hello"
RUN echo "bye"

I'm looking for a way to set a breakpoint on the RUN echo "bye" such that when I debug this Dockerfile, the image will build non-interactively up to the RUN echo "bye" point, exclusive. After then, I would be able to interactively run commands with the container. In the actual Dockerfile I have, there are RUNs before the breakpoint that change the file system of the image being built, and I want to analyze the filesystem of the image at the breakpoint by being able to interactively run commands like cd / ls / find at the time of the breakpoint.

Answer 1

You can't set a breakpoint per se, but you can get an interactive shell at an arbitrary point in your build sequence (between steps).

Let's build your image:

Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu:20.04
 ---> 1e4467b07108
Step 2/3 : RUN echo "hello"
 ---> Running in 917b34190e35
hello
Removing intermediate container 917b34190e35
 ---> 12ebbdc1e72d
Step 3/3 : RUN echo "bye"
 ---> Running in c2a4a71ae444
bye
Removing intermediate container c2a4a71ae444
 ---> 3c52993b0185
Successfully built 3c52993b0185

Each of the lines that says ---> 0123456789ab with a hex ID has a valid image ID. So from here you can

docker run --rm -it 12ebbdc1e72d sh

which will give you an interactive shell on the partial image resulting from the first RUN command.

There's no requirement that the build as a whole succeed. If a RUN step fails, you can use this technique to get an interactive shell on the image immediately before that step and re-run the command by hand. If you have a very long RUN command, you may need to break it into two to be able to get a debugging shell at a specific point within the command sequence.

Answer 2

I don't think this is possible directly - that feature has been discussed and rejected.

What I generally do to debug a Dockerfile is to comment all of the steps after the "breakpoint", then run docker build followed by docker run -it image bash or docker run -it image sh (depending on whether you have bash installed inside the container).
Then, I have an interactive shell, and I can run commands to debug why later stages are failing.

I agree that being able to set a breakpoint and poke around would be a handy feature, though.

Answer 3

You can run commands in intermediate containers using Remote shell debugging tricks.

Make sure your container images include basic utilities like netcat (nc) and fuser. These utilities enable "calling home" from any intermediate container image. At home you'll answer calls with netcat (or socat). This netcat will send your commands to containers, and print their outcomes. This debugging approach will work even on Dockerfiles that are built on unknown worker nodes somewhere in cloud.

Example:

FROM debian:testing-slim

# Set environment variables for calling home from breakpoints (BP)
ENV BP_HOME=<IP-ADDRESS-OF-YOUR-HOST>
ENV BP_PORT=33720
ENV BP_CALLHOME='BP_FIFO=/tmp/$BP.$BP_HOME.$BP_PORT; (rm -f $BP_FIFO; mkfifo $BP_FIFO) && (echo "\"c\" continues"; echo -n "($BP) "; tail -f $BP_FIFO) | nc $BP_HOME $BP_PORT | while read cmd; do if test "$cmd" = "c" ; then echo -n "" >$BP_FIFO; sleep 0.1; fuser -k $BP_FIFO >/dev/null 2>&1; break; else eval $cmd >$BP_FIFO 2>&1; echo -n "($BP) "  >$BP_FIFO; fi; done'

# Install needed utils (netcat, fuser)
RUN apt update && apt install -y netcat psmisc

# Now you are ready to run "eval $BP_CALLHOME" wherever you want to call home.

RUN BP=before-hello eval $BP_CALLHOME

RUN echo "hello"

RUN BP=after-hello eval $BP_CALLHOME

RUN echo "bye"

Start waiting for and answering calls from a Dockerfile before launching a Docker build. On home host run nc -k -l -p 33720 (alternatively socat STDIN TCP-LISTEN:33720,reuseaddr,fork).

This is how above example looks like at home:

$ nc -k -l -p 33720
"c" continues
(before-hello) echo *
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
(before-hello) id
uid=0(root) gid=0(root) groups=0(root)
(before-hello) c
"c" continues
(after-hello)
...

Answer 4

The recent (May 2022) project ktock/buildg offers breakpoints.

See "Interactive debugger for Dockerfile" from Kohei Tokunaga

buildg is a tool to interactively debug Dockerfile based on BuildKit.

• Source-level inspection
• Breakpoints and step execution
• Interactive shell on a step with your own debugigng tools
• Based on BuildKit (needs unmerged patches)
• Supports rootless

The command break, b LINE_NUMBER sets a breakpoint.

Example:

$ buildg.sh debug --image=ubuntu:22.04 /tmp/ctx
WARN[2022-05-09T01:40:21Z] using host network as the default            
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 195B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 3.0s

#4 [build1 1/2] FROM docker.io/library/busybox@sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8
#4 resolve docker.io/library/busybox@sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8 0.0s done
#4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 772.81kB / 772.81kB 0.2s done
Filename: "Dockerfile"
      2| RUN echo hello > /hello
      3| 
      4| FROM busybox AS build2
 =>   5| RUN echo hi > /hi
      6| 
      7| FROM scratch
      8| COPY --from=build1 /hello /
>>> break 2
>>> breakpoints
[0]: line 2
>>> continue
#4 extracting sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0.0s done
#4 DONE 0.3s
...

From PR 24:

Add --cache-reuse option which allows sharing the build cache among invocation of buildg debug to make the 2nd-time debugging faster.
This is useful to speed up running buildg multiple times for debugging an errored step.

Note that breakpoints on cached steps are ignored as of now.
Because of this limitation, this feature is optional as of now. We should fix this limitation and make it the default behaviour in the future.

Answer 5

Man, Docker makes things hard. Here's a workaround I cooked up:

1. Insert FROM scratch where you want the break point.
2. Run docker build . --target=<n-1> where <n> is the number of FROM commands before your "breakpoint". Eg, if it's a single stage build, use --target=0.
   • Alternatively, if you have already named the stage where you want the break point with FROM <image> AS <stage> then you can use --target=<stage> instead.

Docker has cached all your successful layers anyway (even if you can't see them), and because the FROM "breakpoint" comes before the (potentially unsuccessful) point of interest, the build should all come from cache and be very fast.

So for example, if my Dockerfile looks like this:

FROM debian:bullseye AS build

RUN apt-get update && apt-get install -y \
    build-essential cmake ninja-build \
    libfontconfig1-dev libdbus-1-dev libfreetype6-dev libicu-dev libinput-dev libxkbcommon-dev libsqlite3-dev libssl-dev libpng-dev libjpeg-dev libglib2.0-dev

<SNIP lots of other setup commands>

ADD my_source.tar.xz /
WORKDIR /my_source

RUN ./configure -option1 -option2
RUN cmake --build . --parallel
RUN cmake --install .


FROM alpine
COPY --from=build /my_build /my_build

...

Then I can add a "breakpoint" like this:

FROM debian:bullseye AS build

RUN apt-get update && apt-get install -y \
    build-essential cmake ninja-build \
    libfontconfig1-dev libdbus-1-dev libfreetype6-dev libicu-dev libinput-dev libxkbcommon-dev libsqlite3-dev libssl-dev libpng-dev libjpeg-dev libglib2.0-dev

<SNIP lots of other setup commands>

ADD my_source.tar.xz /
WORKDIR /my_source

#### BREAKPOINT ###
FROM scratch
#### BREAKPOINT ###

RUN ./configure -option1 -option2
RUN cmake --build . --parallel
RUN cmake --install .


FROM alpine
COPY --from=build /my_build /my_build

...

and trigger it with docker build . --target=build