3

I have a go web service that I've been building with docker build and running with docker run on my M1 Mac and on various Linux machines for a while. It's a simple binary installed on a distroless base and works great. Now I want to make a multi-platform image export to send to co-workers to run on their computers (M1 and Intel Macs). I've made an OC export like so:

docker buildx build --platform linux/amd64,linux/arm64 -t toy -o type=oci,dest=toy.oci .

And importing works fine:

docker image import toy.oci toy
sha256:02f7342d9d6ec2a1b66440aedb8d9a6ae0e968373fc8f2b698f7c8e73e6747e0

Running it is another matter:

docker run -d --name toy -p 4242:4242 toy:latest
docker: Error response from daemon: No command specified.
See 'docker run --help'.

This is odd, because my Dockerfile has an ENTRYPOINT. Whatever, I try to tell it the command to run:

docker run -d --name toy -p 4242:4242 toy:latest /bin/toy
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/toy": stat /bin/toy: no such file or directory: unknown.

It's almost as if the container has no files in it! docker image ls shows my 32MB image, so I know there's something there, but where is it? Is there a layer missing?

FWIW I tried type=tar, too, and got the same result.

I assume I'm missing a step, but my DuckDuckGo-foo fails me. How does one build exportable multi-platform Docker images that actually work?

theory
  • 9,178
  • 10
  • 59
  • 129

3 Answers3

1

Several things happening at once here:

  • You can only import a single platform image.
  • To import an image, you should use docker load. The docker import command loads a filesystem export from a container, and will not have any image metadata.
  • The docker load command expects its own metadata that is different from an OCI Layout.

I have functionality in regctl to pull out the specific platform and do the conversion that just got checked into main (so it will be released in v0.4.5, or you can pull the binaries direct from GHA if you don't want to build it yourself). The commands to do this look like:

# regclient works with directories rather than tars, so import the OCI tar to a directory
regctl image import ocidir://toy toy.oci
# get the digest from your local platform, you can also change "local" to "linux/amd64"
dig="$(regctl image digest --platform local ocidir://toy)"
# export the single platform image using the digest
regctl image export "ocidir://toy@${dig}" toy-docker.tar
# load into docker
docker load <toy-docker.tar

This behavior from docker will change with the transition to containerd for image management. When that happens, multi-platform images can be loaded directly into docker and their import command should support the OCI Layout directly. As of this update, that is experimental in Docker Desktop and not yet released with the docker engine on Linux.

BMitch
  • 231,797
  • 42
  • 475
  • 450
  • Very helpful, thank you. So if I just wanted to do it with `docker`, would the mean building each platform image as its own tar file? I tried building with only `--platform linux/arm64` and using `-o type=tar`, but `docker import – theory Jul 19 '22 at 18:39
  • Ah, I see, it needs to be `-o type=docker`. – theory Jul 19 '22 at 18:53
  • @theory If you're building a single platform image, you can skip all the `docker load` commands, just run `docker buildx build --load ...` and it will create the image directly. Though that is getting away from the question being asked. – BMitch Jul 19 '22 at 19:50
  • 1
    Yeah no, I was hoping for a single image export file that I could send to others and they could load and run. From what you say above, one cannot import a multi-platform image. So I'm building two separate platform-specific images and telling people which to download. – theory Jul 20 '22 at 20:04
  • also hilarious that you use `import` to import an image or `--load` to have `buildx` load it. – theory Jul 20 '22 at 20:05
  • If you're distributing images, this is what registries are for. And they support multi-platform images. It sounds like you're running into issues by avoiding the well tested pattern. And `import` doesn't import an image, `load` does, second bullet above. – BMitch Jul 20 '22 at 21:12
  • I would love to use a registry, but don't have that option for this project, which is purely internal and our internal registries have much too strict authorization rules and not all the people I want to use the toy even have accounts to use the registries. So this is my fallback. And ugh, I keep conflating import and load, sorry. – theory Jul 20 '22 at 21:34
  • 1
    @theory You are in the camp (and so am I) who think of distributing image artifacts via files. It seems this has never been "the Docker Way" of thought, which has always struck me as network-registry centric. Image files (on your local file system) seem like an afterthought. (Docker way: "If you're distributing images, this is what registries are for." to quote @BMitch) But yes as stated above, Docker has apparently chosen not to support OCI-format multi-arch image FILES, though multi-arch support has emerged in the Docker network registry. – M.Bearden Mar 27 '23 at 22:34
1

The way that I've been building images with multi-arch support on my M1 is:

  • Run docker buildx create --bootstrap
  • Ensure that the node is up and running with the appropriate platforms docker buildx ls and that is selected, there is an asterisk next to the name of the node, otherwise run: docker buildx use [node].
  • Start building your image: docker buildx build --platform linux/amd64,linux/arm64 -t sample:v1 . --push . This will push the images to a registry.

Once the image is in the registry, your co-workers can start pulling it. This is the simplest way, as I've had issues in the past trying other destination outputs.

javierlga
  • 1,409
  • 9
  • 14
  • Sadly a registry is not an option for this use case. Going to be advocating internally to make one tho, as then this whole problem goes away TBH. – theory Jul 21 '22 at 18:39
  • 3
    Honestly, it was a pain to me as well, I tried multiple outputs and just couldn't find one that actually worked, at the end, I just decided to use the private registry option as it was the only one working, also the easiest one. – javierlga Jul 21 '22 at 18:50
0

@BMitch's answer nicely addresses the various issues here, enough to get me to where I could get things to work purely with Docker. The kicker is the lack of support for loading multiplatform images, so there must be separate images per platform. The incantation for that is:

docker buildx build --platform linux/amd64 -t toy -o type=docker,dest=toy-amd64.tar .
docker buildx build --platform linux/arm64 -t toy -o type=docker,dest=toy-arm64.tar .

Then the individual tar files can be shared with whomever, and loaded with:

docker load <toy-arm64.tar
docker run -d --name toy -p 4242:4242 toy:latest /bin/toy

Switch to toy-amd64.tar on AMD64 platforms.

For kicks, I made make targets to do the builds and load like so:

docker-export: $(patsubst %,toy-%.tar, arm64 amd64)

toy-%.tar: vendor cmd/toy/toy.go schema/*.schema.json corpus/user/valid/*.json
    docker buildx build --platform linux/$* -t nytimes/toy -o type=docker,dest=$@ .

# Import an exported docker image for the current architecture.
docker-import: toy-$(patsubst aarch64,$(shell docker --debug info | grep Architecture | awk '{print $2}'),arm64).tar
    docker load <toy-$(patsubst aarch64,$(shell docker --debug info | grep Architecture | awk '{print $2}'),arm64).tar
theory
  • 9,178
  • 10
  • 59
  • 129