2

When I run docker build I get this:

Sending build context to Docker daemon  10.24kB
WARN[11935] Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/21647778a50f097d4535246ec5206960dd909f4bb8b0e3d04fdd53a7402fc2de-init: exec: "auplink": executable file not found in $PATH 
Step 1/2 : FROM debian:jessie
 ---> 86baf4e8cde9
Step 2/2 : RUN apt-get update
WARN[11935] Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/21647778a50f097d4535246ec5206960dd909f4bb8b0e3d04fdd53a7402fc2de: exec: "auplink": executable file not found in $PATH 
 ---> Running in 1fef9bef5bf7
ERRO[11934] containerd: start container                   error="shim error: fork/exec /usr/bin/docker-runc: exec format error" id=1fef9bef5bf77141a97669d2aa785e74f9027a849919a937f714e93fbae3916d
ERRO[11935] Create container failed with error: shim error: fork/exec /usr/bin/docker-runc: exec format error 
ERRO[11934] containerd: deleting container                error="fork/exec /usr/bin/docker-runc: exec format error: \"\""
WARN[11935] Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/21647778a50f097d4535246ec5206960dd909f4bb8b0e3d04fdd53a7402fc2de: exec: "auplink": executable file not found in $PATH 
shim error: fork/exec /usr/bin/docker-runc: exec format error

Here is the content of my Dockerfile:

FROM debian:jessie

RUN apt-get update

What is the issue? It makes no sens to me.

ll /usr/bin | grep docker
-rwxr-xr-x  1 root   root    18471276 Aug  3 22:08 docker*
-rwxr-xr-x  1 root   root     9938352 Aug  3 22:08 docker-containerd*
-rwxr-xr-x  1 root   root     8941944 Aug  3 22:08 docker-containerd-ctr*
-rwxr-xr-x  1 root   root     3824920 Aug  3 22:08 docker-containerd-shim*
-rwxr-xr-x  1 root   root    40328816 Aug  3 22:08 dockerd*
-rwxr-xr-x  1 root   root           0 Aug  3 22:08 docker-init*
-rwxr-xr-x  1 root   root           0 Aug  3 22:08 docker-proxy*
-rwxr-xr-x  1 root   root           0 Aug  3 22:08 docker-runc*
-rwxr-xr-x  1 root   root     8962864 Aug  3 21:40 docker-volume-local-persist*

Installing from https://docs.docker.com/engine/installation/linux/docker-ce/binaries/

file $(which docker-runc):

/usr/bin/docker-runc: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=e3d80e183baf26a9d48c3f0435931d42aa1bf340, not stripped

lsb_release -a

Distributor ID: Ubuntu
Description:    Ubuntu 17.04
Release:    17.04
Codename:   zesty

docker --version

Docker version 17.06.0-ce, build 02c1d87

dockerd --version

Docker version 17.06.0-ce, build 02c1d87

docker-containerd --version

containerd version 0.2.3 commit: cfb82a876ecc11b5ca0977d1733adbe58599088a

docker-containerd-ctr --version

ctr version 0.2.3 commit: cfb82a876ecc11b5ca0977d1733adbe58599088a

docker-init --version

tini version 0.13.0 - git.949e6fa

docker-runc --version

runc version 1.0.0-rc3
commit: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
spec: 1.0.0-rc5
jnbdz
  • 4,863
  • 9
  • 51
  • 93

2 Answers2

0

It looks like something related to aufs FS, What OS Do you use ? and Did you recently updated your machine ?

Update:

For CONFIG_MEMCG_SWAP_ENABLED: missing, CONFIG_RT_GROUP_SCHED: missing and warning: /proc/config.gz does not exist, searching other paths for kernel config ...

These are missing kernel configuration and flags, Please make sure you installed linux-image-extra-$(uname -r) linux-image-extra-virtual, First make sure you stopped docker daemon sudo systemctl stop docker and install these packages (contains extra driver that support containers and aufs check docker docs https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu):

sudo apt-get install \ linux-image-extra-$(uname -r) \ linux-image-extra-virtual

And update your grub GRUB_CMDLINE_LINUX_DEFAULT add thesecgroup_enable=memory swapaccount=1 to your /etc/default/grubconfiguration file then update your grub sudo update-grub check https://github.com/moby/moby/issues/4250 && https://github.com/moby/moby/pull/4251

For aufs problems there is another solution for modern dockers as docker moved from aufs to overlay and overlay2 is to configure your machine and apply overlay but make sure you backed up your docker images and containers as this fix might make you lose them check Unable to start Docker Service in Ubuntu 16.04

misraX
  • 887
  • 5
  • 15
  • I forgot to added sorry. I just added it. You are right. I installed `cgroup-lite` and `aufs-tools`. But I am still getting some of the other errors. – jnbdz Aug 07 '17 at 01:45
  • I did also tried running this script: https://raw.githubusercontent.com/moby/moby/master/contrib/check-config.sh to help track down what the issue or issues are. I get this: `CONFIG_MEMCG_SWAP_ENABLED: missing`, `CONFIG_RT_GROUP_SCHED: missing` and `warning: /proc/config.gz does not exist, searching other paths for kernel config ...` – jnbdz Aug 07 '17 at 01:47
  • `CONFIG_MEMCG_SWAP_ENABLED: missing` -- `(cgroup swap accounting is currently enabled) ` and `CONFIG_RT_GROUP_SCHED: missing` – jnbdz Aug 09 '17 at 16:04
  • I am unsure I need to add linux-image-extra (https://askubuntu.com/questions/153023/what-is-the-linux-image-extra-package-for-and-do-i-need-it). I tried it to no result. – jnbdz Aug 09 '17 at 19:54
  • Yes it's not necessary but as I mentioned it contains extra drivers, Anyway Which docker version do you have? and Did you install it from the package manager or manually? – misraX Aug 09 '17 at 20:44
  • `dockerd --version`: `Docker version 17.06.0-ce, build 02c1d87` – jnbdz Aug 09 '17 at 20:48
  • From a package manager or manually ? and what's the output of `file $(which docker-runc)` – misraX Aug 09 '17 at 20:57
  • I am installing from https://docs.docker.com/engine/installation/linux/docker-ce/binaries/ – jnbdz Aug 09 '17 at 20:59
  • `file $(which docker-runc)`: `/usr/bin/docker-runc: empty` – jnbdz Aug 09 '17 at 21:00
  • I just notice that docker-runc is empty... Very strange. I will re-download it... – jnbdz Aug 09 '17 at 21:01
  • `file $(which docker-runc)` : `/usr/bin/docker-runc: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=e3d80e183baf26a9d48c3f0435931d42aa1bf340, not stripped` – jnbdz Aug 09 '17 at 21:05
  • It's risky to install from source and not necessary specially when you have the package distributed to your package manager that will dynamically link binaries, manage versions and dependencies. Did you mount your cgroupfs, or Did you apt-get install `aufs-tools cgroupfs-mount libltdl7` these are the dependencies for the official docker-ce installation. – misraX Aug 09 '17 at 21:29
  • Not from the source but from the binary. It was already compiled. I installed `cgroup-lite` not `cgroupfs-mount` and not `libltdl7`. I did install `aufs-tools`. – jnbdz Aug 09 '17 at 21:56
  • Alright kindly make sure you have them. mounting cgroups is important for docker. – misraX Aug 09 '17 at 22:13
  • Only `cgroupfs-mount` is not installed. – jnbdz Aug 09 '17 at 22:24
  • I was able to start some of my containers. But I am getting (I don't know if this is related) `failed to cleanup ipc mounts:`, `failed to umount /var/lib/docker/containers/b8335587ad37a36a35fc2f1eb57b3b3d91f61af75262eb075b9045664885ee30/shm: invalid argument`, `libcontainerd: client is out of sync, restore was called on a fully synced container (28c274e176c9c112f8bc73f6916860665c63f1407ca2670944deee7b6ae2e747).`, `libcontainerd: failed to retrieve container 28c274e176c9c112f8bc73f6916860665c63f1407ca2670944deee7b6ae2e747 state: rpc error: code = 2 desc = containerd: container not found` – jnbdz Aug 10 '17 at 14:45
  • It might relate to: `Your kernel does not support cgroup rt period`, `Your kernel does not support cgroup rt runtime` – jnbdz Aug 10 '17 at 14:57
0

Is this running on a non-Intel/ARM 64-bit Ubuntu? For example a Raspberry Pi or ARM64 CPU? This error:

shim error: fork/exec /usr/bin/docker-runc: exec format error

Would imply that either (a) the binary install on your machine is corrupted in some way, or (b) you are attempting to run a binary for a different architecture on your system.

Can you post the output of uname -a and file /usr/bin/docker-runc? That might help narrow down the source of your problem.

Phil E
  • 1,840
  • 14
  • 19
  • uname -a `Linux badger 4.10.0-30-generic #34-Ubuntu SMP Mon Jul 31 19:38:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux` – jnbdz Aug 09 '17 at 16:06
  • No output for `docker-runc` – jnbdz Aug 09 '17 at 16:07
  • I updated my question. For some reason `docker-runc` was empty. Anyways, now is not empty anymore. – jnbdz Aug 09 '17 at 21:10
  • It's disconcerting that you had zero length files on your filesystem; any chance you had an out of disk space condition? Makes me wonder if the layers of the `debian:jessie` image were impacted as well, giving you the exec format errors (e.g. missing/corrupted binary content). Can you try a different image and pull and run it first to verify? E.g. `docker pull alpine:latest` and `docker run -ti --rm alpine:latest sh`? – Phil E Aug 10 '17 at 15:15
  • It's weird because I still have a lot of disk space: `/dev/sda1 19G 4.7G 15G 25% /` I will try out what you are proposing. – jnbdz Aug 10 '17 at 15:35
  • **any chance you had an out of disk space condition?** No idea. Is there a log file I can check? – jnbdz Aug 10 '17 at 15:36
  • `docker pull alpine:latest`: `latest: Pulling from library/alpine Digest: sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe Status: Image is up to date for alpine:latest` – jnbdz Aug 10 '17 at 15:38
  • Right now when I start `dockerd &` I get multiple warnings: `WARN[0000] Your kernel does not support cgroup rt period`, `WARN[0000] Your kernel does not support cgroup rt runtime`, `WARN[0000] libcontainerd: client is out of sync, restore was called on a fully synced container`, `WARN[0000] libcontainerd: failed to retrieve container`, `WARN[0000] failed to cleanup ipc mounts:`, `failed to umount /var/lib/docker/containers/28c274e176c9c112f8bc73f6916860665c63f1407ca2670944deee7b6ae2e747/shm: invalid argument`. Anyways the others are similar. I don't know if it helps. – jnbdz Aug 10 '17 at 15:42
  • Maybe this is related: https://serverfault.com/questions/867996/apt-get-install-linux-image-extra-getting-w-mdadm-etc-mdadm-mdadm-conf-defin – jnbdz Aug 10 '17 at 16:05
  • Also I am running the host on a VM VirtualBox/Vagrant. – jnbdz Aug 10 '17 at 17:12
  • Given you had some weird issues about zero-length files in your Docker installation, and there seems to be some leftover cruft in `/var/lib/docker`; assuming this is a test VM without any critical work, would you be open to stopping the daemon, `rm -fR /var/lib/docker` and then starting the daemon again? Given a fresh pull of an image worked and runs, I don't see any problem with the engine itself or `docker-runc` anymore. – Phil E Aug 11 '17 at 02:53
  • `WARN[0001] failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: no such file or directory. Deleting synchronously` got this new warning. Maybe it's normal. – jnbdz Aug 11 '17 at 03:23
  • Seems to work when I run `docker run hello-world`. But I am still getting these warnings: `WARN[0001] Your kernel does not support cgroup rt period` and `WARN[0001] Your kernel does not support cgroup rt runtime`. – jnbdz Aug 11 '17 at 03:26
  • Another detail: I try installing it with apt-get. So it might be different than the previous binary setup. Let me see tomorrow. By the way thanks for the help. – jnbdz Aug 11 '17 at 03:33