1

I'm trying to build a container on a raspberry pi from a mac over ssh. I'm setting DOCKER_HOST to ssh://root@192.168.1.11 (192.168.1.11 is the IP of my raspberry pi).

The docker build command is failing with the following error message:

ERRO[0000] Can't add file /Users/brian/git/github/proj/backend/apps/accounts/managers.py to tar: io: read/write on closed pipe 
ERRO[0000] Can't close tar writer: io: read/write on closed pipe 
error during connect: Post "http://docker/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=localhost%3A5000%2Fbackend%3Ae150d93&target=&ulimits=null&version=1": write |1: broken pipe

The docker version on my mac is:

docker --version
Docker version 20.10.7, build f0df350

The docker version on my raspberry pi is:

docker --version
Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1

The container I'm trying to build uses this Dockerfile:

FROM --platform=linux/amd64 python:3.9
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip3 install -r requirements.txt
ADD . /code/
RUN useradd -m app
USER app

I am able to build an image with this Docker file on my mac and also on my raspberry pi.

I have found a few other issues on SO/GitHub that may be related:

Here's some more information from the raspberry pi:

ubuntu@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.8G     0  3.8G   0% /dev
tmpfs           782M  2.8M  779M   1% /run
/dev/mmcblk0p2   29G   14G   15G  49% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M  120M  133M  48% /boot/firmware
tmpfs           782M     0  782M   0% /run/user/1000

On the raspberry pi I'm having trouble finding logs from the failed build command. If I restart docker on the raspberry pi:

sudo systemctl restart docker

and then get the logs:

sudo journalctl -fu docker.service

and then run the docker build command again, I don't see any new logs when I run sudo journalctl -fu docker.service again.

Is there another way I can debug the failed docker build command?

Here is docker info from my raspberry pi:

ubuntu@ubuntu:~$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 21
  Running: 1
  Paused: 0
  Stopped: 20
 Images: 95
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1029-raspi
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.628GiB
 Name: ubuntu
 ID: 24DG:6BVV:K3TX:HT64:4U2M:DFHX:2RQA:ENP5:J2WC:V3VS:UKV3:M3TL
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: briancaffey
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

Update 8/8/2021: I didn't have certificate-based SSH access setup between my raspberry pi and Mac. Adding -l "debug" to my docker build command this was more clear:

DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
ERRO[0000] Can't add file <file> to tar: io: read/write on closed pipe 
ERRO[0000] Can't close tar writer: io: read/write on closed pipe 
DEBU[0000] FIXME: Got an status-code for which error does not match any expected type!!!: -1  module=api status_code=-1
error during connect: Post "http://docker/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=localhost%3A5000%2Fbackend%3Ae150d93&target=&ulimits=null&version=1": write |1: broken pipe

I have moved past the original issue, but I'm now facing a separate issue that was raised in the comments about platform compatibility. I had to switch from Python 3.9 to Python 3.8 in the base image and also specified --platform="linux/arm64" in the base image:

FROM --platform="linux/arm64" python:3.8
briancaffey
  • 2,339
  • 6
  • 34
  • 62
  • 1
    What do the dockerd logs say on the pi? Did you run out of disk space? – BMitch Aug 06 '21 at 12:26
  • @BMitch I updated my question with responses to your questions. Disk space looks good and I'm not really able to see anything in docker's logs that indicate any more information about the docker build command failure. – briancaffey Aug 07 '21 at 02:56
  • 2
    I am a little confused; you are specifying `--platform=linux/amd64` but the raspberry pi is not an `amd64` platform. It's not going to be able to run any of the binaries in the `amd64` version of the `python:3.9` image. – larsks Aug 08 '21 at 02:48
  • thanks @larsks you are right, I'm now having that issue. My original error I saw was solved by fixing the SSH connection between my Mac and Raspberry Pi. I'm now seeing `---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested` as the error. I changed the first line of my dockerfile to `FROM --platform="linux/arm64/v8" python:3.9`, but I'm still seeing this issue. I heard that platform detection is broken in docker v20, so I might try using v19 – briancaffey Aug 08 '21 at 04:02

1 Answers1

1

I didn't have certificate-based SSH access setup between my raspberry pi and Mac. Adding -l "debug" to my docker build made this clear:

DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
DEBU[0000] commandconn: starting ssh with [-l ubuntu -- 192.168.1.11 docker system dial-stdio] 
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):Permission denied, please try again.
 EBU[0000] commandconn (ssh):ubuntu@192.168.1.11: Permission denied (publickey,password).
ERRO[0000] Can't add file <file> to tar: io: read/write on closed pipe 
ERRO[0000] Can't close tar writer: io: read/write on closed pipe 
DEBU[0000] FIXME: Got an status-code for which error does not match any expected type!!!: -1  module=api status_code=-1
error during connect: Post "http://docker/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=localhost%3A5000%2Fbackend%3Ae150d93&target=&ulimits=null&version=1": write |1: broken pipe

To setup passworldless authentication to my raspberry pi, I followed this guide: https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md

Also, in order to build the container on raspberry pi, I needed to use the python:3.8 base image with the platform linux/arm64. Here's the docker build command I used:

docker \
    -l "debug" \
    build \
    --platform linux/arm64 \
    -t $REGISTRY/backend:$VERSION \
    -f backend/Dockerfile \
    ./backend/
briancaffey
  • 2,339
  • 6
  • 34
  • 62