2

Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument Happens with any new project after

This error happens when launching any of my ddev projects or when creating a new one with ddev config.

Update: This happens only when docker data (/var/lib/docker by default) is on a ZFS volume. The problem disappeared as soon as I moved it to EXT4.

It does not seem to be a problem with docker or docker compose, as I have launched the example/hello-world containers of both projects successfully.

I have tried removing ~/.ddev, downgrading ddev, and downgrading docker. I was not able to downgrade docker-compose.

Log:

$ ddev start                                                                                                                                     
Starting test... 
mkcert may not be properly installed, we suggest installing it for trusted https support, `brew install mkcert nss`, `choco install -y mkcert`, etc. and then `mkcert -install` 
Running   Command=ip address show dev docker0
Building db 
Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument 
Failed to start test: Failed to run docker-compose [-f /home/tester/htdocs/test/.ddev/.ddev-docker-compose-full.yaml up --build -d], err='exit status 1', stdout='Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM $BASE_IMAGE
 ---> 33f3288968e4
Step 3/6 : ARG username
', stderr='Building db
Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument'

config.yml

name: test
type: php
docroot: ""
php_version: "7.3"
webserver_type: nginx-fpm
router_http_port: "80"
router_https_port: "443"
xdebug_enabled: false
additional_hostnames: []
additional_fqdns: []
mariadb_version: "10.2"
mysql_version: ""
provider: default
use_dns_when_possible: true
composer_version: ""

System:

  • Docker 20.10.1
  • docker-compose 1.27.4
  • Manjaro Linux (Arch derivative)
  • ZFS 2.0.0-1 (on root)

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 27
 Server Version: 20.10.1
 Storage Driver: zfs
  Zpool: rpool
  Zpool Health: ONLINE
  Parent Dataset: rpool/root
  Space Used By Parent: 20958640128
  Space Available: 282678809088
  Parent Quota: no
  Compression: off
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b.m
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.2-2-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 30.9GiB
 Name: testpc
 ID: BKDP:GAWJ:BUCV:OVP6:QUKY:UQCZ:AESP:V3AN:EXHE:BWJU:4TT7:YPEO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Update: This fixed itself after a complete system reinstall; unfortunately I have no idea how the reinstall might have helped.

gbyte
  • 225
  • 2
  • 7
  • 1
    I can't bring Manjaro up on Parallels any more unfortunately. But could you please make a gist of the output of this script: https://gist.github.com/rfay/361c4e1df165774a5262268ef6768884 and post the link here? All that's going on building the db is https://gist.github.com/rfay/20f519f3ad7aecc6f63109857cc92d2a I recommend you delete all images `docker rmi -f $(docker images -q)` - that way if there's a corrupted image it can get re-pulled for you. – rfay Jan 05 '21 at 18:55
  • @rfay Thanks for looking into this. I deleted the images and ran the script. The same error happened when building the db service. https://gist.github.com/gbytedev/aa06933122aca020bbeafdd7783cbd87 – gbyte Jan 05 '21 at 19:24
  • I just did a new install of Manjaro 20.2.1, installed docker (20.10.1) and docker-compose (1.27.4) , installed ddev (1.16.5). Had no trouble of any kind. I'm afraid, as is so often the case, Manjaro/Arch has ***ed you. kernel 5.9.16-1 and 5.10.2-2 both worked. I can only recommend that you completely uninstall docker and purge all data and reinstall again. Beyond that... there's something wrong with your system. – rfay Jan 05 '21 at 23:06
  • 1
    @rfay Thank you for going through all that hassle for me. I just found out the error only happens if the docker folder is on a ZFS volume. I moved it to an external EXT4 formatted SSD and voila, at least I can continue working. Please let me know if you have any more pointers. – gbyte Jan 06 '21 at 13:42
  • 1
    @gbyte Thanks for reporting. I suggest to change title of question: add mention of zfs. E.g. "failed to copy files from zfs partition" – Alex Yu Jan 06 '21 at 13:44

1 Answers1

1

(Don't have enough reputation to comment, but this isn't a full answer.)

I hit the same issue, also with Docker on Arch w/ZFS. I was able to "fix" it by passing DOCKER_BUILDKIT=0 docker build .... It appears to be an issue somewhere in buildkit.

I haven't dug in any further to see what specifically is causing the issue.

Jack
  • 11
  • 1
  • Thanks for the good workaround. You should be able to just turn off buildkit in your envronment with `export DOCKER_BUILDKIT=0` in your .zshrc, .profile, or whatever. – rfay Jan 07 '21 at 22:02
  • Here's the issue: https://github.com/moby/buildkit/issues/1758 - Could one of you follow it up with a test case they can go with? – rfay Jan 07 '21 at 22:03
  • 1
    @rfay The workaround with the environmental variable did not work for me. I put DOCKER_BUILDKIT=0 into /etc/environment and logged back in just to make sure. – gbyte Jan 09 '21 at 14:28
  • @gbyte are you using straight Docker, or docker-compose. If you're using docker-compose, you might also need to set `COMPOSE_DOCKER_CLI_BUILD=1`. edit: at-mentioned wrong user. – Jack Jan 11 '21 at 21:46
  • Thanks @Jack, Yes I believe ddev uses docker-compose, but I am still getting these errors, even after verifying that the two env variables are set correctly. – gbyte Jan 14 '21 at 21:28