0

I'm running docker on Ubuntu 22.04 (CLI, not Docker-Desktop) and I keep getting layer verification error. See the example below:

mjozwikowski:~$ docker -D pull postgres:latest
latest: Pulling from library/postgres
648e0aadf75a: Downloading [==========================>                        ]  15.44MB/29.12MB
f715c8c55756: Download complete 
b11a1dc32c8c: Downloading  2.693MB
f29e8ba9d17c: Verifying Checksum 
78af88a8afb0: Waiting 
b74279c188d9: Waiting 
6e3e5bf64fd2: Waiting 
b62a2c2d2ce5: Waiting 
eba91ca3c7a3: Waiting 
d4a24cdf2433: Waiting 
b20f8a8dfd5c: Waiting 
e0731dd084c3: Waiting 
0361da6a228e: Waiting 
filesystem layer verification failed for digest sha256:f29e8ba9d17cfa147141648b72ff8ab49a86234dfe1194f6220690939f1daa3c

mjozwikowski:~$ docker -D pull postgres:latest
latest: Pulling from library/postgres
648e0aadf75a: Pull complete 
f715c8c55756: Pull complete 
b11a1dc32c8c: Pull complete 
f29e8ba9d17c: Pull complete 
78af88a8afb0: Pull complete 
b74279c188d9: Pull complete 
6e3e5bf64fd2: Pull complete 
b62a2c2d2ce5: Pull complete 
eba91ca3c7a3: Verifying Checksum 
d4a24cdf2433: Download complete 
b20f8a8dfd5c: Download complete 
e0731dd084c3: Download complete 
0361da6a228e: Download complete 
filesystem layer verification failed for digest sha256:eba91ca3c7a37844775569d1771c8acfab80b32d9c24f4a0b5b998d91911d747

I've tested this command multiple times. This issue occurs on different layers (although some tend to have more occurrences) and once in a while (like one in twenty) everything works. But on other images (i.e. mcr.microsoft.com/mssql/server:2019-CU18-ubuntu-20.04 I wasn't able to successfully finish the pull.

It's now a fresh installation. I've completely removed docker along with its configs and reinstalled it all. /var/lib/docker has been removed, so were /etc/docker/daemon.json and ~/.docker/config.json. I don't use proxy, the internet connection is stable, and this is not a VM - it's a native Ubuntu installation. I've tried docker-desktop and there everything is fine, so it's not a repository issue nor a connection one.

Starting the service doesn't report errors:

sie 05 14:02:03 systemd[1]: Starting Docker Application Container Engine...
░░ Subject: A start job for unit docker.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has begun execution.
░░ 
░░ The job identifier is 3852.
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.041443800+02:00" level=info msg="Starting up"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.042248549+02:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.055564872+02:00" level=info msg="Loading containers: start."
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.383176470+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 192.168.128.0/24. Daemon option --bip can be used to set a preferred IP address"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.461435658+02:00" level=info msg="Loading containers: done."
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.474691599+02:00" level=info msg="Docker daemon" commit=a61e2b4 graphdriver=overlay2 version=24.0.5
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.474754307+02:00" level=info msg="Daemon has completed initialization"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.491109520+02:00" level=info msg="API listen on /run/docker.sock"
sie 05 14:02:03 systemd[1]: Started Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has finished successfully.
░░ 
░░ The job identifier is 3852.

I'd appreciate all and any help. I'm running out of ideas.

1 Answers1

0

It seems to be a known and random error. There are some issue open on GitHub about that, such as https://github.com/GoogleContainerTools/kaniko/issues/1962.

The 2 possibile workaround, that not solve the error but only mitigate it, seems to be:

  • removing all the images, for example removing the /var/lib/docker folder, and retry (but you said you already did it)
  • disabling cache, by setting the --cache=true flag
Alez
  • 1,913
  • 3
  • 18
  • 22
  • Thanks for your answer, but that's not it. I'm not using kaniko, not even building the images. Those are pulls from dockerHub that are failing. There's no cache management here. I've tried `docker -D pull --disable-content-trust=false postgres` but with same results :( – Marcin Jóźwikowski Aug 05 '23 at 14:13