Docker error - cannot remove a container

Question

I haven't upgraded Docker recently. I used to be able to remove containers without error. All the time now I get:

docker rm -f 05344fa394a4

Error response from daemon: driver "overlay" failed to remove root filesystem for 
05344fa394a43e5080deb1a43fdeab3a6f141038069f1a49209e29ec8f06a20c: 
remove /var/lib/docker/overlay/c9eb21a91ae1a37a121855a1ef65a89593994dc036affa9ff295b59b4eca1af5/merged: 
device or resource busy

I basically have to reboot (systemctl restart docker does NOT fix) in order to successfully remove and continue. Anyone have any insight as to what might be wrong?

I am running on Centos 7.3 as root with Docker 17.06. Everything in /var/lib/docker and /var/lib/docker/overlay is owned by root:root is protection 700. [/var/lib/docker/network is actually 750]

i meet this issue too, i can deleted it after restart machine. — Shen Yudong, Feb 07 '18 at 13:30
Have you used a utility to determine what process has that file open? — Dave Swersky, Feb 09 '18 at 15:38
Can you try this one https://github.com/moby/moby/issues/34198 as @thaJeztah gave a solution and it's because of lower kernel version in CentOS. — mohan08p, Feb 09 '18 at 16:06
Hi! Did you try docker stop before removing it? docker stop 05344fa394a4 and 2nd: what docker ps -a says? — karol, Feb 10 '18 at 14:48

BMitch · Accepted Answer · 2018-02-16T14:02:00.580

This is a known issue being worked on by docker. I'd recommend upgrading to a newer kernel (the 7.4 centos equivalent is a large improvement) and migrating to overlay2. Once the 18.03 release comes out, I'd recommend upgrading since some of the fixes from the docker side will be included there.

When deleting containers causes this issue, you can try the force option (docker rm -f), which will often still report the error but ultimately remove the container after a few seconds. You'll likely have some leftover pieces that would be best cleaned up by a reboot, but it may help workaround any other issues the filesystem locking problem is causing.

One other thing I've seen improve docker on RH and CentOS systems is to make a file with:

$ cat /etc/systemd/system/docker.service.d/override.conf
[Service]
MountFlags=slave

This should now be the default in current releases of docker, but you may have copied your unit file from an older version that didn't have it. To apply that file, you'll need to systemctl daemon-reload && systemctl restart docker as root.

score 1 · Answer 2 · answered Feb 16 '18 at 13:15

According to the Plesk Site, a solution would be

Stop the container
Reboot the server
Remove the container

Or, if you are running proxy rules, you have to remove them at first:

Remove proxy rule, stop container, remove container -> error
Stop container, remove proxy rule, remove container -> works

In case of a), if you already removed the proxy rule, creating another container and adding a new proxy rule (same domain/subdomain) towards the new one will allow you to remove the older one immediately.

This seems somehow to be linked to modifying nginx config file after the container is stopped.

score 0 · Answer 3 · answered May 23 '19 at 14:22

Just try removing the catalog, which contains too long path of sub-catalogs.

For example

docker rm -f <container_name>

it gives error like the following:

Error response from daemon: driver "overlay" failed to remove root filesystem for aa867db5afb0b70c5680f3265cf6dd4d02816190e23123d8c97f1e7b502fc928: lstat /dockerimgs/docker/overlay/429a98b6053371c82902a2ad845b069037f2403b87c5632ad952ec2879/upper/opt/catalog1/catalog2/catalog3/.../.../.../catalogN ... file name too long

So, having that container stopped, just go to the catalog, used by your container as its file system (/dockerimgs/docker/overlay/429a98b6053371c82902a2ad845b069037f2403b87c5632ad952ec2879/...) and delete the catalog mentioned in the error message which appears on "docker rm".

Docker error - cannot remove a container

3 Answers3