I am having an issue with docker on my local server, after running for a while (around 1day) docker will stop working and i wouldn't be able to connect to the default socket (unix:///var/run/docker.sock). the only way i found to fix it was to restart my server.
The server restart also has issues which is that all ipTables rules for docker get deleted, and that breaks reaching some of the containers.
I am not sure what happened as it was working correctly for months, the server have no open ports to the internet so i don't think it's a hack.
I am not sure how to debug this but i would like to understand the issue and try to mitigate it before reinstalling docker.
docker service status from systemctl (when the issue is occuring) :
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2023-05-27 21:46:50 UTC; 45min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 29160 (dockerd)
Tasks: 9
Memory: 21.9M
CGroup: /system.slice/docker.service
└─29160 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
May 27 21:46:49 scrapserver dockerd[29160]: time="2023-05-27T21:46:49.776649817Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
May 27 21:46:49 scrapserver dockerd[29160]: time="2023-05-27T21:46:49.776670250Z" level=warning msg="Your kernel does not support cgroup blkio weight"
May 27 21:46:49 scrapserver dockerd[29160]: time="2023-05-27T21:46:49.776699385Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
May 27 21:46:49 scrapserver dockerd[29160]: time="2023-05-27T21:46:49.777112108Z" level=info msg="Loading containers: start."
May 27 21:46:50 scrapserver dockerd[29160]: time="2023-05-27T21:46:50.044407601Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
May 27 21:46:50 scrapserver dockerd[29160]: time="2023-05-27T21:46:50.199976869Z" level=info msg="Loading containers: done."
May 27 21:46:50 scrapserver dockerd[29160]: time="2023-05-27T21:46:50.923749703Z" level=info msg="Docker daemon" commit=3056208 graphdriver(s)=overlay2 version=20.10.21
May 27 21:46:50 scrapserver dockerd[29160]: time="2023-05-27T21:46:50.923874210Z" level=info msg="Daemon has completed initialization"
May 27 21:46:50 scrapserver systemd[1]: Started Docker Application Container Engine.
May 27 21:46:50 scrapserver dockerd[29160]: time="2023-05-27T21:46:50.976211800Z" level=info msg="API listen on /run/docker.sock"
