0

We are running a docker swarm and using Monit to see resources utilisation. The Process memory for dockerd keeps on growing over time. This happens on all nodes that at least perform a docker action e.g docker inspect or docker exec. I'm suspecting it might be something related to this these actions but I'm not sure how to replicate it. I have a script like

#!/bin/sh
set -eu

containers=$(docker container ls | awk '{if(NR>1) print $NF}')

# Loop forever
while true; 
do    
    for container in $containers; do
        echo "Running Inspect on $container"
        CONTAINER_STATUS="$(docker inspect $container -f "{{.State}}")"
    done
done

but I'm open to other suggestions

E_K
  • 2,159
  • 23
  • 39

1 Answers1

0

Assuming you can run ansible to run a command via ssh on all servers:

ansible swarm -a "docker stats --no-stream"

A more SRE solution is containerd + Prometheus + AlerManager / Grafana to gather metrics from the swarm nodes and then implement alerting when container thresholds are exceeded.

Don't forget you can simply set a resource constraint on Swarm services to limit the amount of memory and cpu service tasks can consume or be restarted. Then just look for services that keep getting OOM killed.

Chris Becke
  • 34,244
  • 12
  • 79
  • 148
  • Thanks @Chris. I can already get the statistics and from this I can see that `dockerd` keeps on accumulating memory over time. This is now what I'm asking how can I know which docker action is causing this? Does `docker inspect` maybe cache it's results and thus every time the command is run we get a small memory increase? – E_K Nov 28 '22 at 10:03
  • Ive not seen any memory leaks in dockerd, but I have seen an apparent pid-leak when services have an unlimited restart_policy and a task keeps on failing. I don't have monitoring on docker itself, I do monitor docker and alert on container restarts. If `docker stats` doesn't show a container using memory, then its a bug in dockerd itself. – Chris Becke Nov 28 '22 at 10:07
  • Nice suggestion I'll try monitoring restarts too but for now even if I do a `docker container ls -a`, i don't see any dead containers – E_K Nov 28 '22 at 11:17
  • 1
    I'm thinking increasingly that this needs to go to github.com/moby/moby as a bug report. – Chris Becke Nov 28 '22 at 11:34