Docker container auto healing is Kubernetes suitable for one instance?

Question

I have one docker container what is running pyppeteer. It have memory leak, so it will stoped in 24 hours.

I need some auto healing system, I think Kubernetes can do that. No loadbalance, just one instance, one container. It is suitable?

++++

Finally, I selected docker-py, managed by using containers.run, containers.prune.

It is working for me.

score 2 · Accepted Answer · answered Oct 08 '19 at 08:31

If your container has no state, and you know it is going to run out of memory every 24 hours, I would say cronjob is the best option.

You can do what you want on k8s, but that's overkilling. Entire k8s cluster for one container, doesn't sound right to me.

Another thing is if you have more apps, or containers as k8s can run lots of services independent one from another, so you would not be wasting resources.

score 0 · Answer 2 · answered Oct 08 '19 at 07:49

There are several options for your use case, one of them is running kubernetes. But you should consider the overhead on resources and maintenance burden when running kubernetes just for a single container.

I suggest you explore having systemd restart your container in case it crashes or just simple use docker itself: With the --restart=always parmeter the docker daemon ensures the container is running. Note: Even after restarting the system docker will ensure the container is restarted in that case. So a --restart=on-failure might be a better option.

See this page for more information: https://docs.docker.com/config/containers/start-containers-automatically/#use-a-restart-policy

I had tried the option --restart=always, but it doesn't clear memory. so couldn't auto-heal. I want to run new container. — Joon, Oct 08 '19 at 08:18

score 0 · Answer 3 · answered Oct 08 '19 at 12:30

I didn't work with Puppeteer but after short research found this:

By default, Docker runs a container with a /dev/shm shared memory space 64MB. This is typically too small for Chrome and will cause Chrome to crash when rendering large pages. To fix, run the container with docker run --shm-size=1gb to increase the size of /dev/shm. Since Chrome 65, this is no longer necessary. Instead, launch the browser with the --disable-dev-shm-usage flag:

const browser = await puppeteer.launch({
  args: ['--disable-dev-shm-usage']
});

This will write shared memory files into /tmp instead of /dev/shm.

Hope this help.

score 0 · Answer 4 · answered Oct 15 '19 at 11:43

It is possible to use Kubernetes auto-healing feature without creating full-scale Kubernetes cluster. It's only required to install compatible versions of docker and kubelet packages. It could be helpful to install kubeadm package also.

Kubelet is the part of Kubernetes control-plane that takes care of keeping Pods in healthy condition. It runs as a systemd service, and creates static pods using YAML manifest files from /etc/kubernetes/manifests (location is configurable).

All other application troubleshooting can be done using regular docker commands:

docker ps ...
docker inspect
docker logs ...
docker exec ...
docker attach ...
docker cp ...

A good example of this approach from the official documentation is running external etcd cluster instances. (Note: Kubelet configuration part may not work as expected with recent kubelet versions. I've put more details on that below.)

Also kubelet can take care of pod resource usage by applying limits part of a pod spec. So, you can set the memory limit and when container reach this limit kubelet will restart it.

Kubelet can make a health-check of the application in the pod, if liveness probe section is included in the Pod spec. If you can create a command to check your application condition more precisely, kubelet can restart the container when the command return non zero exit code several times in a row (configurable).

If kubelet refuses to start, you can check kubelet logs using the following command:

journalctl -e -u kubelet

Kubelet can refuse to start mostly because of:

absence of kubelet initial config. It can be generated using kubeadm command: kubeadm init phase kubelet-start. (You may also need to generate CA certificate /etc/kubernetes/pki/ca.crt mentioned in the kubelet config. It can be done using kubadm: kubeadm init phase certs ca)
different cgroups driver settings for docker and kubelet. Kubelet works fine with both cgroupsfs and systemd drivers. Docker default driver is cgroupfs. Kubeamd also generates kubelet config with cgroupsfs driver, so just ensure that they are the same. Docker cgroups driver can be specified in the service definition file, e.g /lib/systemd/system/docker.service or /usr/lib/systemd/system/docker.service:
```
#add cgroups driver option to ExecStart:
ExecStart=/usr/bin/dockerd \
      --exec-opt native.cgroupdriver=systemd   # or cgroupfs
```

To change cgroups driver for recent kubelet version it's required to specify kubelet config file for the service, because such command line options are deprecated now:

sed -i 's/ExecStart=\/usr\/bin\/kubelet/ExecStart=\/usr\/bin\/kubelet --config=\/var\/lib\/kubelet\/config.yaml/' /lib/systemd/system/kubelet.service

Then change the cgroups line in the kubelet config. Couple more options also require changes. Here is the kubelet config that I've used for same purpose:

address: 127.0.0.1                           # changed, was 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: false                           # changed, was true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt  # kubeadm init phase certs ca
authorization:
  mode: AlwaysAllow                          # changed, was Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs                       # could be changed to systemd or left as is, as docker default driver is cgroupfs
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeLeaseDurationSeconds: 40
nodeStatusReportFrequency: 1m0s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

Restart docker/kubelet services:

systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet

Docker container auto healing is Kubernetes suitable for one instance?

4 Answers4