3

When we run up a container on a Compute Engine using COS, it writes its logs to JSON files. We are finding an error:

"level=error msg="Failed to log msg \"\" for logger json-file: write /var/lib/docker/containers/[image]-json.log: no space left on device".

I was looking to change the logging settings for Docker and found this article on changing the logging driver settings:

https://docs.docker.com/config/containers/logging/json-file/

My puzzle is I don't know how to set the parameters through the console or gcloud in order to set log-opts.

Kolban
  • 13,794
  • 3
  • 38
  • 60

1 Answers1

1

It seems that /var/lib/docker is on the / filesystem, and if this filesystem is running out of inodes, you will receive that message when you’ll try to run up a container and it tries to write its logs to JSON files. You can check this by running

df -i /var/lib/docker

You can configure your logging drivers to change the default values in ‘/etc/docker/daemon.json’

This is a configuration example of the daemon.json file

cat /etc/docker/daemon.json

{ "live-restore": true, "storage-driver": "overlay2" "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3", "labels": "production_status", "env": "os,customer" } }

Don’t forget to restart the docker daemon after changed the file.:

systemctl restart docker.service

You can check the following documentation for further information about how to configure logging drivers.

Please let me know the results.

Gabo Licea
  • 198
  • 3
  • Your answer should work great. Since the original question this morning, two further items have come to light that show additional information: https://issuetracker.google.com/issues/150372361 and https://github.com/GoogleCloudPlatform/konlet/commit/25259e315c5c85e95d8322261d8f9a5b13001d54 – Kolban Jun 13 '20 at 02:12
  • Kolban I'm glad to know you found this useful, coukd you tell me if my answer helped you to fix your issue? – Gabo Licea Jun 15 '20 at 18:14
  • Hi Gabo ... after posting the question I continued to research and found the links in my comment above. The link https://issuetracker.google.com/issues/150372361 seems to be very close to your own post. The recipe described both by yourself and the link talk about editing daemon.json. This seemed to work great. Further ... it seems that the recipe can be automated by using cloud-init such that the change can happen at boot time. – Kolban Jun 15 '20 at 21:23