Kubernetes with logrotate sidecar mount point issue

Question

I am trying to deploy a test pod with nginx and logrotate sidecar. Logrotate sidecar taken from: logrotate

My Pod yaml configuration:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-apache-log
  labels:
    app: nginx-apache-log

spec:
  containers:

  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
    volumeMounts:
    - name: logs
      mountPath: /var/log

  - name: logrotate
    image: path/to/logrtr:sidecar
    volumeMounts:
    - name: logs
      mountPath: /var/log

  volumes:
  - name: logs
    emptyDir: {}

What I'd like to achieve is Logrotate container watching /var/log//.log, however with the configuration above, nginx container is failing because there is no /var/log/nginx:

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
2018/10/15 10:22:12 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (2: No such file or directory)

However if I change mountPath for nginx from

mountPath: /var/log

to:

mountPath: /var/log/nginx

then it is starting, logging to /var/log/nginx/access.log and error.log, but logrotate sidecar sees all logs in /var/log not /var/log/nginx/. It is not a problem with just one nginx container, but I am planning to have more container apps logging to their own /var/log/appname folders.

Is there any way to fix/workaround that? I don't want to run sidecar for each app.

If I change my pod configuration to:

  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
    volumeMounts:
    - name: logs
      mountPath: /var/log
  initContainers:
    - name: install
      image: busybox
      command:
      - mkdir -p /var/log/nginx
      volumeMounts:
      - name: logs
        mountPath: "/var/log"

then it is failing with:

Warning  Failed     52s (x4 over 105s)  kubelet, k8s-slave1  Error: failed to start container "install": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"mkdir -p /var/log/nginx\": stat mkdir -p /var/log/nginx: no such file or directory": unknown

score 1 · Answer 1 · answered Oct 15 '18 at 11:01

1

Leave the mount path as /var/log. In your nginx container, execute mkdir /var/log/nginx in a startup script. You might have to tweak directory permissions a bit to make this work.

answered Oct 15 '18 at 11:01

Roland Weber

1,865
2
17
27

That depends on your container image. Check which CMD it uses, or with what command it gets started in your kubernetes spec. You have to modify the container image to execute `mkdir`. Alternatively, you can modify the logrotate container image, and pass the directory to be created as an environment variable. – Roland Weber Oct 16 '18 at 07:50

score 0 · Answer 2 · answered Oct 15 '18 at 14:26

0

If you are running nginx in kubernetes, it is probably logging to stdout. When you run kubectl logs <nginx pod> nginx it will show you access and error logs. These logs are automatically logrotated by kubernetes, so you will not need a logrotate sidecar in this case.

If you are ever running pods that are not logging to stdout, this is a bit of an antipattern in kubernetes. It is more to your advantage to always log to stdout: kubernetes can take care of log rotation for you, and it is also easier to see logs with kubectl logs than by running kubectl exec and rummaging around in a running container

answered Oct 15 '18 at 14:26

Lindsay Landry

4,657
1
15
19

nginx is just a example/poc that logrotate sidecar is doing its job. The plan is to rotate logs from other container apps. – DisplayName Oct 15 '18 at 14:54
1

Kubernetes does not do log rotation according to this document: https://kubernetes.io/docs/concepts/cluster-administration/logging – raine Aug 02 '20 at 15:48

Kubernetes with logrotate sidecar mount point issue

2 Answers2