0

When performing a kubectl rolling-update of a replication controller in Kubernetes (Google Container Engine), the Google (Stackdriver) Logging agent doesn't pick up the newly deployed pod. The Log is stuck at the last message produced from the old pod.

Consequently, the logs for the replication controller are out-of-date until we do a manual restart (i.e. kubectl scale and kubectl delete) of the pod and the logs are updated again.

Can anybody else confirm that behaviour? Is there a workaround?

codemoped
  • 245
  • 2
  • 10

1 Answers1

1

I can try to repro the behavior, but first can you try running kubectl logs <pod-name> on the newly created pod after doing the rolling-update to verify that the new version of your app was producing logs at all?

This sounds more likely to be an application problem than an infrastructure problem, but if you can confirm that it is an infra problem I'd love to get to the bottom of it.

Alex Robinson
  • 12,633
  • 2
  • 38
  • 55
  • I just had to do some rolling-updates again and I can confirm that the Stackdriver logs are not updated. `docker logs` on the newly created container confirmed that logs are produced (`kubectl logs` would require me to open a SSH tunnel). I'd say that is as expected given that they are stateless and independent. I suspect the error has something to do with the renaming process during the rolling-update. – codemoped May 25 '16 at 09:17
  • Thank you for the response! I'll start digging into it. – Alex Robinson May 25 '16 at 18:14
  • What version are your nodes at? `gcloud container clusters list` should show the version of your nodes in the NODE_VERSION column. My first attempt to repro this on a cluster with nodes at 1.2.0 failed (meaning that the logs from the new pods worked). – Alex Robinson May 25 '16 at 18:30
  • And also the kubectl version while you're at it? `kubectl version` – Alex Robinson May 25 '16 at 18:41
  • I also can't repro with nodes at version 1.1.1, albeit still with a kubectl client at 1.2.3. Mind helping me repro what you're seeing by explaining what your setup and you're doing in more detail? You can send me an email at `arob at google dot com` if you'd prefer that approach. – Alex Robinson May 25 '16 at 20:27
  • Current details: MASTER_VERSION 1.2.4; NODE_VERSION 1.1.8; kubectl Client/Server Version 1.2.4 – codemoped May 30 '16 at 08:37