0

I have an application deployed on kubernetes on GKE,

Kubernetes version: v1.7.11-gke.1

Stackdriver Logging is enabled on my cluster

fluntd-gcp image on my cluster (by default):

gcr.io/google-containers/fluentd-gcp:2.0.9

my logs were all ok, seen in stackdriver, but since a few days ago logs from one deployment (lets call it my-app ) stopped arriving in stackdriver

even though they are logged from my app :

kubectl logs -f my-app-3270987706-cx0r2 --namespace=production

{"time":"2018-01-30 16:11:13.155","msg":"ignoring xml"}
{"time":"2018-01-30 16:11:14.155","msg":"success blabla"}

I see the following logs from fluentd:

2018-01-30 16:11:46 +0000 [warn]: emit transaction failed: 
error_class=Errno::ENOENT error="No such file or directory @ sys_fail2 -
 (/var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log, /var/log/fluentd-
buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log)" tag="docker"

2018-01-30 16:11:46 +0000 [warn]: suppressed same stacktrace

2018-01-30 16:11:46 +0000 [error]: Exception emitting record: 
No such file or directory @ sys_fail2 - 
(/var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log,
 /var/log/fluentd-buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log)

why logs arent shipped to stackdriver? how can I fix it?

edit:

Ill note that the logs of other apps do appear in stackdriver

the logs of the failing app are very big - maybe thats why they fail to log?

dina
  • 4,039
  • 6
  • 39
  • 67
  • There is a limit size for log entries [with maximum value of 100KB](https://cloud.google.com/logging/quotas#size_and_usage_limits).Nevertheless I believe longer entries would be trimmed but still parsed to Stackdriver logging. I wonder if it could be related to a fluentd overflow error similar to [this one](https://github.com/fluent/fluentd/issues/1485) or maybe the buffer file being placed on an unexpected location. Have you made [any changes to the default installation](https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/#configuring-stackdriver-logging-agents) – Carlos Jan 30 '18 at 21:53
  • this was a bug in 1.7 , I followed [this](https://stackoverflow.com/questions/48442077/log-flood-after-master-upgrade-from-1-6-13-gke-0-to-1-7-11-gke-1) – dina Jan 31 '18 at 09:23

0 Answers0