4

My k8s cluster on GKE has a fluentd pod setup by default (currently named fluentd-gcp-v3.1.0-....), but all the logs come out unstructured jammed into a single textPayload field. It would be much nicer to have my nginx (and all other) logs come out in a jsonPayload format.

The Structured Logging docs have a lot of info if you are going to be installing the agent yourself manually, but there doesn't appear to be any information on getting Structured logging enabled on GKE where fluend is installed behind the scenes by default.

Shelling into one of those fluentd containers I see the below files, which are not the default configs (nginx/syslog/apache) listed in the Structured Logging documentation above.

/etc/google-fluentd/config.d

containers.input.conf
monitoring.conf
output.conf
system.input.conf

So basically, I've got unstructured nginx logs on GKE/Stackdriver. How to I convert them to structured?

xref
  • 1,707
  • 5
  • 19
  • 41

2 Answers2

5

With apologies for the shameless self-promotion... take a look at this:

https://medium.com/google-cloud/customizing-kubernetes-logging-part-1-a1e5791dcda8

Yuri Grinshteyn
  • 727
  • 3
  • 13
  • that's a nice post with a lot of good resources linked as well! What I'm gathering from it though is there is no easy way to start using structured logging on an existing cluster that has a `kube-system` namespace but no `stackdriver-agents` namespace – xref Mar 18 '19 at 23:34
  • I think some of the namespace information is out of date. The point is that you can simply use the existing deployment of fluentd to pull down its configuration, make updates to it, and then redeploy into either default or stackdriver-agents. – Yuri Grinshteyn Mar 19 '19 at 21:35
2

This does not [now] require any customization; single-line JSON logs in container STDOUT/STDERR will, according to the Stackdriver/GKE documentation, be parsed as structured data.

https://cloud.google.com/monitoring/kubernetes-engine/legacy-stackdriver/logging#best_practices

Single-line JSON strings written to standard output or standard error will be read into Stackdriver as structured log entries.

Brad J
  • 217
  • 1
  • 8