1

I’m shipping logs from on-premise to the GCP using google-fluentd agent but on the GCP side it has missing few metadata like resource.labels.instance_id

Is there any way to have it set via fluentd configuration? I saw some solutions for kubernetes but none for logs from VMs.

Fluentd config looks like:

<source>
  @type tail
  path /var/log/my-app/*
  pos_file /var/lib/google-fluentd/pos/my-app.tail.log.pos
  tag test.tail-logs
  format /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3}) (?<severity>[^\s]+) (?<message>.*)/
  time_format %F %T,%L
</source>

<match my-app.**>
  @type record_reformer
  tag ${tag_suffix[1]}
</match>

<filter test.*>
  @type record_transformer
  <record>
    instance_id "#{Socket.gethostname}"
  </record>
</filter>
JackTheKnife
  • 3,795
  • 8
  • 57
  • 117
  • 1
    you should use the record_transformer plugin (https://docs.fluentd.org/filter/record_transformer). instance_id is specific to google cloud, you can just set some hostname or whatever arbitrary value. – SYN Jun 16 '21 at 16:54
  • @SYN Yes, I understand that and I was looking at that option but I was not successful to map it to the `resource.labels.instance_id` object as it came as `instance_id` only and monitoring did not picked up it as it did with the date or the log severity. – JackTheKnife Jun 16 '21 at 18:02
  • Please could you share your deployments file related to google-fluentd agent? Some useful links... [Troubleshooting the Logging Agent](https://cloud.google.com/logging/docs/agent/logging/troubleshooting), also [Configure on-premise and hybrid cloud logging](https://cloud.google.com/logging/docs/blue-medora/on-premise-hybrid-logging) unsure if it will fit your case. There is a similar question on this [Google Groups forum](https://groups.google.com/g/kubernetes-users/c/P_KnOkzGuws) ,[Resources](https://stackoverflow.com/questions/63626898/google-cloud-permissions-by-resource-label-or-id) – Pit Jun 17 '21 at 08:12
  • @Pedro I have updated OP with the fluentd config. Also Google Groups forum link it is what I was talking as kubernetes working example which I didn't get to work for logs posted from a VM. Looking at the google-fluentd code I'm not sure if that is even possible. – JackTheKnife Jun 17 '21 at 16:07
  • Does this answer your question? [How to Send On Premises Kubernetes Logs to Stackdriver](https://stackoverflow.com/questions/53232785/how-to-send-on-premises-kubernetes-logs-to-stackdriver) – Pit Jun 24 '21 at 08:22

0 Answers0