0

I try to get log from my application container and attach fluentd log agent as sidecar container in my project. And I want to get which log is coming from which application in my Kibana dashboard. That's why I configured like that in fluentd.

<source>
  @id fluentd-containers.log
  @type tail
  path /var/log/containers/mylog*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.myapp.container
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<filter kubernetes**>
  @type record_transformer
  enable_ruby true
  <record>
    service_name ${tag_parts[1]}
    instance_name ${record["kubernetes"]["container_name"]}
    log_type ${tag_parts[2]}
    host_name ${hostname}
    send_to "ES"
  </record>
</filter>

<match kubernetes.**>
  @type stdout
</match>

But when I deployed it, ${[record[""]["container_name"]} got null as displaying unknown placeholder ${record["kubernetes"]["container_name"]}. Please help me how to resolve it, thanks.

Got that error message

0 dump an error event: error_class=RuntimeError error="failed to expand record[\"kubernetes\"][\"container_name\"] : error = undefined method []' for nil:NilClass" location="/fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.11.2/lib/fluent/plugin/filter_record_transformer.rb:310:in rescue in expand'" tag="kubernetes.myapp.container" time=2020-09-23 11:29:05.705209241 +0000 record={"message"=>"{"log":"I0923 11:28:59.157177 1 main.go:71] Health check succeeded\n","stream":"stderr","time":"2020-09-23T11:28:59.157256887Z"}"}

`

PPShein
  • 13,309
  • 42
  • 142
  • 227
  • 1
    Are you using [kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter)? – Azeem Sep 23 '20 at 05:24
  • 1
    it appears there is an extra `[` before `record` – mdaniel Sep 23 '20 at 05:24
  • @Azeem no, I'm not. do I need to use it? – PPShein Sep 23 '20 at 06:04
  • @mdaniel even I've removed extra `[`, got following error message `dump an error event: error_class=RuntimeError error="failed to expand `record[\"kubernetes\"][\"container_name\"]`` – PPShein Sep 23 '20 at 06:11
  • @PPShein: Can you verify that `enable_ruby` really works? I believe you have to mention it like this: `enable_ruby true`. See: https://docs.fluentd.org/v/0.12/filter/record_transformer#enable_ruby-optional – Azeem Sep 23 '20 at 10:26
  • @Azeem `enable_ruby true` and `enable_ruby` both are not working though. – PPShein Sep 23 '20 at 10:49
  • @PPShein: Please update the rest of your configuration along with a sample event that you're receiving in the filter. – Azeem Sep 23 '20 at 10:54
  • @Azeem I updated my question and added completed format. – PPShein Sep 23 '20 at 11:31
  • @PPShein: Is this a sample log `{"log":"I0923 11:28:59.157177 1 main.go:71] Health check succeeded\n","stream":"stderr","time":"2020-09-23T11:28:59.157256887Z"}` in those log files i.e. `/var/log/containers/mylog*.log`? – Azeem Sep 23 '20 at 11:35
  • @PPShein: `service_name` would be `myapp`. `instance_name` would be null because the `record` doesn't have any those fileds in it i.e. `record["kubernetes"]["container_name"]`. `log_type` would `container`. And, so on... The `record` is `{"message"=>"{"log":"I0923 11:28:59.157177 1 main.go:71] Health check succeeded\n","stream":"stderr","time":"2020-09-23T11:28:59.157256887Z"}"}`. As you can see that there are no such entries that you're accessing. – Azeem Sep 23 '20 at 11:44
  • 1
    @PPShein: Please go through the Container Deployment article: https://docs.fluentd.org/v/0.12/container-deployment/docker-logging-driver. – Azeem Sep 23 '20 at 11:47
  • @Azeem I found that it's required to install `kubernetes_metadata_filter` plugin. Can you post your first comment as answer then I'll mark as correct answer, thanks. – PPShein Sep 24 '20 at 02:20
  • @PPShein: That's great! I was wondering how those fields were being populated without that plugin, maybe something manual being done by you. But, that was not the case. Anyway, I'm glad that you were able to resolve this. And, you're welcome! :) Answer posted. – Azeem Sep 24 '20 at 03:18

1 Answers1

1

The record doesn't contain the required fields that you want to access i.e. record["kubernetes"]["container_name"].

You need to make sure that it has those fields.

Please go through Container Deployment and kubernetes_metadata_filter plugin for the detailed information on this.

Azeem
  • 11,148
  • 4
  • 27
  • 40