6

I am trying to setup EFK (ElasticSearch 8, FluentD and Kibana) stack on K8S cluster (on-premises)

I followed this link to install elasticsearch and installed it using helm charts and followed this link to install fluentd

Output of fluentd and elasticsearch pods

[root@ctrl01 ~]#  kubectl get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                                   1/1     Running   0          136m

[root@ctrl01 ~]#  kubectl get pods -n kube-system
NAME                                                            READY   STATUS    RESTARTS   AGE
fluentd-cnb7p                                                   1/1     Running   0          107m
fluentd-dbxjk                                                   1/1     Running   0          107m

However, elasticsearch log was piled up with the following warning messages

2021-10-18 12:13:12 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2021-10-18 12:13:42 +0000 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" plugin_id="out_es"
2021-10-18 12:13:12 +0000 [warn]: suppressed same stacktrace

Conf file (tailored output)

2021-10-18 12:09:10 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type tail
    @id in_tail_container_logs
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    tag kubernetes.*
    read_from_head true
    format json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </source>
  <source>
    @type tail
    @id in_tail_minion
    path /var/log/salt/minion
    pos_file /var/log/fluentd-salt.pos
    tag salt
    format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S
  </source>

I am not sure which 'type' field it refers to. I am unable to find an example of ElasticSearch 8 for match and source directives to compare

It seems type field is not supported from ES 8 onwards but I am not sure on that. Kindly let me know the reason for the error

clxoid
  • 2,577
  • 12
  • 21
Sathish Kumar
  • 2,150
  • 10
  • 29
  • 51
  • 4
    Using `type` in request was deprecated in version 7.X and removed in version 8.X, you can read more in this [documentation link](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_schedule_for_removal_of_mapping_types). Also, Elasticsearch 8 is still in Alpha, a lot of things may not work with it yet. – leandrojmp Oct 18 '21 at 14:18
  • @leandrojmp Can you suggest me how to change the conf file ? – Sathish Kumar Oct 18 '21 at 14:21
  • Unfortunately no, I do not use fluentd, you should check their [documentation](https://docs.fluentd.org/output/elasticsearch). But again, Elasticsearch 8 is in alpha, you should stay with version 7 unless you need to test if your applications will work with version 8. – leandrojmp Oct 18 '21 at 14:25
  • the anslike,update you fluent.conf set https://stackoverflow.com/a/71420088/4234116 – venlentine Mar 10 '22 at 07:15

2 Answers2

11

I faced similar errors when I tried to use elasticsearch 8.2.3 with fluentBit 1.9.5. I could see elastic was sending logs but could not see any data in kibana webpage due to which could not create indices and saw the above error in fluent-bit pod logs. I followed this github issue and added Suppress_Type_Name On under outputs: section in my fluent-bit helm chart values.yaml file and it worked fine after that.

      [OUTPUT]
          Name  es
          Match *
          Host  {{ .Values.global.backend.es.host }}
          Port  {{ .Values.global.backend.es.port }}
          Logstash_Format Off
          Retry_Limit False
          Type  _doc
          Time_Key @timestamp
          Replace_Dots On
          Suppress_Type_Name On
          Index {{ .Values.global.backend.es.index }}
      {{ .Values.extraEntries.output }}
Aman_Bhala
  • 157
  • 1
  • 8
4

I was working on the same issue for a few days and I found a solution but just a workaround, not the optimal solution.

If you set TypeName as null for ElasticsearchSinkOptions, you don't face this issue.

Unfortunately, you can't set it from appsettings.json. At least I couldn't find a way.

In background, Serilog.Sinks.ElasticSearch library use this property as _type in HTTP header. But the '_type' header, as leandrojmp pointed out in the comment, it is no longer available in version 8.2 of ElasticSearch.

dogukanarkan
  • 341
  • 2
  • 8