Why are my parser types not picked up in my elastic index?

Question

I'm using fluent-bit to forward logs to an elastic db. All my fields are being indexed in elastic under the default string type but I want some fields indexed as numbers.

I've attempted to set the types in my fluent-bit config by adding a types entry to both the docker parser and the json parser (not sure which one is being used here, these are container logs from a k8s cluster):

[PARSER]
Name   json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Types my_float_field:float my_integer_field:integer

[PARSER]
Name        docker
Format      json
Time_Key    time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep   On
Types my_float_field:float my_integer_field:integer

But these fields continue to appear as string types in fresh elastic indexes under the ids log_processed.my_float_field and log_processed.my_integer_field. I'm sure I'm doing something obviously wrong but I can see what.

Any pointers would be greatly appreciated.

score 0 · Answer 1 · answered Jun 11 '20 at 07:16

Use Elasticsearch index templates.

AFAIK the JSON parser plugin doesn't support "Type" parameter. It keeps the original JSON data types, so if my_float_field and my_integer_field contain quoted values, JSON parser will interpret them as strings as well. See this example from the docs:

A simple configuration that can be found in the default parsers configuration file, is the entry to parse Docker log files (when the tail input plugin is used):
[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S %z
The following log entry is a valid content for the parser defined above:
{"key1": 12345, "key2": "abc", "time": "2006-07-28T13:22:04Z"}
After processing, it internal representation will be:
[1154103724, {"key1"=>12345, "key2"=>"abc"}]

If you are using Logstash format in Elasticsearch output plugin, you can define an Elasticsearch index template containing the desired type mapping. The template will be applied to each newly created index (not to existing indices). The format changed in Elasticsearch 7, so be sure to check the correct documentation version. For the 7.7 version:

PUT _template/template_1
{
  "index_patterns": ["fluent-bit-*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "my_float_field": {
        "type": "float"
      },
      "my_integer_field": {
        "type": "integer"
      }
    }
  }
}

Why are my parser types not picked up in my elastic index?

1 Answers1