2

I'm trying to create a fluent-bit config which uses a record's timestamp to a custom key using a filter. Something like:

[INPUT]
  Name tail
  Path /some/path
  ... 

[FILTER]
  Name record_modifier
  Match *
  Record fluenbit_orig_ts SOME_MAGIC_WAY_TO_GET_UNIXTIME

[OUTPUT]
  Name           stdout
  Match          *

The rationale for this that I'm using several parsers, each has its own time format (Time_Format, as it's used in the regular expression parser. Even if I used Time_Keep, it won't help because the time is specified differently by different services, with a different Time_Format). I'd like the records getting to an [OUTPUT} to have the same key to describe the timestamp. In this example that key would be fluenbit_orig_ts

Is this possible?

BugoK
  • 135
  • 1
  • 10

2 Answers2

1

I got an answer from the fluent-bit slack channel.

Seems like this is possible using lua filters. Specifically, this example seems to be relevant: https://github.com/fluent/fluent-bit/blob/master/scripts/append_tag.lua

BugoK
  • 135
  • 1
  • 10
1

I've same issue. According to @BugoK, I've solved.

This is lua script.

function append_tag(tag, timestamp, record)
    new_record = record
    new_record["log_time"] = os.date("%Y-%m-%d %H:%M:%S")
    return 1, timestamp, new_record
end

And this is td-agent-bit config.

[FILTER]                                                                                                 
Name lua                                                                                             
Match nginx.access                                                                                   
script override_time.lua                                                                             
call append_tag

And then restart td-agent-bit. It works~!

jaemin park
  • 11
  • 1