1

I have been using fluentbit across multiple kubernetes clusters and sending data to a centralized elasticsearch cluster. Logs have a lot of metadata on them that helps up to pin point the origin of a given log. Its missing the one crucial information about logs metadata, cluster_name.

Is there a way to enrich logs with a cluster name using fluentbit?

I can have a filter which adds a custom static value like this,

custom-filter.conf: |
    [FILTER]
        Name                modify
        Match               *
        Add                 cluster_name robust_cluster_1

but this approach seems error prone.

I am looking for an approach where cluster name is dynamically fetched by fluentbit.

Ojas Kale
  • 2,067
  • 2
  • 24
  • 39

1 Answers1

0

You can interpolate environment variables, as shown in the Record Modifier example:

[FILTER]
    Name    record_modifier
    Match *
    Record hostname ${HOSTNAME}

Usage with the modify filter and cluster name would be pretty much the same

[FILTER]
    Name modify
    Match *
    Add cluster_name ${CLUSTER_NAME}
ffflabs
  • 17,166
  • 5
  • 51
  • 77
  • Thanks @ffflabs, Yep, like how mentioned in the question, this is the approach we are using for now. Its can just cause issues if user fails to set the correct cluster name. I was wondering if fluentbit can interpolate the cluster name. Fluentbit sets the value of `kubernetes.host` as nodename, but in multi cluster environment host should be a cluster name. I think this is more of an Kubernetes issue, since kubernetes itself does not keep track of clusterID or cluster name. `modify` or `record_modifier` is the way to go for now, I guess. ` is t – Ojas Kale Jun 07 '22 at 05:35
  • ps: `kubectl config current-context | cut -d '@' -f2` is giving you cluster name which is scoped for your local machine. So this is out of question. – Ojas Kale Jun 07 '22 at 05:36
  • ouch, I took your question pretty much literally, as if you were using a static string. Anyway, these values would be static if treated as config, so if you need something reactive, you could grab part of the environment through an input and leverage the LUA filter https://docs.fluentbit.io/manual/pipeline/filters/lua – ffflabs Jun 07 '22 at 06:03
  • Yep looked into Lua piece as well. The issue is there is no way to fetch a cluster name as there is no name associated with a cluster form a server POV. – Ojas Kale Jun 07 '22 at 09:48