1

I would like to throttle logs per kubernetes service - in other other words apply throttle filter for specific tags or keys. Fluentd had a group_key attribute that enables throttling at a service level but Fluent-bit does not.

This is the configuration of the throttle filter

[FILTER]
        Name     throttle
        Match    kube.*
        Rate     1000
        Window   300
        Interval 1s

The Match key throttles on the whole for all records that matches the tag kube.*. I want to be able to include the namespace name in here and create like number of throttle filters to throttle based on a specific namespaces.

Any thoughts on how to achieve this?

fledgling
  • 991
  • 4
  • 25
  • 48

2 Answers2

0

The following configuration adds throttling per namespace:

[FILTER]
    Name          throttle
    Match         kubernetes.*_my-namespace1_*
    Alias         my-namespace1
    Rate     1000
    Window   300
    Interval 1s

[FILTER]
    Name          throttle
    Match         kubernetes.*_my-namespace2_*
    Alias         my-namespace2
    Rate     1000
    Window   300
    Interval 1s
dudicoco
  • 176
  • 2
  • 5
0

You can use lua script in fluent-bit to do rate limiting based on kubernetes service.

Example script - https://github.com/fluent/fluent-bit/blob/master/scripts/rate_limit.lua

Gokul
  • 97
  • 1
  • 11