1

I use fluent-bit input from systemd and out put to clickhouse.And I code the plugin by golang. example:

func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
    dec := output.NewDecoder(data, int(length))

    batch, err := client.connect.PrepareBatch(context.Background(), insertSql)
    if err != nil {
        return output.FLB_PROXY_OUTPUT_PLUGIN
    }

    count := 0
    startFlushTime := time.Now()
    for {
        ret, ts, record := output.GetRecord(dec)
        // do someting change ret to rowPtr
        err = batch.AppendStruct(rowPtr)
        if err != nil {
            log.Error(nil, "batch append error", zap.Error(err), zap.Any("rowPtr", rowPtr))
        }
    }
    err = batch.Send()
    if err != nil {
        log.Error(nil, "batch send error", zap.Error(err))
        return output.FLB_PROXY_OUTPUT_PLUGIN
    }

When fluent-bit call function FLBPluginFlushCtx. logs will be write in clickhouse.


    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info
        Health_Check On
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Parsers_File parsers.conf
        Parsers_File parsers_custom.conf
    
    [INPUT]
        Name            systemd
        Systemd_Filter  _COMM=***
        DB              /fluent-bit/flag/flag

    [OUTPUT]
        Name                  clickhouse
        Match                 *
        Address               ***
        Database              ***
        Table                 ***
        Username              ***
        Password              ***

fluent-bit has config Flush 5.

fluent-bit call function FLBPluginFlushCtx every five seconds.But if a lot of logs, fluent-bit will call this function About 1500 lines. How can I set this number? I hope fluent-bit call this function every five seconds or 20000 lines.

zhang_
  • 21
  • 1
  • 4

0 Answers0