I use fluent-bit input from systemd and out put to clickhouse.And I code the plugin by golang. example:
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
dec := output.NewDecoder(data, int(length))
batch, err := client.connect.PrepareBatch(context.Background(), insertSql)
if err != nil {
return output.FLB_PROXY_OUTPUT_PLUGIN
}
count := 0
startFlushTime := time.Now()
for {
ret, ts, record := output.GetRecord(dec)
// do someting change ret to rowPtr
err = batch.AppendStruct(rowPtr)
if err != nil {
log.Error(nil, "batch append error", zap.Error(err), zap.Any("rowPtr", rowPtr))
}
}
err = batch.Send()
if err != nil {
log.Error(nil, "batch send error", zap.Error(err))
return output.FLB_PROXY_OUTPUT_PLUGIN
}
When fluent-bit call function FLBPluginFlushCtx
. logs will be write in clickhouse.
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Health_Check On
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Parsers_File parsers.conf
Parsers_File parsers_custom.conf
[INPUT]
Name systemd
Systemd_Filter _COMM=***
DB /fluent-bit/flag/flag
[OUTPUT]
Name clickhouse
Match *
Address ***
Database ***
Table ***
Username ***
Password ***
fluent-bit has config Flush 5
.
fluent-bit call function FLBPluginFlushCtx
every five seconds.But if a lot of logs, fluent-bit will call this function About 1500 lines.
How can I set this number? I hope fluent-bit call this function every five seconds or 20000 lines.