2

Trying to export logs to a fluent-bit TCP input server, from a Python module.

fluent-bit is setup to listen to a localhost port with a TCP server, and in Python the logging creates a SocketHandler logging handler, to redirect the logs on that port.

Python:

import logging
from logging.handlers import SocketHandler
from pythonjsonlogger import jsonlogger


format = '%(asctime)s [%(levelname)s] %(funcName)s %(process)d %(message)s'
logger = logging.getLogger(module_name)

formatter = jsonlogger.JsonFormatter(self.format)
socket_handler = SocketHandler("localhost", 5170)
logger.addHandler(socket_handler)
logger.setFormatter(formatter)

logger.info("Hello Log!")

fluent-bit.conf

[SERVICE]
    Flush        1
    Grace        30
    Log_Level    debug
    Parsers_File parsers.conf

[INPUT]
    Name        tcp
    Tag         ApplicationLogs
    Listen      0.0.0.0
    Port        5170
    Chunk_Size  1024
    Buffer_Size 10024
    format      none

[OUTPUT]
    Name    file
    Match   ApplicationLogs
    Path    /tmp/fluent-test-output.logs

parsers.conf

[PARSER]
    Name   QueryLogSeparator
    Format regex
    Regex  (?<log>-{20,})

[PARSER]
    Name   QueryLog
    Format regex
    Regex  (?<start>-{20,})(?<content>[\S\s]+?EOE)

When the format is set to json in the fluent-bit.conf, fluent-bit return the error:

[2022/07/21 10:12:56] [ warn] [input:tcp:tcp.0] invalid JSON message, skipping ,

and when it set to none, the error is:

[2022/07/21 10:07:46] [debug] [input chunk] skip ingesting data with 0 bytes.

Has anybody been able to intergate fluent-bit TCP server, with Python, and solve the above problem?

Sarriman
  • 382
  • 5
  • 22

0 Answers0