Trying to export logs to a fluent-bit TCP input server, from a Python module.
fluent-bit is setup to listen to a localhost port with a TCP server, and in Python the logging creates a SocketHandler logging handler, to redirect the logs on that port.
Python:
import logging
from logging.handlers import SocketHandler
from pythonjsonlogger import jsonlogger
format = '%(asctime)s [%(levelname)s] %(funcName)s %(process)d %(message)s'
logger = logging.getLogger(module_name)
formatter = jsonlogger.JsonFormatter(self.format)
socket_handler = SocketHandler("localhost", 5170)
logger.addHandler(socket_handler)
logger.setFormatter(formatter)
logger.info("Hello Log!")
fluent-bit.conf
[SERVICE]
Flush 1
Grace 30
Log_Level debug
Parsers_File parsers.conf
[INPUT]
Name tcp
Tag ApplicationLogs
Listen 0.0.0.0
Port 5170
Chunk_Size 1024
Buffer_Size 10024
format none
[OUTPUT]
Name file
Match ApplicationLogs
Path /tmp/fluent-test-output.logs
parsers.conf
[PARSER]
Name QueryLogSeparator
Format regex
Regex (?<log>-{20,})
[PARSER]
Name QueryLog
Format regex
Regex (?<start>-{20,})(?<content>[\S\s]+?EOE)
When the format is set to json
in the fluent-bit.conf
, fluent-bit return the error:
[2022/07/21 10:12:56] [ warn] [input:tcp:tcp.0] invalid JSON message, skipping
,
and when it set to none
, the error is:
[2022/07/21 10:07:46] [debug] [input chunk] skip ingesting data with 0 bytes
.
Has anybody been able to intergate fluent-bit
TCP server, with Python, and solve the above problem?