I have a basic fluent-bit configuration that outputs Kubernetes logs to New Relic. The log message format is just horrible and I couldn't really find a proper way to parse them, they look like this:
"\"{\"correlation_id\":\"058537f6906593eb8b5de90e515e6f79\",\"datetime\":\"2021-01-12T09:16:57Z\",\"level\":\"info\",\"message\":\"request\",\"payload\":{\"request\":{\"host\":\"hidden.co.uk\",\"method\":\"POST\",\"path\":\"/api/v1/access/acs\",\"proto\":\"HTTP/1.1\",\"realIP\":\"10.99.26.212\",\"referer\":\"http://m.facebook.com\",\"remoteAddr\":\"0.0.0.0:58576\",\"userAgent\":\"Mozilla/5.0 (iPhone; CPU iPhone OS 14_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 [FBAN/FBIOS;FBDV/iPhone12,1;FBMD/iPhone;FBSN/iOS;FBSV/14.3;FBSS/2;FBID/phone;FBLC/en_GB;FBOP/5]\"},\"response\":{\"latency\":6,\"size\":0,\"status\":307}},\"request\":{\"host\":\"hidden.co.uk\",\"method\":\"POST\",\"path\":\"/api/v1/access/acs\",\"proto\":\"HTTP/1.1\",\"realIP\":\"10.99.26.212\",\"referer\":\"http://m.facebook.com\",\"remoteAddr\":\"0.0.0.0:58576\",\"userAgent\":\"Mozilla/5.0 (iPhone; CPU iPhone OS 14_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 [FBAN/FBIOS;FBDV/iPhone12,1;FBMD/iPhone;FBSN/iOS;FBSV/14.3;FBSS/2;FBID/phone;FBLC/en_GB;FBOP/5]\"},\"request_id\":\"81efd22f-5022-44d8-9415-c6ff6b98bb54\",\"response\":{\"latency\":6,\"size\":0,\"status\":307},\"timestamp\":1610443017}\"\n"
Config is as follows:
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level ${LOG_LEVEL}
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input-kubernetes.conf
@INCLUDE output-newrelic.conf
@INCLUDE filter-kubernetes.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path ${PATH}
Parser ${LOG_PARSER}
DB /var/log/flb_kube.db
Mem_Buf_Limit 7MB
Skip_Long_Lines On
Refresh_Interval 10
filter-kubernetes.conf: |
[FILTER]
Name record_modifier
Match kube.*
Record cluster ${CLUSTER_NAME}
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log Off
K8S-Logging.Exclude On
Labels On
Annotations On
output-newrelic.conf: |
[OUTPUT]
Name newrelic
Match *
licenseKey ${NRIA_LICENSE_KEY}
endpoint ${ENDPOINT}
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
Decode_Field_As escaped log
[PARSER]
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
I'm new to Fluent-bit and the documentation seems rather confusing, so I would really appreciate if someone can point me in the right direction here.
