0

I'm using fluent-bit to parse logs.

Input logs:

"log": "2023-02-14T14:11:38,880 INFO  [bwEngThread:In-Memory Process Worker-6] c.t.b.p.g.L.C.Log-Event-JSON - {\"CorrelationId\":\"84e7dade-9919-47dc-a1f9-299137504c2e\",\"APIName\":\"iManageOrganisation\",\"AuditCategory\":\"AUDIT_EXIT\",\"Timestamp\":\"2023-02-14T14:11:38.878Z\",\"HostName\":\"imanageorganisationv2-rest-api-6d65f89ddc-cjrtk\",\"SourceID\":\"WO0000000012265\",\"Payload\":\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><tns3:itemList xmlns:tns3=\\\"http://www.neom.com/tibcoesb/organisationResponse.xsd\\\" xmlns:tns7=\\\"http://www.neom.com/tibcoesb/CallCreateCompartmentOCI.xsd\\\" xmlns:tns16=\\\"http://www.neom.com/tibcoesb/CallCreateOrganisationUnitAWS.xsd\\\"><tns3:item><tns3:name>FDS_Ali_SubDep</tns3:name><tns3:ocId>ocid1.compartment.oc1..aaaaaaaaygtnwsb7lec7ufj3ryma3j2pbmdit4zi2b7dvnntqa4rjwlvryfa</tns3:ocId></tns3:item><tns3:item><tns3:name>FDS_Ali_Dep</tns3:name><tns3:ocId>ocid1.compartment.oc1..aaaaaaaavvxam5fgathj4ipsyeujwbxvkceiv3uq3rmh7d5hjo5zpxw2dvkq</tns3:ocId></tns3:item><tns3:item><tns3:name>FDS_Ali_Dep</tns3:name><tns3:awsId>ou-80i3-g96emafc</tns3:awsId></tns3:item><tns3:item><tns3:name>FDS_Ali_SubDep</tns3:name><tns3:awsId>ou-80i3-2y5rq2wt</tns3:awsId></tns3:item></tns3:itemList>\"}",

Expeceted output:

CorrelationId: 84e7dade-9919-47dc-a1f9-299137504c2e
APIName: iManageOrganisation
AuditCategory: AUDIT_EXIT
Timestamp: 2023-02-14T14:11:38.878Z
HostName: manageorganisationv2-rest-api-6d65f89ddc-cjrtk
SourceID: WO0000000012265
Payload:

fluentbit config:

apiVersion: v1
data:
  custom_parsers.conf: |
    [PARSER]
        Name tibco
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/imanageorganisation*.log
        Parser tibco
        multiline.parser docker, cri
        Tag imanageorganisation.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines Off
        Skip_Empty_Lines On
        Path_key FileName
        Parser_Firstline json
Azeem
  • 11,148
  • 4
  • 27
  • 40
Praveen Prakasan
  • 1
  • 1
  • Are you getting some specific error? Please include that in your question also. – Azeem Feb 21 '23 at 04:41
  • 1
    The log and the regex to parse are completely different. You can use https://regex101.com/ or https://fluentular.herokuapp.com/ to come up with the right regex that you need. – Azeem Feb 21 '23 at 04:54
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Feb 27 '23 at 13:55

0 Answers0