Benthos grok log parse

Question

So I have this log and I was trying to parse it using benthos grok. What I need to do is return json 5 elements: • Timestamp • Connection direction (inbound/outbound) • Source IP • Destination IP • Source Port in json format of this log:

<134>1 2023-01-21T17:18:05Z CHKPGWMGMT CheckPoint 16575 - [action:"Accept"; flags:"411908"; ifdir:"outbound"; ifname:"eth0"; logid:"0"; loguid:"{0x80c5f24,0x273f572f,0x1a6c6aae,0x5f835b6e}"; origin:"10.21.10.2"; originsicname:"cn=cp_mgmt,o=CHKPGWMGMT..f6b99b"; sequencenum:"4"; time:"1674314285"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={F7CAC520-C428-484E-8004-06A1FAC151A3};mgmt=CHKPGWMGMT;date=1667399823;policy_name=Standard]"; dst:"10.21.10.2"; inzone:"Local"; layer_name:"Network"; layer_uuid:"8a994dd3-993e-4c0c-92a1-a8630b153f4c"; match_id:"1"; parent_rule:"0"; rule_action:"Accept"; rule_uid:"102f52bf-da21-49cd-b2e2-6affe347215d"; outzone:"Local"; product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"46540"; service:"1433"; service_id:"https"; src:"10.21.9.1"]

input:
  type: file
  file:
    paths: [./intput.txt]
    codec: lines

pipeline:
  processors:
    - grok:
        expressions:
          - '%{NGFWLOGFILE}'
        pattern_definitions:
            NGFWLOGFILE: '%{NOTSPACE:interfaceid} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:Letters} %{NOTSPACE:Mhm} %{NOTSPACE:Skaicius} %{NOTSPACE:AA} %{NOTSPACE:Action}'
#    - mapping: |
#       root.timestamp = this.timestamp
#        root.Action = this.Action

output:
  stdout: {}
#output:
 # label: ""
 # file:
 #   path: "Output.txt"
 #  codec: lines

So I tried using grok to parse the log to json format and mapping to filter the part I want to get. The part there I got stuck is pattern_definitions how to extract data from the list which already has names at the log file or I should use some better approach to the task?

Mihai Todor · Answer 1 · 2023-01-30T23:39:23.407

Grok translates to a regular expression under the covers, so I don't think it has any notion of lists and such. Try this:

input:
  generate:
    count: 1
    interval: 0s
    mapping: |
      root = """<134>1 2023-01-21T17:18:05Z CHKPGWMGMT CheckPoint 16575 - [action:"Accept"; flags:"411908"; ifdir:"outbound"; ifname:"eth0"; logid:"0"; loguid:"{0x80c5f24,0x273f572f,0x1a6c6aae,0x5f835b6e}"; origin:"10.21.10.2"; originsicname:"cn=cp_mgmt,o=CHKPGWMGMT..f6b99b"; sequencenum:"4"; time:"1674314285"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={F7CAC520-C428-484E-8004-06A1FAC151A3};mgmt=CHKPGWMGMT;date=1667399823;policy_name=Standard]"; dst:"10.21.10.2"; inzone:"Local"; layer_name:"Network"; layer_uuid:"8a994dd3-993e-4c0c-92a1-a8630b153f4c"; match_id:"1"; parent_rule:"0"; rule_action:"Accept"; rule_uid:"102f52bf-da21-49cd-b2e2-6affe347215d"; outzone:"Local"; product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"46540"; service:"1433"; service_id:"https"; src:"10.21.9.1"]"""

pipeline:
  processors:
    - grok:
        expressions:
          - "%{NGFWLOGFILE}"
        pattern_definitions:
          NGFWLOGFILE: |-
            %{NOTSPACE:interfaceid} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:Letters} %{NOTSPACE:Mhm} %{NOTSPACE:Skaicius} %{NOTSPACE:AA} \[%{GREEDYDATA}; ifdir:"%{DATA:connectionDirection}"; %{GREEDYDATA}; dst:"%{DATA:destinationIP}"; %{GREEDYDATA}; s_port:"%{DATA:sourcePort}"; %{GREEDYDATA}; src:"%{DATA:sourceIP}"\]

output:
  stdout: {}

Benthos grok log parse

1 Answers1