Filebeat - Multiline configuration for log files containing JSON along text

Question

I have below log file as a sample and want to see JSON in one row in logz.io . I have used a couple of configurations. Lastly, I used the below configuration in Filebeat. However, I still see each line in the text file separately. I will be glad if you can guide me.

Filebeat Configuration

    filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /tmp/*.log
      multiline.type: pattern
      multiline.pattern: '%{TIMESTAMP_ISO8601}'
      multiline.negate: true
      multiline.match: after

Log File

2021-05-07T11:11:31.8298765+02:00 0HM8GHNH0KK9V:00000002 [INF] Request starting HTTP/1.1 GET http://11.11.11.11/swagger/index.html - - (ca22a1cb)
2021-05-07T11:11:31.8302819+02:00 0HM8GHNH0KK9V:00000002 [INF] Request finished HTTP/1.1 GET http://11.11.11.11/swagger/index.html - - - 200 - text/html;charset=utf-8 0.4099ms (791a596a)
2021-05-07T11:11:38.1371767+02:00 0HM8GHNH0KKA0:00000002 [DBG] ###DATA_REQUEST_888888###{
  "operation": "new_business",
  "policy": {
    "originalContractStartTimestamp": 1620165600000,
    "originalContractEndTimestamp": 1651701600000,
    "effectiveContractEndTimestamp": 1651701600000,
    "fees": [],
    "documents": [],
    "invoices": [],
    "productName": "personal-auto",
    "createdTimestamp": 1620377486896,
    "updatedTimestamp": 1620378697551,
    "cancellation": null,
    "depositAccount": null,
    "issuedTimestamp": null,
    "paymentScheduleName": "annually",
    "OriginalContractStart": "2021-05-04T22:00:00Z",
    "OriginalContractEnd": "2022-05-04T22:00:00Z",
    "EffectiveContractEnd": "2022-05-04T22:00:00Z",
    "Created": "2021-05-07T08:51:26.896Z",
    "Updated": "2021-05-07T09:11:37.551Z",
    "Issued": "0001-01-01T00:00:00"
  }
  "endorsementLocator": null
} (0596056d)
2021-05-07T11:11:38.2993754+02:00 0HM8GHNH0KKA0:00000002 [INF] Already stored pricing version is used, calculation date 05/07/2021 08:51:47 (79d888c3)
2021-05-07T11:11:38.3039358+02:00 0HM8GHNH0KKA0:00000002 [INF] Already stored pricing version is used, calculation date 05/07/2021 08:51:47 (79d888c3)
2021-05-07T11:11:38.3093491+02:00 0HM8GHNH0KKA0:00000002 [INF] Already stored pricing version is used, calculation date 05/07/2021 08:51:47 (79d888c3)
2021-05-07T11:11:38.3143367+02:00 0HM8GHNH0KKA0:00000002 [INF] Already stored pricing version is used, calculation date 05/07/2021 08:51:47 (79d888c3)

Logs from logz.io

score 1 · Answer 1 · edited May 10 '21 at 13:27

First I have changed the daemonset configuration. Set codec type to "plain".

 - name: LOGZIO_CODEC
   value: plain

Then I used the below multiline configuration in configmap. There are duplicates for codec type. Until I changed the daemonset codec type, the below configuration did not help. Now it is working as I expected. I can see JSON in one row.

    filebeat.inputs:
    - type: log
      logzio_codec: "plain"
      enabled: true
      paths:
        - /tmp/*.log
      multiline.type: pattern
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

    processors:
      - add_cloud_metadata: ~
    fields:
      logzio_codec: 'plain'
      token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
      cluster: ${CLUSTER_NAME}
      type: ${LOGZIO_TYPE}

Filebeat - Multiline configuration for log files containing JSON along text

1 Answers1