4

I am trying to add a new source to my fluentd to parse a JSON file. The JSON file looks like this

    {
        "local": {
            "cmd_|-start_cmd_2_|-echo 'Starting to run second time'_|-run": {
                "comment": "Command \"echo 'Starting to run second time'\" run",
                "name": "echo 'Starting to run second time'",
                "result": true,
                "duration": 11.144,
                "__sls__": "test",
                "changes": {
                    "pid": 26010,
                    "retcode": 0,
                    "stderr": "",
                    "stdout": "Starting to run second time"
                },
                "__id__": "start_cmd_2"
            },
        }
    }

This is the JSON output from salt stack execution. I have added this source

    <source>
      @type tail
      tag salt-new
      path /var/log/salt_new.json
      pos_file /tmp/fluentd/new.pos
      <parse>
        @type json
      </parse>
      refresh_interval 10s
    </source>

I tried few variations such as using 'format json' and it does not work. It always throws

    2020-03-19 18:05:13 +0000 [info]: #0 following tail of /var/log/salt_new.json
    2020-03-19 18:05:13 +0000 [warn]: #0 pattern not matched: "{"
    2020-03-19 18:05:13 +0000 [warn]: #0 pattern not matched: "    \"local\": {"
    2020-03-19 18:05:13 +0000 [warn]: #0 pattern not matched: "        \"cmd_|-start_cmd_2_|-echo 'Starting to run second time'_|-run\": {"
    .
    .

I do not want to use regex parsing, I am trying to get JSON parser plugin work.

Any suggestions will be super helpful. Thanks

Magic
  • 505
  • 2
  • 6
  • 19
  • did you solved this problem? I got similar one. – Ray Sep 24 '21 at 08:21
  • The issue is that fluentd tries to match each line as a JSON but your JSON output is split over multiple files. I'm not sure if you can configure it in fluentd but if you can make the producer output each JSON in a single line, it will definitely solve the problem. – Gram Jan 07 '22 at 08:50

0 Answers0