Is there any way where we can generate output file based on the input data in benthos?

Question

For example: Input Data:

{"date":"03-11-22", "message":"This is message"},
{"date":"03-30-22", "message":"This is message"},
{"date":"04-03-22", "message":"This is message"},
{"date":"04-15-22", "message":"This is message"},
{"date":"08-18-22", "message":"This is message"},
{"date":"08-28-22", "message":"This is message"}

The output should generate the file name according to the month and push the data in that month's file.

Output: Given input should create 3 files,

032022_data.log
042022_data.log
082022_data.log

Mihai Todor · Accepted Answer · 2022-08-30T18:27:39.923

The path field of the file output supports interpolation. Please try the following config:

input:
  generate:
    mapping: |
      root = [
               {"date":"03-11-22", "message":"This is message"},
               {"date":"03-30-22", "message":"This is message"},
               {"date":"04-03-22", "message":"This is message"},
               {"date":"04-15-22", "message":"This is message"},
               {"date":"08-18-22", "message":"This is message"},
               {"date":"08-28-22", "message":"This is message"}
             ]
    count: 1
    interval: 0s
  processors:
    - unarchive:
        format: json_array

output:
  file:
    path: ${! json("date").replace_all("-", "") }_data.log

It produces the following files:

031122_data.log
033022_data.log
040322_data.log
041522_data.log
081822_data.log
082822_data.log

Update: Based on the comments, I believe this pipeline should do what you need:

input:
  generate:
    mapping: |
      root = [
               {"date":"03-11-22", "message":"This is message"},
               {"date":"03-30-22", "message":"This is message"},
               {"date":"04-03-22", "message":"This is message"},
               {"date":"04-15-22", "message":"This is message"},
               {"date":"08-18-22", "message":"This is message"},
               {"date":"08-28-22", "message":"This is message"}
             ]
    count: 1
    interval: 0s
  processors:
    - unarchive:
        format: json_array
    - mapping: |
        meta month = this.date.re_replace_all("-.*-","")
    - group_by_value:
        value: ${! meta("month") }
    - select_parts:
        parts:
          - 1

output:
  file:
    path: ${! meta("month") }_data.log

It creates these files:

0322_data.log
0422_data.log
0822_data.log

I'm not really sure where that 20 in between the month and the year is supposed to come from, so I left it out.

Thanks for the help, but Is there any way there we can generate month-wise data and dump all the data according to months in the files? In the output, It should only generate 3 files for 3 months. 032022_data.log 042022_data.log 082022_data.log please let me know If it is possible. — Yash Chauhan, Aug 30 '22 at 11:07

Is there any way where we can generate output file based on the input data in benthos?

1 Answers1