How can I add fields dynamically in Filebeats via the command line tool?

Question

I am trying to add two dynamic fields in Filebeats by calling the command via Python. The fields themselves are populated after some processing is done so I cannot pre-populate it in a .yml file. Otherwise I know I can have the following code in my .yml file:

.
.
processors:
  - add_fields:
      fields:
        field1: "str1"
        field2: "str2"
.
.

Since I cannot change field1 or field2 to the actual values with this file. I have built a command that looks like:

filebeat -c /etc/filebeat/config.yml --once --path.data=UUID -E "processors=[{{add_fields: {{fields: [ field1:['foo'], field2:['bar']]}}]"

When I run this command I get the following error: Error: invalid argument "processors=[{add_fields: {fields: [ field1:['foo'], field2:['bar']]}]" for "-E, --E" flag: dictionary expected ',' or '}' when parsing '[{add_fields: {fields: [ field1:['foo'], field2:['bar']]}]'

How can I dynamically add fields to Filebeat?

score 1 · Accepted Answer · answered Nov 27 '20 at 01:41

You can use it like this:

filebeat -c /etc/filebeat/config.yml --once --path.data=UUID -E \ 
  processors.0.add_fields.fields={field1:foo} -E processors.0.add_fields.fields={field2:bar}

Result:

{
  "@timestamp": "2020-11-27T01:33:31.124Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.0"
  },
  "fields": {
    "field2": "bar",
    "field1": "foo"
  },
(...)
}

There might be a way to hack it to pass multiple fields in one -E but it should be easier for you to generate this sort of arguments for filebeat within your python script with this very same way I've provided.

Another way is to overload filebeat with two -c config.yml -c config_dynamic.yml, where the config_dynamic.yml is generated in run-time before your call filebeat. The content should only have the processor definition. Filebeat will merge both configuration files and it should work.

Example of filebeat_dynamic.yml:

processors:
  - add_fields:
      fields:
        field3: boo

By running:

filebeat -c /etc/filebeat/config.yml -c /tmp/filebeat_dynamic.yml --once --path.data=UUID -E \ 
  processors.0.add_fields.fields={field1:foo} -E processors.0.add_fields.fields={field2:bar}

You would get:

{
  "@timestamp": "2020-11-27T01:38:36.184Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.0"
  },
  "fields": {
    "field2": "bar",
    "field1": "foo",
    "field3": "boo"
  },
(...)
}

How can I add fields dynamically in Filebeats via the command line tool?

1 Answers1