Is there a way on Elasticsearch ingest-pipelines to use convert processor on multiple fields based on field name suffix?

Question

We have an index with default pipeline who converts a field to boolean, like documented here https://www.elastic.co/guide/en/elasticsearch/reference/7.10/convert-processor.html We're using Elasticsearch v7.10.

Is there a way to create a pipeline to convert multiple fields based on field name suffix like '_b'? Something like:

PUT _ingest/pipeline/string-to-bool
{
  "description": "converts fields ending in '_b' to boolean",
  "processors" : [
    {
      "convert" : {
        "field": "*_b",
        "type": "boolean"
      }
    }
  ]
}

I don't think you can. It is going to match the exact name that is provided. — Paulo, May 24 '22 at 16:27
Is there an alternative to pipelines? Or maybe using a script processor? — Stefano Lazzaro, May 24 '22 at 16:48

score 1 · Accepted Answer · answered May 24 '22 at 17:38

Tldr;

As far as I know the convert processor does not allow fuzzy matching.

To workaround

As you proposed you could write a script doing the job.

PUT _ingest/pipeline/string-to-bool-script
{
  "processors" : [
    {
        "script": {
          "lang": "painless",
          "source": """
          for (key in ctx.keySet()) {
            if (key.endsWith("_b")) {
                ctx[key] = Boolean.parseBoolean(ctx[key]);
            }
          }
          """
        }
      }
  ]
}

POST /_ingest/pipeline/string-to-bool-script/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "plop_b": "true",
        "_b": "false"
      }
    }
  ]
}

Keep in mind this is very simplistic, and would only work for strings with value of true / false (not case sensitive).

It's quite easy to adapt this snippet to also accept 0 or 1. — Stefano Lazzaro, May 25 '22 at 07:06
Indeed, I was more thinking of the truthy / falsy you have in Js — Paulo, May 25 '22 at 07:10

Is there a way on Elasticsearch ingest-pipelines to use convert processor on multiple fields based on field name suffix?

1 Answers1

Tldr;

To workaround