1

I am trying to run an ingest pipeline to replace instances of "on" and off" to true and false in an array.

This works perfectly with normal strings eg with data like this

[{onoffboolean: "on"}]

I am able to process this with the following:

processors: [
  {
    set: {
      field: 'onoffboolean',
      description: 'String represented trues to native true',
      if: "ctx?.onoffboolean == 'on'",
      value: true
    }
  },
  {
    set: {
      field: 'onoffboolean',
      description: 'String represented falses to native true',
      if: "ctx?.onoffboolean == 'off'",
      value: false
    }
  },
],

However when its an array of values eg:

["on", "on", "off"] to process into [true, true, false]

I am unable to get the right processor to handle this. I have attempted to use foreach but it seems the "_ingest._value" is not available when using an "if" conditional.

This elastic forum thread suggests using a painless script instead

https://discuss.elastic.co/t/foreach-ingest-processor-conditional-append-processor/216884/2

However I don't have enough of an understanding of painless scripting to work this out.

AndyJamesN
  • 468
  • 4
  • 14

1 Answers1

3

If you have a concrete array field (let's call it list_of_attributes), you can use the following script processor:

PUT _ingest/pipeline/bool_converter
{
  "description": "Trims and lowercases all string values",
  "processors": [
    {
      "script": {
        "source": """
          ctx.list_of_attributes = ctx.list_of_attributes.stream()
                                                         .map(str -> str == 'on' ? true : false)
                                                         .collect(Collectors.toList()) 
        """
      }
    }
  ]
}

and then apply it when you ingest your docs:

POST your-index/_doc?pipeline=bool_converter
{
  "list_of_attributes": ["on", "on", "off"]
}

If you have more than one such array field, you can iterate on the document's fields by adapting my answer to the question Run Elasticsearch processor on all the fields of a document.

Shameless plug: I dedicated a whole chapter to ingesting & pipelines in my recently released Elasticsearch Handbook. If you're new to ES, give it a shot!

Joe - GMapsBook.com
  • 15,787
  • 4
  • 23
  • 68
  • This works perfectly! I have other similar scenarios I can adapt this for. One alteration I am curious about `.map(str -> str == 'on' ? true : false)` how would I modify this to explicitly only return true if str == 'on', as in not using a ternary true false?. Your Elasticsearch Handbook looks fantastic! – AndyJamesN Apr 08 '21 at 21:57
  • Another way to describe it would be. Let's say I only wanted to change the values of 'on' to true when it matches and leave 'off' unchanged so it would be `[true, "off", "off"]` – AndyJamesN Apr 08 '21 at 22:35
  • Nice! Well, you could adjust the ternary operator to `str -> str == 'on' ? true : str`. But note that "When adding a field dynamically, the first value in the array determines the field type. All subsequent values must be of the same data type or it must at least be possible to coerce subsequent values to the same data type." [[source](https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html)] – Joe - GMapsBook.com Apr 09 '21 at 07:46