1

I have a JSON object with some of the fields like this:-

{
    "desc": "this is test description",
    "name": "some random name",
}

While indexing this document, I would like to change the field names and my document after indexing should look like this:-

{
    "description": "this is test description",
    "user_name": "some random name",
}

I've read about the Ingest pipeline processor but they only rename after a field is created. Is there any way that I could change the field names while indexing?

Revanth
  • 87
  • 2
  • 13

1 Answers1

2

The way to do this is to use a Pipeline. The general idea is you define the pipeline and give it a name on your cluster. Then you can reference it when indexing data and the data you send will be passed through that pipeline to transform it. Note pipelines will only run on nodes marked as "ingest" nodes.

https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html

To rename specifically, you can use this processor: https://www.elastic.co/guide/en/elasticsearch/reference/current/rename-processor.html

I didn't explicitly test this, but the code would be along the lines of:

Define the pipeline on your cluster with a name:

PUT _ingest/pipeline/my-pipeline-name
{
  "description" : "rename user name",
  "processors" : [
    {
      "rename" : {
        "field": "name",
        "target_field": "user_name"
      },
      "rename" : {
        "field": "field2",
        "target_field": "newfield2"
      }
    }
  ]
}

Load your doc, using the pipeline:

POST /some_index/_doc/?pipeline=my-pipeline-name
{
    "desc": "this is test description",
    "name": "some random name",
}
Ryan Widmaier
  • 7,948
  • 2
  • 30
  • 32
  • thank you, it does work. But how can I do this on multiple fields but not just on one single field. Do I have to create a pipeline for each and every field and add `pipeline=pipeline1&pipeline2&pipeline3` ? Or is there any other simple way to achieve multiple fields renaming? – Revanth Oct 18 '19 at 05:41
  • You can add multiple rename transforms to the processors section of the pipeline json. Updated the answer – Ryan Widmaier Oct 18 '19 at 11:30
  • I've tried it already but I keep getting this error `Duplicate key "rename"` – Revanth Oct 19 '19 at 04:32
  • You need to rewrap the rename json key/value in an object in the processors list. Notice in the example that processors is not an object, it is a list of objects – Ryan Widmaier Oct 19 '19 at 12:34
  • I've wrapped each rename object inside another object and it works now! Thank you! There was a little bit of confusion since the example doesn't wrap each rename object inside an object itself. – Revanth Oct 21 '19 at 05:47