1

Need help on defining pipeline on index / type.

I have tried the ingest pipeline on the document for given index and type.

Actual JSON :

{
    "geopoint": {
        "lon": 48.845877,
        "lat": 8.821861,
        "alt": 0.0
    }
}

I have defined a pipeline like below

PUT _ingest/pipeline/geo-pipeline
{
  "description" : "remove unsupported altitude field",
  "processors" : [
    {
      "remove" : {
        "field": "geopoint.alt",
        "ignore_failure" : true
      }
    }
  ]
}

This works fine when I tried to add manually a single data like this 

PUT index1/type1/1234?pipeline=geo-pipeline
{
    "geopoint": {
        "lon": 48.845877,
        "lat": 8.821861,
        "alt": 0.0
    }
}

I use mongo-connector which automatically index all data from MongoDB to ES. Initially while defining mappings I used to write script to remove the data. Since I have upgraded my ES version to 5.4.0 while indexing I want this pipeline to be executed. Like if I have 1000 docs and this pipeline has to be executed when mongo-connector syncs data to ES, how do I do that?

THOUGHT?

Cant we define pipeline during mappings? Some what like this

PUT index1/_mapping/geo?pipeline=geopipeline
{
  "properties": {
     "geopoint": {
        "type": "geo_point"
     }
  }
}
Shreyas Rao B
  • 193
  • 4
  • 17
  • This answer might help: https://stackoverflow.com/questions/36772351/does-mongo-connector-supports-adding-fields-before-inserting-to-elasticsearch Also have a look at this issue: https://github.com/mongodb-labs/mongo-connector/issues/607 – Val May 30 '17 at 04:31
  • @Val ES 5.4.0 doc says you can use pipeline on index? how to achieve that? The above pipeline I defined is applied on the single document. – Shreyas Rao B May 30 '17 at 05:38
  • @Val Like in ES 2.4.X we had _transform_ were in script was written to remove the `alt` . `alt` field will still be visible in the stored _source but it will not be indexed. But the in ES 5.x, remove pipeline would remove that field while indexing the document. We dont want that to happen – Shreyas Rao B May 31 '17 at 07:15
  • @Val The current problem we are facing is we migrated from 2.4.x to 5.x. So `alt` field was taken care by _trasform_ (was defined while creating mappings) which used to retain the field and geo_quries were woking fine. But in ES 5.4.0 we don't know how to achieve that. – Shreyas Rao B May 31 '17 at 07:24

0 Answers0