2

I'm trying to extract some data from elasticsearch with pyspark. I want to extract only few fields (not all) from the documents. So, I'm making a post request from the software "Postman" (for testing purpose) with following url and body.It's giving perfect output as expected. But when I'm using same body with spark code, it's extracting all the fields from the specified documents which is not desired. Can anyone tell what might be the reason for such weird behavior ? Thanks in advance !

Spark version 2.3, Elasticsearch version 6.2, postman body type = application/json

This is what I'm doing with postman : 

`url : localhost:9200/test-index4/school/_search`

`body : 
{
    "query":
     {
         "ids":
           {
               "values":["8","9","10"]
           }
     },
     "_source":
     {
         "includes":["name"]
     }
}`

Below is what I'm doing with pyspark : 

`body = "{"query":{"ids":{"values":["8","9","10"]}},"_source":{"includes":["name"]}}"
df = self.__sql_context.read.format("org.elasticsearch.spark.sql") \
            .option("es.nodes", "localhost") \
            .option("es.port", "9200") \
            .option("es.query", body) \
            .option("es.resource", "test-index4/school") \
            .option("es.read.metadata", "true") \
            .option("es.read.metadata.version", "true") \
            .option("es.read.field.as.array.include", "true") \
            .load()

`

amol_shaligram
  • 226
  • 1
  • 11
  • [This](https://stackoverflow.com/a/43773267/3433323) might be what you need. The `es.read.field.include` config. – mkaran May 10 '18 at 09:38
  • It worked ! Thanks a lot @mkaran. But still do you know reason for such weired behavior ? Also `es.read.metadata.version` doesn't read version. Can you please explain why ? – amol_shaligram May 10 '18 at 09:43
  • I'm glad it worked! Well, by default, elasticsearch will bring you all the fields, from the [docs](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html): if `es.read.field.include` is not set, then it defaults to null and all fields are returned. As for the `metadata.version` I don't know, since you've set both `es.read.metadata` and `es.read.metadata.version` to true which is the correct configuration from what I can tell. – mkaran May 10 '18 at 09:53
  • Alright, thanks @mkaran ! – amol_shaligram May 10 '18 at 10:21

1 Answers1

4

Try setting es.read.field.include in config with value as comma seperated field list. e.g. "es.read.field.include","field1,field2,..."

priyanshi
  • 81
  • 6