2

I'm trying to write a dataframe from spark to Elastic with a custom mapping id. and when I do that I'm getting the below error.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 14.0 failed 16 times, most recent failure: Lost task 0.15 in stage 14.0 (TID 860, ip-10-122-28-111.ec2.internal, executor 1): org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: [DataFrameFieldExtractor for field [[paraId]]] cannot extract value from entity [class java.lang.String] | instance

and below is the configuration used writing to ES.

var config= Map("es.nodes"->node,
 "es.port"->port,
 "es.clustername"->clustername,
 "es.net.http.auth.user" -> login,
 "es.net.http.auth.pass" -> password,
 "es.write.operation" -> "upsert",
 "es.mapping.id" -> "paraId",
 "es.resource" -> "test/type")

df.saveToEs(config)

I'm using the 5.6 version of ES and 2.2.0 of Spark. Let me know if you guys have any insight on this.

Thanks.!

knowledge_seeker
  • 362
  • 3
  • 20
  • 1
    Please make sure that `paraId` is present in each document you are sending to elasticsearch. It might be as simple as typo or a wrong case, like in [this question](https://stackoverflow.com/a/47944451/5095957). – Nikolay Vasiliev May 12 '18 at 11:55
  • Yeah @NikolayVasiliev, you are correct this was a typo and I debugged it and fixed it. Thanks for you response.! – knowledge_seeker May 14 '18 at 20:04
  • For those who had the same issue: look at the issue in the values for the column you put in config `es.write.operation`, `paraId` in this example. – Sarye Haddadi Oct 11 '22 at 13:36

0 Answers0