0

trying with both the dataframe Api and the rdd API

val map =collection.mutable.Map[String, String]()
map("es.nodes.wan.only") = "true"
map("es.port") = "reducted"
map("es.net.http.auth.user") = "reducted"
map("es.net.http.auth.pass") = "reducted"
map("es.net.ssl") = "true"
map("es.mapping.date.rich") = "false"
map("es.read.field.include") = "data_scope_id"

map("es.nodes") = "reducted"

val rdd = sc.esRDD("index name", map)
rdd.take(1)

But anything I try I get this error

EsHadoopIllegalArgumentException: invalid map received dynamic=strict

I've tried limiting the fields being read with es.read.field.include But even if I choose one field which I'm sure doesn't have any varient I still get this error

How can I work around this? I'll be glad for any advice

Versions:

  • eshadoop-7.13.4
  • client Spark 3.1.2
  • Scala 2.12

Clarification

This is about reading from elasticsearch in spark, not indexing

alonisser
  • 11,542
  • 21
  • 85
  • 139
  • Hello @Alonisser, could you explain in plain word what your are trying to achieve ? From my understanding you are trying to index a document right ? Could you update your question to make your objectives clearer ? – Paulo Jan 03 '22 at 09:53
  • Nope, it's about READING from elastic search index from spark – alonisser Jan 04 '22 at 15:45

1 Answers1

0

So if I understand correctly, your aim is to index the values in map to index name.

TLDR;

Update the mapping of your index to allow for new fields to be indexed. As of new the value of dynamic is strict which does not allow for new field and throw an exception.

PUT /index name/
{
  "mappings": {
    "dynamic": true
  }
}

To understand

The issue is with the mapping of your index. There is a setting called [dynamic(https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html) on the mapping of your index.

I bet it is set to strict, which according to the doc:

If new fields are detected, an exception is thrown and the document is rejected. New fields must be explicitly added to the mapping.

So, my understanding is, you have one or many fields that are new in your document.

Either:

  • Fix the document
  • Fix the mapping
  • Switch dynamic to true, false or runtime according to your needs
Paulo
  • 8,690
  • 5
  • 20
  • 34
  • 1
    Actually, I think the OP is simply reading. See [`esRDD()`](https://javadoc.io/static/org.elasticsearch/elasticsearch-spark-20_2.11/7.13.4/org/elasticsearch/spark/rdd/EsSpark.html#esRDD-org.apache.spark.SparkContext-scala.collection.Map-) – blackbishop Jan 03 '22 at 11:51
  • Indeed, hmmm – Paulo Jan 03 '22 at 14:15