Spark 3.0 scala.None$ is not a valid external type for schema of string

Question

While using elasticsearch-hadoop library for reading elasticsearch index with empty attribute, getting the exception

Caused by: java.lang.RuntimeException: scala.None$ is not a valid external type for schema of string

There is open defect in github for the same with steps to reproduce it: https://github.com/elastic/elasticsearch-hadoop/issues/1635

Spark: 3.1.1
Elasticsearch-Hadoop : elasticsearch-spark-30_2.12-7.12.0
Elasticsearch : 2.3.4

score 3 · Accepted Answer · answered Apr 30 '21 at 05:45

It worked by setting elasticsearch-hadoop property es.field.read.empty.as.null = no

.option("es.field.read.empty.as.null", "no")

From Elasticsearch Link:
es.field.read.empty.as.null (default yes)
Whether elasticsearch-hadoop will treat empty fields as null.

1 Answers1