1

I have a spark dataframe which has a column say "name". The name could have different values in a single dataframe. When I write my data to elasticsearch using spark (scala), I want to write the data to different indexes based on the value of the column "name". 

dataframe.saveToEs("index-name")

The saveToEs expects a string, I am looking for something on the lines of : 

dataframe.saveToEs(col(""))

or something similar where I can assign the value during write time.

mythic
  • 535
  • 7
  • 21

1 Answers1

0

Mythic,

I just saw in the documentation you can use something like this :

rdd.saveToEs("my-collection-{media_type}/doc")

which allows you to :

Save each object based on its resource pattern, in this example based on media_type. For each document/object about to be written, elasticsearch-hadoop will extract the media_type field and use its value to determine the target resource.

Source : https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-write-dyn-scala

baitmbarek
  • 2,440
  • 4
  • 18
  • 26