Elasticsearch Spark, how to query multiple times?

Question

I'm on jupyter notebook.

I'd like to use query dsl to prepare initial Dataframe.

I use conf.set("es.query", dsl_query) for that. (https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#_querying)

But then, I want to apply different query to prepare another Dataframe, and I can't find a way to apply a new dsl_query without creating new SparkContext

But I didn't find a way to recreate a SparkContext inside jupyter environment either.

I want to run analysis using QueryDSL-1 as baseline then run another anlysis using QueryDSL-2 as another baseline

Is there a way of doing this without creating two notebook?

you can specify the es.query as an option to your DataFrameReader i.e `spark.read.option("es.query", dsl_query).option("...", "...")` ... — eliasah, Feb 13 '19 at 15:22

score 0 · Accepted Answer · answered Nov 25 '19 at 10:19

0

You'll just need to specify the es.query as an option to your DataFrameReader i.e :

spark.read.option("es.query", dsl_query).option("...", "...")

answered Nov 25 '19 at 10:19

eliasah

39,588
11
124
154

Elasticsearch Spark, how to query multiple times?

1 Answers1