RestHighLevelClient Not Found in Scala

Question

I am trying to insert into ElasticSearch(ES) in a Scala Program.

In build.sbt I have added

libraryDependencies += "org.elasticsearch.client" % "elasticsearch-rest-high-level-client" % "7.5.2" ,
libraryDependencies += "org.elasticsearch" % "elasticsearch" % "7.5.2"

My code is

val client = new RestHighLevelClient( RestClient.builder(new HttpHost("localhost", 9200, "http")))

While compiling I am getting errors as below

not found: type RestHighLevelClient
not found: value RestClient

Am I missing some import? My goal is to get a stream from Flink and insert into ElasticSearch Any help is greatly appreciated.

David Anderson · Answer 1 · 2020-02-05T10:44:36.117

0

To use Elasticsearch with Flink it's going to be easier if you use Flink's ElasticsearchSink, rather than working with RestHighLevelClient directly. However, a version of that sink for Elasticsearch 7.x is coming in Flink 1.10, which hasn't been released yet (it's coming very soon; RC1 is already out).

Using this connector requires an extra dependency, such as flink-connector-elasticsearch6_2.11 (or flink-connector-elasticsearch7_2.11, coming with Flink 1.10).

See the docs on using Elasticsearch with Flink.

The reason to prefer Flink's sink over using RestHighLevelClient yourself is that the Flink sink makes bulk requests, handles errors and retries, and it's tied in with Flink's checkpointing mechanism, so it's able to guarantee that nothing is lost if something fails.

As for your actual question, maybe you need to add

libraryDependencies += "org.elasticsearch.client" % "elasticsearch-rest-client" % "7.5.2"

edited Feb 05 '20 at 10:44

answered Feb 05 '20 at 10:06

David Anderson

39,434
4
33
60

I saw the above URL, before posting the question here. It looked complicated, a simple example would have been good. And going by your reply, it is not yet ready On the other hand RestHighLevelClient is simple to use – Sarma Feb 05 '20 at 10:20
I've expanded my answer. – David Anderson Feb 05 '20 at 10:45
I tried to put the code from the above URL and tried to compile. I got error as below identifier expected but 'type' found. [error] .type("my-type") [error] ^ – Sarma Feb 05 '20 at 10:50
Even after adding dependency, I am still getting the same error – Sarma Feb 05 '20 at 10:57
Perhaps you are missing `import org.elasticsearch.action.index.IndexRequest;` – David Anderson Feb 05 '20 at 11:02
For a more straightforward experience with a simpler API, you might prefer using the Table API and its ES connector. https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connect.html#elasticsearch-connector – David Anderson Feb 05 '20 at 11:03
After adding the two statements, compilation error disappeared import org.elasticsearch.client.RestClient import org.elasticsearch.client.RestHighLevelClient But execution failed with the error message java.lang.NoClassDefFoundError: org/elasticsearch/client/RestHighLevelClient I am working with your suggestion too – Sarma Feb 05 '20 at 11:15

score 0 · Answer 2 · answered May 07 '20 at 09:11

We don't need to use these dependencies separately to insert data in Elasticsearch by using Flink Streaming.

libraryDependencies += "org.elasticsearch.client" % "elasticsearch-rest-high-level-client" % "7.5.2" ,
libraryDependencies += "org.elasticsearch" % "elasticsearch" % "7.5.2"

Just use this flink-connector-elasticsearch7 or flink-connector-elasticsearch6

libraryDependencies += "org.apache.flink" %% "flink-connector-elasticsearch7" % "1.10.0"

All dependencies of Elasticsearch come along with the Flink-Elastic connector. So we don't need to include them separately in build.sbt file.

build.sbt file for Flink Elasticsearch

name := "flink-streaming-demo"

scalaVersion := "2.12.11"

val flinkVersion = "1.10.0"

libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-elasticsearch7" % flinkVersion

For more details please go through this working Flink-Elasticsearch code which I have provided here.

Note: Since Elastic 6.x onwards they started full support of the REST elastic client. And till Elastic5.x they were using Transport elastic client.

RestHighLevelClient Not Found in Scala

2 Answers2