0

My problem is somewhat as described here. Part of Code (actually took from apache site) is as below

val httpHosts = new java.util.ArrayList[HttpHost]
httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"))
httpHosts.add(new HttpHost("10.2.3.1", 9200, "http"))

val esSinkBuilder = new ElasticsearchSink.Builder[String](
  httpHosts,
  new ElasticsearchSinkFunction[String] {
    def createIndexRequest(element: String): IndexRequest = {
      val json = new java.util.HashMap[String, String]
      json.put("data", element)


      return Requests.indexRequest()
              .index("my-index")
              .`type`("my-type")
              .source(json)

If I add these three statements, I am getting error as below

import org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction
import org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer
import org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink

Error I am getting

object elasticsearch is not a member of package org.apache.flink.streaming.connectors
object elasticsearch6 is not a member of package org.apache.flink.streaming.connectors

If I do not add those import statements, I get error as below

Compiling 1 Scala source to E:\sar\scala\practice\readstbdata\target\scala-2.11\classes ...
[error] E:\sar\scala\practice\readstbdata\src\main\scala\example\readcsv.scala:35:25: not found: value ElasticsearchSink
[error] val esSinkBuilder = new ElasticsearchSink.Builder[String](
[error]                         ^
[error] E:\sar\scala\practice\readstbdata\src\main\scala\example\readcsv.scala:37:7: not found: type ElasticsearchSinkFunction
[error]   new ElasticsearchSinkFunction[String] {
[error]       ^
[error] two errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 1 s, completed 10 Feb, 2020 2:15:04 PM

Stackflow question I referred above, some function has been extended. My understanding is, flink.streaming.connectors.elasticsearch have to be extended into REST libraries. 1) Is my understanding correct 2) if Yes, can I have complete extensions 3)If my understanding is wrong, please give me a solution.

Note: I added the following statements in build.sbt

libraryDependencies += "org.elasticsearch.client" % "elasticsearch-rest-high-level-client" % "7.5.2" ,
    libraryDependencies += "org.elasticsearch" % "elasticsearch" % "7.5.2",
Sarma
  • 11
  • 4

2 Answers2

0

The streaming connectors are not part of the flink binary distribution. You have to package them with your application.

For elasticsearch6 you need to add flink-connector-elasticsearch6_2.11, which you can do as

libraryDependencies += "org.apache.flink" %% "flink-connector-elasticsearch6" % "1.6.0"

Once this jar is part of your build, then the compiler will find the missing components. However, I don't know if this ES6 client will work with version 7.5.2.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • I added the below statement to my build.sbt. libraryDependencies += "org.apache.flink" % "flink-connector-elasticsearch6_2.11" % "1.9.0" % "provided" I am getting error as below object creation impossible, since method process in trait ElasticsearchSinkFunction of type (x$1: String, x$2: org.apache.flink.api.common.functions.RuntimeContext, x$3: org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer)Unit is not defined [error] new ElasticsearchSinkFunction[String] { [error] ^ – Sarma Feb 10 '20 at 09:30
  • I don't know why the Scala example in the flink docs for ES6 doesn't define a `process` method, but the Java version does. Sounds like maybe this method is needed. – David Anderson Feb 10 '20 at 09:54
0

Flink Elasticsearch Connector 7

Please look at the working and detailed answer which I have provided here, Which is written in Scala.

Keshav Lodhi
  • 2,641
  • 2
  • 17
  • 23