0

"non-serializable" error occurs when I follow flink document to write data via flink streaming. I use flink1.6,Elastic-Search-6.4 and flink-connector-elasticsearch6.
My code is like 

@Test
  public void testStringInsert() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
    env.enableCheckpointing(100); //
    DataStreamSource<String> input = env.fromCollection(Collections.singleton("testData"));
    List<HttpHost> httpHosts = new ArrayList<>();
    httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));
    ElasticsearchSink.Builder<String> esSinkBuilder = new ElasticsearchSink.Builder<>(
        httpHosts,
        new ElasticsearchSinkFunction<String> ()  {
          public IndexRequest createIndexRequest(String element) {
            Map<String, String> json = new HashMap<>();
            json.put("data", element);
            return Requests.indexRequest()
                .index("my-index")
                .type("my-type")
                .source(json);
          }
          @Override
          public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element));
          }
        }
    );
    esSinkBuilder.setBulkFlushMaxActions(1);
    input.addSink(esSinkBuilder.build());
    env.execute("test es string insert");
  }

When I run the code above,I got the exception 

java.lang.IllegalArgumentException: The implementation of the provided ElasticsearchSinkFunction is not serializable. The object probably contains or references non-serializable fields.

 at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139)
 at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.<init>(ElasticsearchSinkBase.java:216)
 at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:71)
 at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:60)
 at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink$Builder.build(ElasticsearchSink.java:208)
 at com.lianlianpay.erebus.erebusaccess.ElasticsearchSinkTest.testStringInsert(ElasticsearchSinkTest.java:151)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
 at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
 at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
 at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

I searched a lot and still puzzled.I did not pass Object to ElasticsearchSink,other than String,and String is serializable without a doubt.I really can not understand , what is wrong with my code or the develop enviroment ?

moyiguke
  • 70
  • 1
  • 10
  • What happens if you define `ElasticsearchSinkFunction` as a regular class instead of an anonymous class? – twalthr Oct 19 '18 at 15:24
  • Thank you for your advice, I implement the ElasticsearchSinkFunction and run the case.After doing this,no error occurs,but no data indexed to ElasticSearch.Some thing more strange, I set breakpoint in process function,then debug,I can not step into the code. – moyiguke Oct 20 '18 at 14:16
  • After some research, I successed to write data to ES with My own Implement of RichSinkFunction and [RestHighLevelClient](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-getting-started-initialization.html) – moyiguke Oct 20 '18 at 14:30
  • If you can reproduce the issue in a small application, it would be very helpful for the Flink community if you open a JIRA ticket for it. – twalthr Oct 22 '18 at 11:41
  • Thank you for your reply.I publish code to [github](https://github.com/moyiguket/flink-issue/blob/master/src/main/test/FlinkToESTest.java),and open an issue in [JIRA](https://issues.apache.org/jira/browse/FLINK-10660) – moyiguke Oct 24 '18 at 07:14
  • Hi, Any update on this one? – ashdnik Feb 05 '19 at 06:05

2 Answers2

0

I just met the same error message. the problem was with credentialsProvider which should be instantiated within the RestClientFactory

Example in Scala:

class SecuredRestClientFactory (username: String, password: String) extends RestClientFactory {
  override def configureRestClientBuilder(builder: RestClientBuilder): Unit = {
    builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
      override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
        val credentialsProvider = new BasicCredentialsProvider
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password))
        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
      }
    })
  }}
esSinkBuilder.setRestClientFactory(new SecuredRestClientFactory(elasticUser,elasticPass))
Harold
  • 455
  • 4
  • 10
-1

Flink Elasticsearch Connector 7

Please find a working and detailed answer which I have provided here, Which is written in Scala.

Keshav Lodhi
  • 2,641
  • 2
  • 17
  • 23