The problem of connecting Apache Flink to elasticsearch

Question

I used a piece of code inside the Flink site to connect Apache Flink to Elastic Search. I want to run this piece of code from NetBeans software through maven project.

public class FlinkElasticCon {

 public static void main(String[] args) throws Exception {

    final int port;
    try {
        final ParameterTool params = ParameterTool.fromArgs(args);
        port = params.getInt("port");
    } catch (Exception e) {
        System.err.println("No port specified. Please run 'SocketWindowWordCount --port <port>'");
        return;
    }

    // get the execution environment
    final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    // get input data by connecting to the socket
    DataStream<String> text = env.socketTextStream("localhost", port, "\n");

    // parse the data, group it, window it, and aggregate the counts
    DataStream<WordWithCount> windowCounts = text

        .flatMap((String value, Collector<WordWithCount> out) -> {
            for (String word : value.split("\\s")) {
                out.collect(new WordWithCount(word, 1L));
            }
    })

       .keyBy("word")
        .timeWindow(Time.seconds(5))

        .reduce(new ReduceFunction<WordWithCount>() {
            @Override
            public WordWithCount reduce(WordWithCount a, WordWithCount b) {
                return new WordWithCount(a.word, a.count + b.count);
            }
        });

    // print the results with a single thread, rather than in parallel
    //windowCounts.print().setParallelism(1);

    env.execute("Socket Window WordCount");


    List<HttpHost> httpHosts = new ArrayList<>();
    httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));
    httpHosts.add(new HttpHost("10.2.3.1", 9200, "http"));       

    ElasticsearchSink.Builder<String> esSinkBuilder = new ElasticsearchSink.Builder<>(
        httpHosts,
        new ElasticsearchSinkFunction<String>() {
            public IndexRequest createIndexRequest(String element) {
                Map<String, String> json = new HashMap<>();
                json.put("data", element);

                return Requests
                        .indexRequest()
                        .index("my-index")
                        .type("my-type")
                        .source(json);
            }

            @Override
            public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
                indexer.add(createIndexRequest(element));
            }
        }
    );


    windowCounts.addSink((SinkFunction<WordWithCount>) esSinkBuilder);

}

     public static class WordWithCount {

    public String word;
    public long count;

    public WordWithCount() {}

    public WordWithCount(String word, long count) {
        this.word = word;
        this.count = count;
    }

    @Override
    public String toString() {
        return word + " : " + count;
    }
}
}

When adding Dependency, it does not identify the elasticsearchsink class. Given that I added different Dependency to it, but the problem is still not resolved. When importing :

import org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink

The red line is created as unknown in the code.

my pom:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema- 
instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven- 
4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.flink</groupId>
<artifactId>mavenproject1</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencies>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-elasticsearch6_2.11</artifactId>
        <version>1.8.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_2.11</artifactId>
        <version>1.8.1</version>
         <scope>provided</scope>
        <type>jar</type>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-elasticsearch-base_2.11</artifactId>
        <version>1.8.1</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>6.0.0-alpha1</version>
        <!--<version>6.0.0-alpha1</version>-->
        <type>jar</type>
    </dependency>




        <dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-java</artifactId>
    <version>1.8.1</version>
              <type>jar</type>
        </dependency>

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients_2.11</artifactId>
    <version>1.8.1</version>
</dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-core</artifactId>
        <version>0.8.1</version>
    </dependency>
</dependencies>



<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>

</properties>

apache flink version : 1.8.1 elasticsearch version : 7.4.2 netbeans version : 8.2 java version : 8

please help me.

Answer 1

Flink Elasticsearch Connector 7

Please find a working and detailed answer which I have provided here.