Reading latest data from Kafka broker in Apache Flink

Question

I want to receive the latest data from Kafka to Flink Program, but Flink is reading the historical data. I have set auto.offset.reset to latest as shown below, but it did not work

properties.setProperty("auto.offset.reset", "latest");

Flink Programm is receiving the data from Kafka using below code

//getting stream from Kafka and giving it assignTimestampsAndWatermarks

        DataStream<JoinedStreamEvent> raw_stream = envrionment.addSource(new FlinkKafkaConsumer09<JoinedStreamEvent>("test",
                new JoinSchema(), properties)).assignTimestampsAndWatermarks(new IngestionTimeExtractor<>());

I was following the discussion on https://issues.apache.org/jira/browse/FLINK-4280 , which suggests adding the source in below mentioned way

Properties props = new Properties();
...

FlinkKafkaConsumer kafka = new FlinkKafkaConsumer("topic", schema, props);
kafka.setStartFromEarliest();
kafka.setStartFromLatest();
kafka.setEnableCommitOffsets(boolean); // if true, commits on checkpoint if checkpointing is enabled, otherwise, periodically.
kafka.setForwardMetrics(boolean);
...

env.addSource(kafka)

I did the same, however, I was not able to access the setStartFromLatest()

 FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09<JoinedStreamEvent>( "test", new JoinSchema(),properties);

What should I do to receive the latest values that are being sent to Kafka rather than receiving values from history?

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

0

The problem was solved by creating the new group id named test1 both for the sender and consumer and keeping the topic name same as test.

Now I am wondering, is this the best way to solve this issue? because every time I need to give a new group id

Is there some way I can just read data that is being sent to Kafka?

edited Jun 20 '20 at 09:12

Community

1
1

answered Aug 23 '17 at 18:43

Amarjit Dhillon

2,718
3
23
62

score 0 · Answer 2 · answered Aug 28 '17 at 07:38

I believe this could work for you. It did for me. Modify the properties and your kafka topic.

public static void main(String[] args) throws Exception {


        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();


        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "ip:port");
        properties.setProperty("zookeeper.connect", "ip:port");
        properties.setProperty("group.id", "your-group-id");


        DataStream<String> stream = env
                .addSource(new FlinkKafkaConsumer09<>("your-topic", new SimpleStringSchema(), properties));



        stream.writeAsText("your-path", FileSystem.WriteMode.OVERWRITE)
            .setParallelism(1);

        env.execute();

    }

Reading latest data from Kafka broker in Apache Flink

2 Answers2