How Apacheflink's DataStream API supporting batch processing of events

Question

My producer is apache Kafka and we want to listen batch of events to process them and write that processed events into the database. If I use stream/batch every event will hit one query to DB. I don't want to hit every event as one query. How can I batch some of the events and write this bulk data into DB?

Note: We are using DataStream API

In what sense are you processing batches? And please clarify your concern about "every event will hit one query to the DB". What DB are you using? Most sinks do bulk inserts. — David Anderson, Apr 28 '21 at 13:55
I'm using Neo4j DB. Can you suggest that are there any sink implementations for Neo4j? — Nageswara Rao Rayala, Apr 29 '21 at 14:59

score 0 · Answer 1 · answered Apr 29 '21 at 21:12

No, there isn't an official Neo4j sink for Flink. If your goal is to implement exactly once end-to-end by doing buffered, batched transactional updates, you might start by reading An Overview of End-to-End Exactly-Once Processing in Apache Flink, and then reach out to the flink user mailing list for further guidance.

How Apacheflink's DataStream API supporting batch processing of events

1 Answers1