I'm currently working with Memgraph. I'm trying to figure out if there's a way to have a changes feed similar to what Couchbase offers. I'm interested in the possibility of streaming data from Memgraph out into Apache Kafka. Ideally, I would like to send updates from Memgraph to a Kafka topic every time there's a change in the data.
I know that I can stream data from Kafka, but can I stream it out to Kafka?
It is possible to stream out data changes with the help of triggers + custom procedure. So, on any change (create/update) you can set a trigger which will run a certain procedure (can be written in Python, C/C++, Rust). For what you mentioned, we have an example of changes being pushed to Kafka in Python procedure.
Here is the procedure to send to Kafka: https://github.com/memgraph/twitter-network-analysis/blob/main/memgraph/procedures/publisher.py
Here is how it was set up and called: https://github.com/memgraph/twitter-network-analysis/blob/main/backend/server/setup.py#L37-L57
Here is a tutorial on custom procedures: https://memgraph.com/docs/memgraph/tutorials/implement-custom-query-module-in-python And on stream processing with Kafka: https://memgraph.com/docs/memgraph/tutorials/graph-stream-processing-with-kafka Also, docs on triggers: https://memgraph.com/docs/memgraph/reference-guide/triggers/
