How does Memgraph work with Kafka streams?

Question

Does Memgraph act as a sink for streaming sources from Kafka, and then once the messages have been received, organize them into a graph database? If yes, how are these messages being organized. I don't get how the messages can be transformed from the format they are in a certain topic to the something that is understandable to the graph database such as Memgraph.

Answer 1

Memgraph is consuming messages from Kafka with its internal consumer. This means that you need to create a stream inside Memgraph and define from which topic data will be read. Before doing that, you have to create and load a transformation module for a certain stream. Transformation module is a procedure that tells Memgraph how to transform Kafka messages to Cypher queries which will be executed for each consumed message. Once you create a stream, you have to start a stream to start ingesting data from the stream. Because of the transformation module, the data from your Kafka topic will seamlessly be stored into the graph database.