0

I've started the Kafka stream from mgconsole using this code:

CREATE KAFKA STREAM music_stream
TOPICS music-stars
TRANSFORM music_books.music_stars
CONSUMER_GROUP randomseed141234
BOOTSTRAP_SERVERS 'localhost:9093'
CREDENTIALS {'sasl.username':'public',
                'sasl.password':'public',
                'security.protocol':'SASL_PLAINTEXT',
                'sasl.mechanism':'PLAIN'};

I want to listen all of the messages from beginning. How do I do that?

KWriter
  • 1,024
  • 4
  • 22

2 Answers2

1

There are many possibilites to consume a kafka topic. If you just want to take a look for debugging purposes I suggest using a tool like this vscode-extension where you can just input the connection info.

If you want to consume programmatically, there are many connectors for Kafka, for example this nice npm-package.

But as far as I understand, you would use Memgraph to consume streaming data into your databases. So there is no need to consume the data from the stream - Memgraph does the consuming and you can query the db as usual. Take a look at memgraphs-docs for stream-processing with kafka if you haven't already.

jbuese
  • 26
  • 4
1

When using a Kafka stream, you can manually set the offset of the next consumed message with a call to the query procedure mg.kafka_set_stream_offset:

CALL mg.kafka_set_stream_offset(stream_name, offset)

An offset of -1 denotes the start of the stream, i.e., the beginning offset available for the given topic/partition.

An offset of -2 denotes the end of the stream, i.e., for each topic/partition, its logical end such that only the next produced message will be consumed.

So the exact code for your case should be:

CALL mg.kafka_set_stream_offset(music_stream, -1)
KWriter
  • 1,024
  • 4
  • 22