How to print KStream to console?

Question

I have created a Kafka Topic and pushed a message to it.

So

bin/kafka-console-consumer --bootstrap-server abc.xyz.com:9092 --topic myTopic --from-beginning --property print.key=true --property key.separator="-"

prints

key1-customer1

on the command line.

I want to create a Kafka Stream out of this topic and want to print this key1-customer1 on the console.

I wrote the following for it:

final Properties streamsConfiguration = new Properties();

streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "app-id");
streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG, "client-id");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "abc.xyz.com:9092");

streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());

// Records should be flushed every 10 seconds. This is less than the default
// in order to keep this example interactive.
streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10 * 1000);
// For illustrative purposes we disable record caches
streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);

final StreamsBuilder builder = new StreamsBuilder();
final KStream<String, String> customerStream = builder.stream("myTopic");
customerStream.foreach(new ForeachAction<String, String>() {
    public void apply(String key, String value) {
        System.out.println(key + ": " + value);
    }
});

final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);

streams.start();   

Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

This does not fail. However, this does not print anything on the console as this answer suggests.

I am new to Kafka. So any suggestions to make this work would help me a lot.

score 8 · Accepted Answer · answered Dec 21 '17 at 12:06

8

TL;DR Use Printed.

import org.apache.kafka.streams.kstream.Printed
val sysout = Printed
  .toSysOut[String, String]
  .withLabel("customerStream")
customerStream.print(sysout)

answered Dec 21 '17 at 12:06

Jacek Laskowski

72,696
27
242
420

score 0 · Answer 2 · answered Dec 21 '17 at 08:15

I would try unsetting CLIENT_ID_CONFIG and leaving only APPLICATION_ID_CONFIG. Kafka Streams uses the application ID to set the client ID.

I would also verify the offsets for the consumer group ID that your Kafka Streams application is using (this consumer group ID is also based on your application ID). Use the the kafka-consumer-groups.sh tool. It could be that your Streams application is ahead of all the records you've produced to that topic, possibly because of auto offset reset being set to latest, or possibly for some other reason not easily discernible from your question.

How to print KStream to console?

2 Answers2