1

I am new to kafka streams and I would like to read a topic and write a part of it in a new topic using kafka streams api. My key is string and value is Avro Is there a documentation/example that I can use ?

Edit :

    final StreamsBuilder builder = new StreamsBuilder();
    final KStream<String, GenericRecord> inputStream = builder.stream("Test_CX_TEST_KAFKA_X");
    final KStream<String, String> newStream = inputStream.mapValues(value -> value.get("ID").toString());
    newStream.to("SUB_TOPIC",Produced.with(Serdes.String(),Serdes.String()));
    final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);
    streams.start();

In SUB_TOPIC I have :

Key: { "ID": "145" } Timestamp: Mar 14th, 2019 17:52:23.43 Offset: 12 Partition: 0

my Input topic :

{ "ID": "145", "TIMESTAMP": 1552585938545, "WEEK": "\u0000", "SOURCE": { "string": "TMP" }, "BODY": { "string": "{\"operation_type\":\"INSERT\",\"old\":{\"ROW_ID\":null,\"LAST_UPD\":null,\"DENOMINATION\":null,\"SIREN_SIRET\":null},\"new\":{\"ROW_ID\":\"170309-********\",\"LAST_UPD\":\"2019-03-14T17:52:18\",\"DENOMINATION\":\"1-******\",\"SIREN_SIRET\":null}}" }, "TYPE_ACTION": { "string": "INSERT" } }

How can I add other fields from Body in the new topic ? example :

{ "ID": "145", "TIMESTAMP": 1552585938545, "WEEK": "\u0000", "SOURCE": { "string": "TMP" }, "BODY": { "string": "{\"operation_type\":\"INSERT\",\"old\":{\"ROW_ID\":null,\"LAST_UPD\":null},\"new\":{\"ROW_ID\":\"170309-********\",\"LAST_UPD\":\"2019-03-14T17:52:18\"}}" }, "TYPE_ACTION": { "string": "INSERT" } }

KnowledgeSeeker
  • 77
  • 1
  • 12
  • 1
    Have you checked their own documentation? https://kafka.apache.org/21/documentation/streams/developer-guide/ – t3ng1l Mar 14 '19 at 14:10
  • If you just want to read from one topic, filter the data and produce the filtered data to a new topic, there's no need to use Kafka Streams. You can use the standard Kafka. ```final StreamsBuilder builder = new StreamsBuilder(); builder.stream(srcTopic) .flatMapValues(value -> filterEvent(value, addresses)) .to(destTopic);``` Where you can design your `filterEvent` however you'd like. – t3ng1l Mar 14 '19 at 14:19
  • Does it have to be Kafka Streams? The simplest way is with KSQL, e.g. `CREATE STREAM topic2 AS SELECT col1, col2 FROM topic1`. – Robin Moffatt Mar 14 '19 at 14:34
  • My value is a json and I can't create a proper json using ksql (is there a way to do it ?) besides can a stream be consumed like a topic ? – KnowledgeSeeker Mar 14 '19 at 14:43

1 Answers1

1

You can simply consume a topic as stream and modifythe value/KeyValues using .map()/.mapValues() functions.

Example: Let's say if you want to pick a column from avro record and publish to new output topic.

// If you are using Schema registry, make sure to add the schema registry url 
// in streamConfiguration. Also specify the AvroSerde for VALUE_SERDE

final Properties streamsConfiguration = new Properties();
streamsConfiguration.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
final StreamsBuilder builder = new StreamsBuilder();
final KStream<String, GenericRecord> inputStream = builder.stream("inputTopic");
final KStream<String, String> newStream = userProfiles.mapValues(value -> value.get("fieldName").toString());
subStream.to("outputTopic",Produced.with(Serdes.String(),Serdes.String());
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);

Also, you can look into examples on github :
https://github.com/confluentinc/kafka-streams-examples/blob/5.1.2-post/src/main/java/io/confluent/examples/streams/WikipediaFeedAvroExample.java

Nishu Tayal
  • 20,106
  • 8
  • 49
  • 101