Overriding KStreams default serializer (ByteArraySerializer)

Question

I can't seem to override the serializer of a topic to Serdes.String(). I'm trying a simple use case of reading from a topic (stream), and writing to a KTable. What I have so far:

@Component
class Processor {
    @Autowired
    public void process(final StreamsBuilder builder) {
        final Serde<String> stringSerde = Serdes.String();
        builder.stream("input_topic", Consumed.with(stringSerde, stringSerde))
                .filter((key, value) -> value.contains("ACTION"))
                .toTable(Materialized.as("output_table_materialized"))
                .toStream().to("output_table", Produced.with(stringSerde, stringSerde)); // EDIT: added this last line

    }
}

The exception I get is:

org.apache.kafka.streams.errors.StreamsException: A serializer (org.apache.kafka.common.serialization.ByteArraySerializer) is not compatible to the actual key type (key type: java.lang.String). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters.

From what I gather, it understands the message is a String but it's using the default deserializer ByteArraySerializer. Where am I going wrong in the above code?

score 1 · Answer 1 · answered May 17 '22 at 18:30

I faced a similar issue and the solution was to specify the serdes on the Materialized instance, i.e. swapping

.toTable(Materialized.as("output_table_materialized"))

with

.toTable(Materialized.as("output_table_materialized").withKeySerde(stringSerde).withValueSerde(stringSerde))

score -1 · Answer 2 · answered Dec 16 '20 at 00:23

-1

The Consumed.with would be a Deserializer.

The error is on the Serializer, or the toTable call, which you may add Produced.with or modify your application properties to configure the defaults there

answered Dec 16 '20 at 00:23

OneCricketeer

179,855
19
132
245

Thanks! Did some tests and arrived to the same conclusion. So I then tried re-converting the KTable back into a stream and writing to a topic there; however I get the same exception. I've edited the question if you don't mind looking at it again! – Tiberiu Dec 16 '20 at 00:38
The KTable still needs types – OneCricketeer Dec 16 '20 at 00:56
not sure I understand...the toTable() doesn't seem to have Serde parameters. I even tried creating the `KTable` variable first then converting it (though more out of desperation). Also I don't see any mention of this in their documentation, been following: https://kafka-tutorials.confluent.io/kafka-streams-convert-to-ktable/kstreams.html – Tiberiu Dec 16 '20 at 01:07
The Materialized object takes a serde parameter as well https://kafka.apache.org/25/javadoc/org/apache/kafka/streams/kstream/KStream.html#toTable-org.apache.kafka.streams.kstream.Named-org.apache.kafka.streams.kstream.Materialized- in that link, they do set `StreamsConfig.DEFAULT_*_SERDE_CLASS_CONFIG`... Do you not have that? – OneCricketeer Dec 16 '20 at 05:32

Overriding KStreams default serializer (ByteArraySerializer)

2 Answers2