Unit testing Kafka-streams with TopologyTestDriver and Micronaut

Question

When I try to run a basic unit test I get the following error:

java.lang.IllegalArgumentException: Unknown topic: streams-wordcount-output

    at org.apache.kafka.streams.TopologyTestDriver.getRecordsQueue(TopologyTestDriver.java:705)
    at org.apache.kafka.streams.TopologyTestDriver.readRecord(TopologyTestDriver.java:777)
    at org.apache.kafka.streams.TestOutputTopic.readRecord(TestOutputTopic.java:100)
    at at.wrwks.bmp.projekthistorie.streams.WordCountIntegrationTest.test(WordCountIntegrationTest.kt:70)

This is the stream code (based on the Micronaut documentation:

@Factory
class WordCountStream {

    @Singleton
    @Named(STREAM_WORD_COUNT)
    fun wordCountStream(builder: ConfiguredStreamBuilder): KStream<String, String> {
        // set default serdes
        val props: Properties = builder.configuration
        props[StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG] = Serdes.String()::class.jvmName
        props[StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG] = Serdes.String()::class.jvmName
        props[ConsumerConfig.AUTO_OFFSET_RESET_CONFIG] = "earliest"
        val source: KStream<String, String> = builder.stream(INPUT)
        val groupedByWord: KTable<String, Long> = source
            .flatMapValues { value -> value.toLowerCase().split("\\W+") }
            .groupBy({ _, word -> word }, Grouped.with(Serdes.String(), Serdes.String()))
            //Store the result in a store for lookup later
            .count(Materialized.`as`(WORD_COUNT_STORE))
        groupedByWord
            //convert to stream
            .toStream()
            //send to output using specific serdes
            .to(OUTPUT, Produced.with(Serdes.String(), Serdes.Long()))
        return source
    }

    companion object {
        const val STREAM_WORD_COUNT = "word-count"
        const val INPUT = "streams-plaintext-input"
        const val OUTPUT = "streams-wordcount-output"
        const val WORD_COUNT_STORE = "word-count-store"
    }
}

This is the test code:

@Tag("integration")
@MicronautTest(application = ProjekthistorieApplication::class)
class WordCountIntegrationTest {
    companion object {
        val builder = mockk<ConfiguredStreamBuilder>()
    }
    @Inject
    private lateinit var context: ApplicationContext
    @Inject
    private lateinit var wordCountStream: KStream<String, String>
    @Inject
    private lateinit var wordCountStreamFactory: WordCountStream
    lateinit var inputWordCount: TestInputTopic<String, String>
    lateinit var outputWordCount: TestOutputTopic<String, Long>
    lateinit var wordCount: TestOutputTopic<String, Long>
    lateinit var testDriver: TopologyTestDriver
    private lateinit var before: LocalDateTime

    @MockBean
    fun configuredStreamBuilder(): ConfiguredStreamBuilder {
        return builder
    }

    @BeforeEach
    fun setUp() {

    }

    @AfterEach
    fun tearDown() {
        testDriver.close()
    }

    @Test
    fun test() {
        before = LocalDateTime.now()
        val streamsBuilder = StreamsBuilder()
        every { builder.stream<String, String>(WordCountStream.INPUT) } returns streamsBuilder.stream(WordCountStream.INPUT)
        val config = Properties()
        config[StreamsConfig.APPLICATION_ID_CONFIG] = "bmp.projekthistorie.test"
        config[StreamsConfig.BOOTSTRAP_SERVERS_CONFIG] = "dummy:1234"
        testDriver = TopologyTestDriver(streamsBuilder.build(), config)
        inputWordCount = testDriver.createInputTopic(WordCountStream.INPUT, Serdes.StringSerde().serializer(), Serdes.StringSerde().serializer())
        outputWordCount = testDriver.createOutputTopic(WordCountStream.OUTPUT, Serdes.StringSerde().deserializer(), Serdes.LongSerde().deserializer())
        wordCount = testDriver.createOutputTopic(WordCountStream.STREAM_WORD_COUNT, Serdes.StringSerde().deserializer(), Serdes.LongSerde().deserializer())

        inputWordCount.pipeInput("word word la la")
        val readRecord = outputWordCount.readRecord()

        Assertions.assertThat(readRecord).isNotNull()
    }
}

What do I miss?

score 0 · Answer 1 · answered Jun 04 '20 at 13:20

0

I think the problem come from

@MockBean
fun configuredStreamBuilder(): ConfiguredStreamBuilder {
   return builder
}

you create a test driven topology:

val streamsBuilder = StreamsBuilder()
testDriver = TopologyTestDriver(streamsBuilder.build(), config)

but you never inject it in you streaming service. You inject a mock ConfiguredStreamBuilder.

Try to inject with MockBean:

val streamsBuilder = StreamsBuilder()

answered Jun 04 '20 at 13:20

Anorgar

159
8

`ConfiguredStreamBuilder`belongs to Micronaut and will be injected in the `wordCountStream()` bean factory, StreamsBuilder belongs to Kafka Streams and will be used to create the streams for the test. – Andras Hatvani Jun 04 '20 at 14:37
Sorry for that, I worked on similar code, but in java, not kotlin. Did you tried to debug the stream ? You can check if the value "word word la la" goes through the stream – Anorgar Jun 04 '20 at 15:06

score 0 · Answer 2 · answered Jun 05 '20 at 05:13

Since ConfiguredStreamBuilder is a subclass of StreamsBuilder there was no need for mocking and the ConfiguredStreamBuilder can be configured as a bean which will be used in the stream factory bean. Here's the working test:

@Tag("integration")
@MicronautTest(environments = ["dev"])
class WordCountIntegrationTest(private val configuredStreamBuilder: ConfiguredStreamBuilder) {
    lateinit var inputWordCount: TestInputTopic<String, String>
    lateinit var outputWordCount: TestOutputTopic<String, Long>
    lateinit var testDriver: TopologyTestDriver

    @Bean
    fun configuredStreamBuilder(): ConfiguredStreamBuilder {
        val config = Properties()
        config[StreamsConfig.APPLICATION_ID_CONFIG] = "app"
        config[StreamsConfig.BOOTSTRAP_SERVERS_CONFIG] = "dummy:1234"
        config[StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG] = Serdes.String()::class.jvmName
        config[StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG] = Serdes.String()::class.jvmName
        return ConfiguredStreamBuilder(config)
    }

    @Test
    fun test() {
        testDriver = TopologyTestDriver(configuredStreamBuilder.build(), configuredStreamBuilder.configuration)
        inputWordCount = testDriver.createInputTopic(WordCountStream.INPUT, Serdes.StringSerde().serializer(), Serdes.StringSerde().serializer())
        outputWordCount = testDriver.createOutputTopic(WordCountStream.OUTPUT, Serdes.StringSerde().deserializer(), Serdes.LongSerde().deserializer())

        inputWordCount.pipeInput("word word la la")
        val readRecord = outputWordCount.readRecord()

        assertThat(readRecord).isNotNull()
    }
}

Unit testing Kafka-streams with TopologyTestDriver and Micronaut

2 Answers2