EmbeddedKafkaCluster missing?

Question

This blog entry https://www.confluent.io/blog/stream-processing-part-2-testing-your-streaming-application/ refers to the class EmbeddedKafkaCluster, which is supposed to be in the library kafka-streams-test-utils.

However, this class is missing in the library, e.g. org.apache.kafka/kafka-streams-test-utils/2.5.1.

I thought that I can use the source code from github https://github.com/a0x8o/kafka/blob/master/streams/src/test/java/org/apache/kafka/streams/integration/utils/EmbeddedKafkaCluster.java

But this source code refers to some classes, e.g. kafka.zk.EmbeddedZookeeper and kafka.utils.MockTime, that I assumed must be in the library like org.apache.kafka/kafka_2.13/2.5.1. Unfortunately, they are also missing.

What is the best way to configure a project to use the EmbeddedKafkaCluster in this case?

Thanks

Boris

That repo isnt the actual Kafka source code. Also, are you using Maven? If so, there are classifiers that add test classes — OneCricketeer, Oct 06 '20 at 15:15
I know there was a JIRA about making the embedded servers easier to integrate. The spring-kafka one is widely used otherwise. Or, are you able to use docker? https://www.testcontainers.org/modules/kafka/ — OneCricketeer, Oct 06 '20 at 17:20
Part of our development takes place on Windows where docker is not installed (cannot be installed). As of now, we will look into github.com/salesforce/kafka-junit . We will also use some of the source code available for EmbeddedSingleNodeCluster to integrate the schema registry. — Boris, Oct 07 '20 at 07:48
There's this https://medium.com/bakdata/transparent-schema-registry-for-kafka-streams-6b43a3e7a15c Though, I wonder if mock objects will work fine for you rather than actually starting servers — OneCricketeer, Oct 07 '20 at 14:04
thanks a lot! this might be useful in setting up the schema registry! — Boris, Oct 07 '20 at 19:03

Kubus · Answer 1 · 2021-01-12T21:13:18.887

Add the following dependencies:

    //build.gradle

    testCompile group: 'junit', name: 'junit', version: '4.13'
    testCompile group: 'org.hamcrest', name: 'hamcrest-junit', version: '2.0.0.0'

    compile group: 'org.apache.kafka', name: 'kafka_2.13', version: '2.7.0'
    testCompile group: 'org.apache.kafka', name: 'kafka_2.13', version: '2.7.0', classifier:'test'
    compile group: 'org.apache.kafka', name: 'kafka-streams', version: '2.7.0'
    testCompile group: 'org.apache.kafka', name: 'kafka-streams', version: '2.7.0', classifier: 'test'
    compile group: 'org.apache.kafka', name: 'kafka-streams-test-utils', version:  '2.7.0'
    testCompile group: 'org.apache.kafka', name: 'kafka-clients', version: '2.7.0', classifier: 'test'

If you’re using Maven, convert all dependencies based on this code:

//pom.xml
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>2.7.0</version>
    <scope>test</scope>
    <classifier>test</classifier>
</dependency>

And create the EmbeddedKafkaCluster in the following way (Kotlin example):

    @Test
    fun createEmbeddedKafkaClusterTest() {

        val NUM_BROKERS = 1
        val embeddedKafkaCluster = EmbeddedKafkaCluster(NUM_BROKERS)
        Assert.assertNotNull(embeddedKafkaCluster)

        embeddedKafkaCluster.start()

        embeddedKafkaCluster.createTopic("TestTopic")
    }

EmbeddedKafkaCluster missing?

1 Answers1