Alpakka Kafka Java Test

Question

I am trying to write Junit tests for an app that uses kafka streams to emit data from kafka to a websocket connection. I have been able to run the application locally publishing to the topic and seeing the data returned on a socket connection. However I am struggling to correctly setup the test environment.

Code To Test

Currenlty I have this as the method I want to test:

    public Source<JsonNode, ?> getKafkaStream(String groupId, Set<Id> idsBelongingToUser) {
        return Consumer.plainSource(getKafkaConsumerSettings(groupId), Subscriptions.topics(this.kafkaTopic))
            .map(ConsumerRecord::value)
            .filter(event -> idsBelongingToUser.contains(event.getId())
            .map(consumerRecord -> objectMapper.convertValue(consumerRecord, JsonNode.class));
    }

I then use this source in a flow here:

    public Flow<JsonNode, JsonNode, ?> getDataStream(String groupId, Set<Id> idsBelogingToUser) {
        return Flow.fromSinkAndSource(
            Sink.ignore(),
            this.getKafkaStream(accountOfListener, idsBelogingToUser).runWith(BroadcastHub.of(JsonNode.class), this.materializer)
        );
    }

My attempted tests

My latest try has been to test the code with TestcontainersKafkaJunit4Test and here is a modified file of how I designed it.

import akka.actor.ActorSystem;
import akka.japi.Pair;
import akka.kafka.testkit.KafkaTestkitTestcontainersSettings;
import akka.kafka.testkit.javadsl.TestcontainersKafkaJunit4Test;
import akka.stream.testkit.javadsl.TestSink;
import akka.testkit.javadsl.TestKit;
import com.fasterxml.jackson.databind.JsonNode;
import lombok.SneakyThrows;
import org.apache.kafka.common.serialization.StringSerializer;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.TestInstance.Lifecycle;

import java.util.concurrent.CompletionStage;

import static java.util.concurrent.TimeUnit.SECONDS;
import static org.awaitility.Awaitility.await;


@TestInstance(Lifecycle.PER_CLASS)
public class KafkaTest extends TestcontainersKafkaJunit4Test {
    KafkaService toTest;
    String topic;

    private static final ActorSystem system = ActorSystem.create("TestkitTestcontainersTest");
    KafkaTest() {
        super(
            system,

            KafkaTestkitTestcontainersSettings.create(system)
                .withSchemaRegistry(true)
                .withNumBrokers(1)
                .withInternalTopicsReplicationFactor(1)
        );
        this.setUpAdminClient();
    }

    @SneakyThrows
    public <T> T getResultFromFuture(CompletionStage<T> future) {
        await().atMost(5, SECONDS).until(() -> future.toCompletableFuture().isDone());
        return future.toCompletableFuture().get();
    }

    @Test
    void test() {
        this.topic = createTopic();
        toTest = new KafkaService();

        JsonNode event1 = createRandomEvent();
        JsonNode event2 = createRandomEvent();
        String group = "some-key";

        //produce some messages
        getResultFromFuture(this.produce(this.topic, new StringSerializer(), new JsonSerdes<>(JsonNode.class).serializer(),
            Pair.create(group, event1),
            Pair.create(group, event2)));


        //Here I want to test the source is getting the messages produced above

        //One Attempt
        toTest.getKafkaStream(group, getSetOfIds())
            .runWith(TestSink.probe(system), materializer)
            .expectNext(event1)
            .expectNext(event2);
    }

    JsonNode createRandomEvent() { ... }
    @AfterAll
    void shutdownActorSystem() {
        this.cleanUpAdminClient();
        TestKit.shutdownActorSystem(system);
    }
}

I believe the setup is close to working since when I change from using the source stream to consume the messages and consume them with a consumer object I can see the events. I have tried printing in the map(ConsumerRecord::value) function and there are no logs which leads me to believe it is never executed. Unfortunately alpakka kafka does not have many examples on how to test this, and as a beginner to both kafka and scala I am struggling to apply what examples exist to my code. I have used the following links to set this up: alpakka kafka testing docs akka streams testing

I have modified the code slightly to keep it more general. If any information is unclear I will explain/edit the question. Thanks in advance for any help/suggestions

Answer 1

The approach that worked for me was in my test case I had to set the kafka auto offset configuration to be earliest whereas it is latest in the consumer properties I used to create the source