spring boot kafka failed with "Broker may not be available" while using testcontainers with kafka, zookeeper, schema registry

Question

I've run testcontainers like below in the test code.

@Testcontainers
public class TestEnvironmentSupport {
    static String version = "5.4.0";
    static DockerImageName kafkaImage = DockerImageName.parse("confluentinc/cp-server").withTag(version);
    static DockerImageName zookeeperImage = DockerImageName.parse("confluentinc/cp-zookeeper").withTag(version);
    static DockerImageName schemaRegistryImage = DockerImageName.parse("confluentinc/cp-schema-registry").withTag(version);

    static Network network = Network.newNetwork();

    @Container
    static GenericContainer zookeeper = new GenericContainer<>(zookeeperImage)
        .withNetwork(network)
        .withCreateContainerCmdModifier(cmd -> cmd.withHostName("zookeeper"))
        .withExposedPorts(2181)
        .withEnv("ZOOKEEPER_CLIENT_PORT", "2181")
        .withEnv("ZOOKEEPER_TICK_TIME", "2000");

    @Container
    static GenericContainer kafka = new GenericContainer<>(kafkaImage)
        .withNetwork(network)
        .withCreateContainerCmdModifier(cmd -> cmd.withHostName("kafka"))
        .withExposedPorts(9092)
        .dependsOn(zookeeper)
        .withEnv("KAFKA_BROKER_ID", "1")
        .withEnv("KAFKA_ZOOKEEPER_CONNECT", "zookeeper:2181")
        .withEnv("KAFKA_LISTENER_SECURITY_PROTOCOL_MAP", "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT")
        .withEnv("KAFKA_ADVERTISED_LISTENERS", "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092")
        .withEnv("KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL", "schema-registry:8081");

    @Container
    static GenericContainer schemaRegistry = new GenericContainer<>(schemaRegistryImage)
        .withNetwork(network)
        .withCreateContainerCmdModifier(cmd -> cmd.withHostName("schema-registry"))
        .withExposedPorts(8081)
        .dependsOn(zookeeper, kafka)
        .withEnv("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
        .withEnv("SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL", "zookeeper:2181");

    @Test
    void test() {
        assertTrue(zookeeper.isRunning());
        assertTrue(kafka.isRunning());
        assertTrue(schemaRegistry.isRunning());
    }
}

and it works totally fine. But the problem occurred when I tried to run spring boot test with the above testcontainer configuration because testcontainer dynamically generates broker ports but NetworkClient continuously accessing the broker with localhost:9092 even though I dynamically override properties on @SpringBootTest code like below

@DynamicPropertySource
static void testcontainerProperties(final DynamicPropertyRegistry registry) {
    var bootstrapServers = kafka.getHost() + ":" + kafka.getMappedPort(9092);
    var schemaRegistryUrl = "http://" + schemaRegistry.getHost() + ":" + schemaRegistry.getMappedPort(8081);
    registry.add("spring.cloud.stream.kafka.binder.brokers", () -> bootstrapServers);
    registry.add("bootstrap.servers", () -> bootstrapServers);
    registry.add("schema.registry.url", () -> schemaRegistryUrl);
    registry.add("spring.cloud.stream.kafka.default.consumer.configuration.schema.registry.url", () -> schemaRegistryUrl);
}

Below is AdminClientConfig log on startup time, and it shows that bootstrap.servers = [localhost:56001] on which the port is dynamically binded by testcontainer.

2021-02-21 20:28:52.291  INFO 78241 --- [    Test worker] o.a.k.clients.admin.AdminClientConfig    : AdminClientConfig values:
        bootstrap.servers = [localhost:56013]
        client.dns.lookup = use_all_dns_ips

Even though I set like these it keeps trying to connect to localhost:9092 like below.

2021-02-21 20:28:52.457  INFO 78241 --- [    Test worker] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1613906932454
2021-02-21 20:28:53.095  WARN 78241 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2021-02-21 20:28:53.202  WARN 78241 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2021-02-21 20:28:53.407  WARN 78241 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

And below is the result of docker ps while running spring boot test.

CONTAINER ID   IMAGE                                   COMMAND                  CREATED              STATUS              PORTS                                                                       NAMES
e340d9e15fe4   confluentinc/cp-schema-registry:5.4.0   "/etc/confluent/dock…"   46 seconds ago       Up 46 seconds       0.0.0.0:56014->8081/tcp                                                     optimistic_joliot
ad3bf06df4b3   confluentinc/cp-server:5.4.0            "/etc/confluent/dock…"   55 seconds ago       Up 54 seconds       0.0.0.0:56013->9092/tcp                                                     infallible_brown
f7fa5f4ae23c   confluentinc/cp-zookeeper:5.4.0         "/etc/confluent/dock…"   About a minute ago   Up 59 seconds       0.0.0.0:56012->2181/tcp, 0.0.0.0:56011->2888/tcp, 0.0.0.0:56010->3888/tcp   agitated_leavitt
b1c036cdf00b   testcontainers/ryuk:0.3.0               "/app"                   About a minute ago   Up About a minute   0.0.0.0:56009->8080/tcp                                                     testcontainers-ryuk-68190eaa-8513-4dd8-ab67-175275f15a82

I tried run testcontainers with docker compose module but it has the same issue. What am I doing wrong? Please help.

OneCricketeer · Answer 1 · 2021-02-21T18:43:23.967

It's trying to connect to localhost:9092 because you've tried to connect to the advertised PLAINTEXT_HOST port, and that's the address it'll return. You shouldn't need to advertise two listeners for tests, so try using kafka:29092 directly instead of calling the mapped port method. Also, unless you have a specific need for server-side schema validation, you only need confluentinc/cp-kafka image.

spring-kafka embedded broker should work your tests, too, so you wouldn't need testcontainers

spring boot kafka failed with "Broker may not be available" while using testcontainers with kafka, zookeeper, schema registry

1 Answers1