Connection to external Kafka Server using confluent-kafka-dotnet fails

Question

I need to read Kafka messages with .Net from an external server. As the first step, I have installed Kafka on my local machine and then wrote the .Net code. It worked as wanted. Then, I moved to the cloud but the code did not work. Here is the setup that I have.

I have a Kafka Server deployed on a Windows VM (VM1: 10.0.0.4) on Azure. It is up and running. I have created a test topic and produced some messages with cmd. To test that everything is working I have opened a consumer with cmd and received the generated messages.

Then I have deployed another Windows VM (VM2, 10.0.0.5) with Visual Studio. Both of the VMs are deployed on the same virtual network so that I do not have to worry about opening ports or any other network configuration.

then, I have copied my Visual Studio project code and then changed the IP address of the bootstrap-server to point to the Kafka server. It did not work then, I read that I have to change the server configuration of Kafka, so I opened the server.properties and modified the listeners property to listeners=PLAINTEXT://10.0.0.4:9092. It still does not work.

I have searched online and tried many of the tips but it does not work. I think first of all to provide the credential to an external server (vm1), and probably some other configuration. Unfortunately, the official documentation of confluent is very short with very few examples. There is also no example to my case on the official GitHub. I have played with the "Sasl" properties in the Consumer Config class, but also no success.

the error message is:

%3|1622220986.498|FAIL|rdkafka#consumer-1| [thrd:10.0.0.4:9092/bootstrap]: 10.0.0.4:9092/bootstrap: Connect to ipv4#10.0.0.4:9092 failed: Unknown error (after 21038ms in state CONNECT) Error: 10.0.0.4:9092/bootstrap: Connect to ipv4#10.0.0.4:9092 failed: Unknown error (after 21038ms in state CONNECT) Error: 1/1 brokers are down

Here is my .Net core code:

static void Main(string[] args)
    {
        string topic = "AzureTopic";
        var config = new ConsumerConfig
        {
            BootstrapServers = "10.0.0.4:9092",
            GroupId = "test",
            //SecurityProtocol = SecurityProtocol.SaslPlaintext,
            //SaslMechanism = SaslMechanism.Plain,
            //SaslUsername = "[User]",
            //SaslPassword = "[Password]",
            AutoOffsetReset = AutoOffsetReset.Latest,
            //EnableAutoCommit = false
        };

        int x = 0;
        
        using (var consumer = new ConsumerBuilder<Ignore, string>(config)
             .SetErrorHandler((_, e) => Console.WriteLine($"Error: {e.Reason}"))
            .Build())
        {
            consumer.Subscribe(topic);
            var cancelToken = new CancellationTokenSource();

            while (true)
            {
                // some tasks
            }
            consumer.Close();

Answer 1

If you set listeners to a hard-coded IP, it'll only start the server binding and accepting traffic to that ip

And your listener isn't defined as SASL, so I'm not sure why you've tried using that in the client. While using credentials is strongly encouraged when sending data to cloud resources, it's not required to fix a network connectivity problem. You definitely shouldn't send credentials over plaintext, however

Start with these settings

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://10.0.0.4:9092

That alone should work within the VM shared network. You can use the console tools included with Kafka to test it.

And if that still doesn't work from your local client, then it's because 10.0.0.0/8 address space is considered a private network and you must advertise the VM's public IP and allow TCP traffic on port 9092 through Azure Firewall. It'd also make sense to expose multiple listeners for internal Azure network and external, forwarded network traffic

Details here discuss AWS and Docker, but the basics still apply

Overall, I think it'd be easier to setup Azure EventHub with Kafka support