1

I have a code, which uses the Confluent Kafka library, written in C#.
I Am able to connect to the Azure event hub from my local system properly.
However, when I put the same in the container. I get the following error.

3|1656401522.469|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: sasl_ssl://xyz.servicebus.windows.net:9093/bootstrap: Disconnected (after 148ms in state UP)
%6|1656401522.626|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://xyz.servicebus.windows.net:9093/bootstrap]: sasl_ssl://xyz.servicebus.windows.net:9093/bootstrap: Disconnected (after 124ms in state UP, 1 identical error(s) suppressed)


%3|1656399913.942|ERROR|rdkafka#consumer-1| [thrd:sasl_ssl://xyz.servicebus.windows.net:9093/bootstrap]: 1/1 brokers are down
%3|1656399922.791|ERROR|rdkafka#consumer-1| [thrd:sasl_ssl://xyz.servicebus.windows.net:9093/bootstrap]: 1/1 brokers are down
fail: Microsoft.Extensions.Hosting.Internal.Host[9]
      BackgroundService failed
      Confluent.Kafka.ConsumeException: Broker: Topic authorization failed

Any suggestions here ? Anyone ?

New Error

Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY, 4 identical error(s) suppressed)

Any my configuration

                    EnableAutoOffsetStore = false,
                    BootstrapServers = _configuration["KAFKA_SERVER"],
                    GroupId = consumerGroup,
                    AutoOffsetReset = AutoOffsetReset.Earliest,
                    SecurityProtocol = SecurityProtocol.SaslSsl,
                    SaslMechanism = SaslMechanism.Plain,
                    SaslUsername = "$ConnectionString",
                    SaslPassword = _configuration["EVENTHUB_CONNSTR"],
                    SslCaLocation = _configuration["EVENTHUB_CA_CERT_LOCATION"]
user2010220
  • 11
  • 2
  • 1
    How are you authenticating yourself to kafka? It looks like this authorization method is not passed to the container as expected. – Yaron Idan Jun 28 '22 at 18:26
  • @YaronIdan You mean the settings/configuration you are asking for ? Something like below ones EnableAutoOffsetStore = false, BootstrapServers = brokerList, GroupId = consumerGroup, AutoOffsetReset = AutoOffsetReset.Earliest, SecurityProtocol = SecurityProtocol.SaslSsl, SaslMechanism = SaslMechanism.Plain, SaslUsername = "$ConnectionString", SaslPassword = connStr, SslCaLocation = caCertLocation – user2010220 Jun 29 '22 at 06:16
  • My point was that some of the configurations you've mentioned in your comments are not passed as expected to the container. It might be the value of the `$ConnectionString` env var, or the contents of the `caCertLocation` path. Look into these details since they are most probably the root cause of the error you've met. – Yaron Idan Jun 29 '22 at 12:00
  • @YaronIdan Thanks, i did check those, for caCert i used to get a error, when the path was incorrect, I had fixed that earlier. Regarding the ConnectionString, its not the env variable. I read in document that to connect to EventHub from confluentKafka, we need to pass that as a string. That $ConnectionString itself is a value. Ref : https://notetoself.tech/2018/06/03/acessing-event-hubs-with-confluent-kafka-library/ – user2010220 Jun 29 '22 at 12:52
  • @YaronIdan Any further thoughts ? – user2010220 Jul 01 '22 at 06:04
  • Have you gone over all of the authorization parameters and made sure they are passed correctly to the container? – Yaron Idan Jul 04 '22 at 07:13
  • Yes, i did check again. Not getting what am missing. – user2010220 Jul 04 '22 at 11:56
  • Today am getting a new error Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 0ms in state APIVERSION_QUERY, 4 identical error(s) suppressed) – user2010220 Jul 04 '22 at 12:04
  • @YaronIdan More updates in my description – user2010220 Jul 04 '22 at 12:07
  • Am getting closer, it looks like AKS is not able to connect to Azure Eventhub, I have configured eventhub to work for specific whitelisted IPs. So, question is where would i get the IP of kubernetes (AKS) ? @YaronIdan – user2010220 Jul 07 '22 at 11:18
  • The IP should be the same as the loadbalancer used by your ingress controller, unless you are using a NAT gateway - in which case it would be its IP. – Yaron Idan Jul 08 '22 at 12:23

0 Answers0