0

I have been trying to find documentation to connect to IBM Message Hub through C# Apache Kafka Confluent API, but have not been successful. The official repo on github does not have sample for C#. Has anyone been able to communicate to ibm message hub using C#, if yes can you share the process.

Thanks.

Update: I have had success in communicating with the IBM Message Hub.

Libraries:

  1. librdkafka -... 0.11.0-RC2

  2. Certificate From : https://curl.haxx.se/docs/caextract.html

  3. Confluent.kafka.dll Confluent.Kafka 0.11.0-RC1

Config:

private static Dictionary<string, object> constructConfig(string brokerList, bool enableAutoCommit) =>
            new Dictionary<string, object>
            {
                { "group.id", "history" },
                { "enable.auto.commit", enableAutoCommit },
                { "auto.commit.interval.ms", 5000 },
                { "statistics.interval.ms", 60000 },
                { "bootstrap.servers", "ibmserver:port" },
                { "default.topic.config", new Dictionary<string, object>()
                    {
                        { "auto.offset.reset", "smallest" }
                    }
                },
                {"ssl.ca.location",@"E:\cert\cacert.pem" },              
                {"api.version.request","true" },
                {"security.protocol","sasl_ssl" },
                {"sasl.mechanisms","PLAIN" },
                {"sasl.username","xxxx" },
                {"sasl.password","xxxxx" }

            };

.net Version: 4.5.2

Hope it saves time for someone.

Thanks to Edoardo Comar for guiding me to much needed information.

Suchit Patel
  • 13
  • 2

1 Answers1

1

We have not written a C# sample yet.

The Confluent C# Kafka client is a wrapper around the C library librdkafka which until version 0.9.5 (the last release at the time of writing this post) could not be built for Windows with SASL_SSL support which is necessary for authenticating against Message Hub.

However that has changed with librdkafka 0.11 (in release candidate as I'm posting this comment) thanks to some very recent commits.

I verified that librdkafka 0.11 (master branch) is able to authenticate with Message Hub on Windows. You will need to set these configuration properties:

api.version.request=true
security.protocol=sasl_ssl
ssl.ca.location=<path to a valid cert.pem file>
sasl.mechanisms=PLAIN
sasl.username=<username from your Bluemix credentials>
sasl.password=<password from your Bluemix credentials>

I didn't know how to obtain a valid .pem certificate file in Windows, so I copied over a cert.pem file from macOS (installed via brew openssl in /usr/local/etc/openssl ). A cert.pem from Ubuntu (found in /etc/openssl) should work as well.

Good luck and please keep me updated with your progress,

Edoardo Comar
  • 531
  • 2
  • 5
  • Thanks Edoardo . I will work on the above and update the progress on this thread. – Suchit Patel Jun 29 '17 at 10:17
  • could your share where to placed in windows, i keep getting the error, "Failed to verify broker certificate: unable to get local issuer certificate". – Suchit Patel Jun 29 '17 at 11:36
  • Updated: i added the path to cert.pem file that i generated using openssl. The library i am using is Confluent.Kafka.dll version: 0.11.0 pre release. I am getting Acessviolation exception. Can you help out – Suchit Patel Jun 29 '17 at 12:52