1

This is the first time I am trying to connect to Kafka server using Asp.Net console app and I ran into a bunch of issues. I solved most but can't shake this one off.

I am testing the producer part and keep getting "SSL Handshake Failed" error. I followed some suggestions here, including setting SslEndpointIdentificationAlgorithm to blank.

This is my producer config section:

_producerConfig = new ProducerConfig
{
    BootstrapServers = bootstrapServer,
    EnableDeliveryReports = true,
    ClientId = Dns.GetHostAddresses(),

    //SecurityProtocol = SecurityProtocol.SaslSsl,
    SecurityProtocol = SecurityProtocol.Ssl,
    SslCaLocation = @"D:\Something\Certificate\myCert.pem",
    SslKeyPassword = "blahblah",
    SslEndpointIdentificationAlgorithm=SslEndpointIdentificationAlgorithm.None
};

...

public async Task StartSendingMessages(string topicName)
{
    using (var producer = new ProducerBuilder<long, string>(_producerConfig)
        .SetKeySerializer(Serializers.Int64)
        .SetValueSerializer(Serializers.Utf8)
        .SetLogHandler((_, message) => Console.WriteLine($"Facility: {message.Facility}-{message.Level} Message: {message.Message}"))
        .SetErrorHandler((_, e) => Console.WriteLine($"Error: {e.Reason}. Is Fatal: {e.IsFatal}"))
        .Build())
    try
    {
        Console.WriteLine("\nProducer loop started...\n\n");
        for (var character = 'A'; character <= 'C'; character++)
        {
            var message = $"Character #{character} sent at {DateTime.Now:yyyy-MM-dd_HH:mm:ss}";

            var deliveryReport = await producer.ProduceAsync(topicName,
                new Message<long, string>
                {
                    Key = DateTime.UtcNow.Ticks,
                    Value = message
                });

When I run the application, I get:

Facility: FAIL-Error Message: [thrd:ssl://1.2.3.4:9093/bootstrap]: ssl://1.2.3.4:9093/bootstrap: SSL handshake failed: ssl\statem\statem_clnt.c:1890:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (add broker's CA certificate to the Windows Root certificate store) (after 46ms in state SSL_HANDSHAKE)
Error: ssl://1.2.3.4:9093/bootstrap: SSL handshake failed: ssl\statem\statem_clnt.c:1890:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (add broker's CA certificate to the Windows Root certificate store) (after 46ms in state SSL_HANDSHAKE). Is Fatal: False

What am I doing wrong, or missing?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
NoBullMan
  • 2,032
  • 5
  • 40
  • 93
  • Your code seems fine. Is there any other way you can check the PEM file is correct? Maybe some config is missing on the actual Kafka brokers. Are there any other clients that have connected over SSL? – OneCricketeer Feb 18 '23 at 03:56
  • The folks who setup Kafka servers, or brokers, are Java shop and they use .JKS certificates. Because we develop in .Net and C#, they converted the certificate to .PEM and sent it to us. I have been getting all sorts of errors from handshake failure to certificate location error (I changed relative path to full path) to weak md (OpenSLL error) even though they says the CSR and CER files are TLS1.2 compliant, SHA256withRSA 2048-bit RSA key. This is my first attempt at Kafka connection and getting a bit overwhelmed. – NoBullMan Feb 18 '23 at 04:24
  • You can download Kafka CLI tools, and still use your pem files for debugging connectivity – OneCricketeer Feb 21 '23 at 14:17
  • @Mike This is not my question – OneCricketeer Mar 20 '23 at 16:18
  • My problem was certificate structure. I asked Kafka admins to convert their JKS file to PEM. I then used the PEM file to create 3 separate files: xxxsah2,cer, certificate.cer and certificate.key. This fixed my problem. – NoBullMan Mar 20 '23 at 21:08
  • Could you please show how you are using these 3 files? – nnnn Mar 29 '23 at 20:57
  • @nnnn I added the steps I had to take to create certificate files. – NoBullMan Apr 21 '23 at 13:18

1 Answers1

1

As I mentioned, I resolved the issue by breaking .pem file, provided to us by middleware team, into 3 certificate/key files. I am outlining the steps I took, maybe it will help someone facing the same issue.

  • Request .PEM version of .JKS certificate and key. Certificate is password protected and therefore they would need to provide the password as well.

  • In the remainder of these staps, we assume PEM certificate received is called mykafka.pem.

  • Open mykafka.pem in a text editor, such as Notepad++. This file contains entire certificate chain which needs to be broken up into different certificate documents.

  • Locate the line starting with "-----BEGIN ENCRYPTED PRIVATE KEY-----"

  • Copy the text between "-----BEGIN ENCRYPTED PRIVATE KEY-----" and "-----END ENCRYPTED PRIVATE KEY-----", inclusive.

  • Paste the content of copied text to a new file and name it "mykafka.key"

  • Locate the text "friendlyName: mykafka"

  • Copy the content between lines "-----BEGIN CERTIFICATE-----" and "-----BEGIN CERTIFICATE-----", inclusive.

  • Paste the content of copied text to a new file and name it "mykafka.cer"

  • Locate the text sha2ca.cer in certificate section's friendly name (e.g. "friendlyName: blah2021sha2ca.cer")

  • Copy the content between lines "-----BEGIN CERTIFICATE-----" and "-----BEGIN CERTIFICATE-----", inclusive.

  • Paste the content of copied text to a new file and name it as the friendly name that was listed in certificate section (e.g., blah2021sha2ca.cer)

  • You should now have 3 certificate/key files: mykafka.key, mykafka.cer, blah2021sha2ca.cer.

  • In your producer class, declare a client config variable and set its SSL settings like below:

    public class Producer {

     private readonly ProducerConfig _producerConfig;

 public Producer(string bootstrapServer)
 {
     _producerConfig = new ProducerConfig
     {
         BootstrapServers = bootstrapServer,
         ....
         SecurityProtocol = SecurityProtocol.Ssl,
         SslCaLocation = "<path_to_certificate_folder>\blah2021sha2ca.cer",
         SslCertificateLocation = "<path_to_certificate_folder>mykafka.cer",
         SslKeyLocation = "<path_to_certificate_folder>mykafka.key",
         SslKeyPassword = "<password_received_from_whoever_provided_pem_file>",
         ....
         EnableSslCertificateVerification = false,
     }
 }

    }

I added the last entry, EnableSslCertificateVerification, because without it I was getting handshake failure.

NoBullMan
  • 2,032
  • 5
  • 40
  • 93