0

I have implemented Kafka two way SSL authentication on a 17 node cluster. I have tested by running console consumer/producer commands from few nodes of the cluster. But when I try to do that from local network ( Laptop ) it doesn't work. I get SSL handshake error. I am suspecting it to be advertised listener issue as there is no adv. listener defined on server.properties. We are using private ips/private dns in all our configurations. From the local network below command works ( ip address is private ip of one of the brokers)

openssl c_client -connect 10.97.33.111:9093

My server.properties file has below entries

listeners=EXTERNAL://:9092,INTERNAL://:9091,CLIENT://:9093 
listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL,CLIENT:SSL 
## Inter Broker Listener Configuration 
inter.broker.listener.name=INTERNAL

Please suggest what is required to fix this issue.

Anirban
  • 257
  • 4
  • 12
  • Hi @cricket_007 , Any advice ? I have copied the Truststore file from my Broker to Laptop....generated a Keystore on Laptop using same CA cert & key that were used by Brokers...When I try to run console producer it errors out saying SSL handshake failed. – Anirban Mar 12 '20 at 19:49
  • Not sure. I would start over from here https://docs.confluent.io/current/security/security_tutorial.html In particular, client certs should be different from inter-broker certs if you really wanted to be secure – OneCricketeer Mar 12 '20 at 20:59
  • Also, you might be interested in https://www.openpolicyagent.org/docs/latest/kafka-authorization/ or http://coheigea.blogspot.com/2017/02/securing-apache-kafka-broker-using.html – OneCricketeer Mar 12 '20 at 21:02
  • @cricket_007, I am suspecting it to be Advertised listener issue. Currently there is no Adv listener defined. We have 3 listeners ( internal:9091 , external:9092 , client:9093 ) and all using SSL. When I define Adv listener as SSL://0000:9093 , Brokers don't start. Our cluster of EC2 nodes are using AWS private ips and those are not visible from local network. If you think that is the issue I can open a separate thread. – Anirban Mar 13 '20 at 21:38
  • Yes that's a problem 0.0.0.0 is not a real ip. You must advertise the real ip. If brokers aren't in a VPC, they can't communicate. If they're not visible from local network, what's the point of "external"? – OneCricketeer Mar 14 '20 at 13:21
  • Hi @cricket_007, ok....I will open a new thread. Thx. – Anirban Mar 14 '20 at 14:19
  • That's not what I said. You can edit this post with more details – OneCricketeer Mar 14 '20 at 16:00
  • Ok...my server.properties has this ...We are using private ips and when I do "openssl c_client -connect 10.97.33.111:9093" from laptop it works. 10.97.33.111 is private ip of one of the brokers But when I try to run console consumer/producer, it gives SSL handshake error. listeners=EXTERNAL://:9092,INTERNAL://:9091,CLIENT://:9093 listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL,CLIENT:SSL ## Inter Broker Listener Configuration inter.broker.listener.name=INTERNAL – Anirban Mar 15 '20 at 20:32
  • Please [edit] your question to include formatted code, but again, if EC2 doesn't have an open port in the VPC to connect from, then you shouldn't have an external listener. You'd have internal VPC traffic and "client" AWS traffic – OneCricketeer Mar 15 '20 at 23:23
  • Hi @cricket_007, 9092 was getting used ( along with Adv listener ) before implementing SSL.But after implemented SSL , the Adv setting was causing issue and brokers were failing. So we removed it. I have updated my question. – Anirban Mar 16 '20 at 00:57
  • You need the advertised listeners. You still have issues, right? So, why remove it? https://confluent.io/blog/kafka-listeners-explained/ You can use keystore explorer to check your keys – OneCricketeer Mar 16 '20 at 01:50
  • HI @cricket_007, This is the error I got when adv listener was defined and brokers didn't start java.lang.IllegalArgumentException: Error creating broker listeners from 'SSL://0.0.0.0:9093': No security protocol defined for listener SSL – Anirban Mar 16 '20 at 18:38
  • Seems to work here. https://docs.confluent.io/current/kafka/authentication_ssl.html . I think I already said it, but can you please try https://github.com/confluentinc/cp-ansible – OneCricketeer Mar 16 '20 at 18:52
  • 1
    We are using that Ansible script. It doesn't create Adv Listener on server.properties. We had to make an enhancement to add that. But the value is causing issue. – Anirban Mar 17 '20 at 13:35
  • What kind of enhancement? What value is it being set to? All the properties are able to be set https://github.com/confluentinc/cp-ansible/blob/5.4.1-post/roles/confluent.kafka_broker/templates/server.properties.j2#L21 – OneCricketeer Mar 18 '20 at 06:58
  • 1
    I was able to fix the issue. The latest version sets the adv listener. The issue was that on AWS Broker side truststore/keystore were created by Ansible script using private DNS. On the laptop we used Private IP. The fix was to use route53 names so that same name can be used from both server and client side. – Anirban Mar 19 '20 at 20:24
  • Cool. Feel free to answer your own questions below – OneCricketeer Mar 19 '20 at 23:25

0 Answers0