2

We have a secured Kafka cluster in our production environment (using TLS encryption, certificate based client authentication and ACL's). I am trying to figure out how to configure the Flink Kafka connector with the appropriate settings so it can connect securely to our Kafka cluster. Is this possible with Flink? Do i have to pass the security configuration stuff via the properties?

Our Flink cluster is running on Kubernetes (1.14.2) and using the latest Flink stable release (v1.8) with the integrated kafka connector.

Sebastian Jaekel
  • 101
  • 1
  • 6

1 Answers1

1

After some fiddling with the docs I have got it working by myself. I am now providing the required Java keystores at deployment time when running helm (We deploy the whole Flink stuff through helm charts). The keystores will be base64 encoded and saved as a Kubernetes secret. The taskmanager pods mount the secret at the given location.

I can now pass the location of the keystore / truststore and their passwords on the command-line as params when running the Flink job. These params are finally used to configure the Kafka client via properties.

Dharman
  • 30,962
  • 25
  • 85
  • 135
Sebastian Jaekel
  • 101
  • 1
  • 6
  • So I assume you are passing this parameters as environment properties. I have 2 doubts, are you using kafka ssl properties (eg. ssl.keystore.location="", ssl.keystore.password="", etc..) ? Also how are you setting this in org.apache.flink.connector.kafka.source.KafkaSource ? – Devas Mar 23 '23 at 13:46