1

We have MongoDB cluster with three replicas. I have enabled preferred TLS and authentication type as MongoDB-X509.

We have three broker strimzi kafka cluster and connect cluster with all required plugins (i.e. mongoDB provided by debezium) up and running.

Below provided partial connect.yaml file used for kafka connect deployment:-

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect
spec:
  config:
    config.providers: directory
    config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
  externalConfiguration:
    volumes:
      - name: connector-config
        secret:
          secretName: mysecret

deployment works fine and able to see ca.pem and mongo-server.pem file in /opt/kafka/external-configuration/connector-config directory.

After then I am trying to create mongoDB connector with configuration files as give below, but not sure on exact way of passing certificates. As there is no sample configuration file available for mongoDb connectors. Could you please help on this by providing some sample configuration.

I tried below configuration file:-

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: io.debezium.connector.mongodb.MongoDbConnector
  tasksMax: 2
  config:
    ssl.truststore.type: PEM
    ssl.truststore.location: "${directory:/opt/kafka/external-configuration/connector-config:ca.pem}"
    ssl.keystore.type: PEM
    ssl.keystore.location: "${directory:/opt/kafka/external-configuration/connector-config:mongo-server.pem}"
    "mongodb.hosts": "rs0/192.168.99.100:27017" 
    "mongodb.name": "fullfillment" 
    "collection.include.list": "inventory[.]*"
    "mongodb.ssl.enabled": true
    "mongodb.ssl.invalid.hostname.allowed": true

but it was throwing syntax error. Please help on this by providing sample mongoDB connector.yaml?

andy
  • 525
  • 3
  • 6
  • 22

1 Answers1

0

As for Strimzi, you can use the external configuration to mount Secrets or Config Maps into the Strimzi Kafka Connect deployment: https://strimzi.io/docs/operators/latest/full/using.html#type-ExternalConfiguration-reference. Once it is loaded into the Pods, you can either just refer to it using the file path, or you can use the Kafka configuration providers to load the data into the configuration options.

Jakub
  • 3,506
  • 12
  • 20
  • Thanks Jakub! After following above link got to know we can create connector either via connect REST EP (through CURL) or via configuration file.yaml. Still not sure on creating mongoDB connector part on how to pass certificate as there is no sample configuration file.yaml available to create mongoDB connector. Have tired refering what is available for mysql but getting syntax error on deployment. I have also re-phrase the question, please take a look on it and suggest on it. Thanks!! – andy Dec 24 '21 at 07:02
  • I'm afraid I know the Strimzi part, but not the Debezium part. So not sure what options Debezium requires for the TLS certs etc., Sorry. – Jakub Dec 24 '21 at 10:32