How to run Kafka-Connect in Minikube?

Question

To run cp S3-connect to consume kafka topic in my local mac, I did something like below

    1. Installed Confluent Kafka connector and ran the kafka connect-standalone.sh
ML-C02Z605SLVDQ:kafka_2.12-2.5.0 e192270$ confluent-hub install confluentinc/kafka-connect-s3:latest --component-dir /usr/local/share/java --worker-configs config/connect-distributed.properties 

ML-C02Z605SLVDQ:kafka_2.12-2.5.0 e192270$ cd kafka_2.12-2.5.0

ML-C02Z605SLVDQ:kafka_2.12-2.5.0 e192270$ bin/connect-standalone.sh config/connect-standalone.properties s3-sink.properties. // s3-sink.properties connector.class=io.confluent.connect.s3.S3SinkConnector

Now, to run Kafka S3 connect in minikube I have installed Kafka-connect(kafka-connect-s3) in minikube using cp-helm-charts with help this tutorial Using a connector with Helm-installed Kafka/Confluent.

1. How to copy kafka config and script files inside kafka-connect pod ?
2. Should I need to login kafka-connect pod to run connect-standalone.sh command?

Answer 1

There is a from scratch procedure here. The only requirement is Minikube.

The steps you need are the following:

0. Start Minikube
1. Deploy a Kafka cluster using the Strimzi Operator
2. Build your own custom image including required plugins and dependencies
3. Deploy Kafka Connect cluster in distributed mode using that image
4. Create a KafkaConnector instance passing a configuration YAML

Answer 2

How to copy kafka config and script files inside kafka-connect pod

You shouldn't copy anything. Everything is configured by env-vars. The Helm charts should be mostly documenting how those variables are working.

The Docker image uses Connect Distributed, which is started via a REST API, not a property file. And confluentinc/cp-kafka-connect already contains S3 Connect

Answer 3

You can also take a look at https://strimzi.io/. The project is aimed at making the installation and management of a Kafka and Kafka Connect cluster on Kubernetes very easy.