0

it looks like if we like to set up Kafka connector in distributed mode, we will need to have a unique hostname at CONNECT_REST_ADVERTISED_HOST_NAME. However, if we deploy the connector on AWS with an auto-scale group, there is no known hostname for that, not sure how can I do the setup?

carfield
  • 2,011
  • 2
  • 15
  • 15

1 Answers1

0

You can achieve this scenario using the following steps in Pre-created EC2 instance.

1. Configure rest.advertised.host.name property connect-distributed.properties file

rest.advertised.host.name=hostname

2. Create kafka-connect.service file

nano /etc/systemd/system/kafka-connect.service

[Unit]
Description=Kakfka-connect
After=network.target

[Service]
User=ubuntu
Group=ubuntu
Environmet="KAFKA_HEAP_OPTS=-Xmx4G -Xms2G"
Environment="KAFKA_OPTS=-javaagent:/home/ubuntu/prometheus/jmx_prometheus_javaagent-0.16.1.jar=8080:/home/ubuntu/prometheus/kafka-connect.yml"
ExecStart=/home/ubuntu/kafka/kafka_2.13-2.7.0/bin/connect-distributed.sh /home/ubuntu/config/connect-distributed.properties

[Install]
WantedBy=multi-user.target

3. Now take the snapshot of this EC2 instance's volume.

4. Create launch configuration

Use the snapshot as volume. and use following User data.

#!/bin/bash

apt-get update
apt-get -y upgrade
sed -i "s/hostname/$(hostname -I)/g" /home/ubuntu/config/connect-distributed.properties
systemctl start kafka-connect
systemctl enable kafka-connect

sed command will replace the 'hostname' string in connect-distributed.properties file for rest.advertised.host.name with instance's private IP when new instance starts from Auto-scaling group.

Achyut Vyas
  • 471
  • 1
  • 6
  • 18
  • HI, but Pre-created EC2 instance is not auto-scale isn't it? it is auto-heal isn't it? – carfield Dec 14 '21 at 22:20
  • Pre-created EC2 instance is for configuration purpose ( you have to configure this instance, and use it's volume snapshot for auto-scale instance), this pre-created EC2 instance will not be the part of auto-scaling group. – Achyut Vyas Dec 15 '21 at 08:36
  • Thanks, there is this line: [sed -i "s/hostname/$(hostname -I)/g" /home/ubuntu/config/connect-distributed.properties], will the sed command run before the auto-scale host get created? Will the config file updated when build and then the auto-scale group spin a new instance that use the outdated config? – carfield Dec 15 '21 at 23:21
  • That line is in user data for EC2 instance, basically user data is the file which gets execute after spinning up EC2 instance. In short this sed command will run after your new EC2 instance will be created. ( I'm using the same approach to scale kafka-connect service using auto-scaling group ) – Achyut Vyas Dec 16 '21 at 04:53
  • @carfield, does this answer helped you to resolve the issue, or is there problem you are facing to implement this. If yes then please let me know I'll update my answer accordingly. – Achyut Vyas Dec 20 '21 at 05:47
  • Thanks, the answer help and I come up with some other script doing similar thing – carfield Dec 21 '21 at 11:45