I am trying to connect to Celonis EMS using EmsSinkConnector using Kafka-Connect in distributed mode. I ran kafka in distributed mode using below command.
nohup bin/connect-distributed.sh config/connect-distributed.properties >> kafka-connect-distributed.log 2>&1 &
Below are the connect-distributed.properties
group.id=cbm-connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.topic=***
offset.storage.replication.factor=1
config.storage.topic=***
config.storage.replication.factor=1
status.storage.topic=**
status.storage.replication.factor=1
offset.flush.interval.ms=10000
listeners=HTTP://:8083
connector.client.config.override.policy=All
plugin.path=/home/centos/downloads/celonis-kafka-connect-ems-v1.3.4/lib
bootstrap.servers=***
security.protocol=SSL
ssl.truststore.location=***
ssl.truststore.password=***
ssl.keystore.location=***
ssl.keystore.password=***
ssl.key.password=****
After I ran the Kafka-Connect in distributed mode, I am trying to add the Celonis EMS connector using curl command.
curl -i -X PUT -H "Content-Type:application/json" \
http://***:8083/connectors/kafka-celonis-ems-connector/config \
-d '{"name":"kafka-celonis-ems-connector","connector.class":"com.celonis.kafka.connect.ems.sink.EmsSinkConnector","tasks.max":"1","key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","topics":"test","allow.auto.create.topics":"true","connect.ems.endpoint":"***","connect.ems.target.table":"test","connect.ems.connection.id":"","connect.ems.commit.size.bytes":"10000000","connect.ems.commit.records":"1000","connect.ems.commit.interval.ms":"30000","connect.ems.tmp.dir":"/tmp/ems","connect.ems.authorization.key":"","connect.ems.error.policy":"RETRY","connect.ems.max.retries":"20","connect.ems.retry.interval":"60000","connect.ems.parquet.write.flush.records":"1000","connect.ems.debug.keep.parquet.files":"false","key.converter.schemas.enable":"false","value.converter.schemas.enable":"false","errors.tolerance":"all"}'
It says created with 201 status.
HTTP/1.1 201 Created
Date: Thu, 09 Mar 2023 10:12:35 GMT
Location: http://***:8083/connectors/kafka-celonis-ems-connector
But this is not created when I ran the connectors list API. Always empty. No idea where this connector is getting created.
curl -s -XGET http://***:8083/connectors
[]
Any help on what I am doing here? No errors in those logs. How to debug what I am doing wrong in the connector config?
Thank you.
