0

I am using Kafka connector to write kafka topic data into s3 bucket in the form of csv. I am using the following curl command

curl -X POST \
   localhost:8084/connectors \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -d '{
  "name": "S3SinkConnector",
  "config": {
   "connector.class": "io.confluent.connect.s3.S3SinkConnector",
  "s3.region": "",
  "flush.size": "1000",
  "topics": "mytopic.value",
  "tasks.max": "1",
  "aws.secret.access.key": "",
  "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
  "aws.access.key.id": "",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "value.converter.schemas.enable": "false",
  "storage.class": "io.confluent.connect.s3.storage.S3Storage",
  "s3.bucket.name": "",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "key.converter.schemas.enable": "false"
  }
}'

No error logs seen in the kafka connector logs. But I can't see topic data written into s3 bucket. Can anyone point me to what is wrong ?

user3279893
  • 147
  • 3
  • 16
  • Did you check the /status endpoint? Is there data in the topic? Your `format.class` isn't set to write as CSV, anyway – OneCricketeer Apr 06 '21 at 19:33
  • @OneCricketeer there are data in the topic and s3 bucket is available. even with format.class its notwriting to s3 bucket – user3279893 Apr 07 '21 at 01:11
  • Sure, and the `/status` connector endpoint is not saying it `FAILED`? Overall, my point was that CSV from JSON records isn't possible, so I think you should definitely change the format class to be the plaintext one – OneCricketeer Apr 07 '21 at 01:29

0 Answers0