0

I am trying to create an MSK connector using kafka-connect-aws-s3-kafka-2-8-4.0.0.zip, but it is stuck in ** Creating...** status and it fails with an unknown error. So a bit about my connector config: I am trying to deploy this Connector in order to sink data from MSK to s3 ( kafka-connect-aws )

here are the connector properties:

connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector
behavior.on.null.values=ignore
s3.region=eu-central-1
flush.size=5
tasks.max=8
topics=MyTopic
s3.part.size=5242880
connect.s3.vhost.bucket=true
schema.enable=false
format.bytearray.separator=\n--==SEPARATOR==--\n
key.converter.schemas.enable=false
connect.s3.kcql=INSERT INTO kafka-connect-aws2:evelin-msk SELECT * FROM MyTopic WITH_FLUSH_INTERVAL = 300
format.class=io.confluent.connect.s3.format.bytearray.ByteArrayFormat
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
value.converter.schemas.enable=false
connect.s3.aws.region=eu-central-1
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
storage.class=io.confluent.connect.s3.storage.S3Storage
errors.log.enable=true
s3.bucket.name=kafka-connect-aws

Any idea what could be the reason? Please let me know if I need to provide more information

MBA
  • 353
  • 2
  • 11
  • 'unknown error' - what do you exactly see? is there any error message? – Ermiya Eskandary Dec 16 '22 at 14:42
  • And how are you looking to authenticate? – Ermiya Eskandary Dec 16 '22 at 14:45
  • so the error is: There is an issue with the connector Code: UnknownError.Unknown Message: The last operation failed. Retry the operation. – MBA Dec 16 '22 at 20:33
  • and for the authentication I am using a dedicated role for MSK where it has the needed policies: { "Effect": "Allow", "Principal": { "Service": "kafkaconnect.amazonaws.com" }, "Action": "sts:AssumeRole" } – MBA Dec 16 '22 at 20:34
  • Did you solve the error? How did you manage to debug it? – Dhananjai Pai Jan 04 '23 at 15:35
  • Some of the errors are shown as "unknown" in the UI, but you should be able to find the error in the logs. You could send the logs to S3 with something like `--log-delivery '{"workerLogDelivery":{"s3":{"bucket":"my-bucket","enabled":true,"prefix":"my-connector-logs"}}}'` and check what's happening. If it's in "Creating..." for a long time, like 20 minutes, I'd say you are having a timeout somewhere. Check that your MSK Connect has access to Kafka and to S3, you might need to check security groups and IAM policies and roles. – Javier Montón Jan 11 '23 at 10:17

1 Answers1

0

When you create the Kafka connector, you have choice to write the log.

For example, you can write to S3. This should help further locate your issues.

enter image description here

Hongbo Miao
  • 45,290
  • 60
  • 174
  • 267