1

I have an MSK cluster setup on 2 public subnets in eu-west-1 region. I have a topic named 'awsuseravro1' and Glue Schema Registry with name 'msk-registry' and schema name same as the topic name. I have created an MSK connector with a custom plugin which contains latest kafka-connect-s3 plugin. I also manually copied to the lib folder of this plugin schema-registry-kafkaconnect-converter 1.1.5 version in order for aws glue avro converter configs to work. I uploaded this custom zip to s3, created a custom plugin and launched a connector with following configs:

connector.class=io.confluent.connect.s3.S3SinkConnector
s3.region=eu-west-1
flush.size=1
schema.compatibility=FULL
tasks.max=2
topics=awsuseravro1
format.class=io.confluent.connect.s3.format.avro.AvroFormat
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
storage.class=io.confluent.connect.s3.storage.S3Storage
s3.bucket.name=hfs3testing
topics.dir=avrotopics1
key.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter
value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter
key.converter.region=eu-west-1
value.converter.region=eu-west-1
key.converter.schemaAutoRegistrationEnabled=true
value.converter.schemaAutoRegistrationEnabled=true
key.converter.avroRecordType=GENERIC_RECORD
value.converter.avroRecordType=GENERIC_RECORD
key.converter.schemaName=awsuseravro1
value.converter.schemaName=awsuseravro1
key.converter.registry.name=msk-registry
value.converter.registry.name=msk-registry

My msk cluster has no authentication and is plaintext. The connector starts but I get a following error: https://pastebin.com/zUAXgsDP

I also gave the connector role all the required s3 policies and full access to aws glue registry. When I run some custom producers and consumers that I wrote on my EC2 instance which use the glue registry it works great. When I use S3 connector with JSON format without glue registry with plain text topic it saves to the bucket normally

Konrad Paniec
  • 11
  • 1
  • Having the same issue with Debezium mysql connector. Even I created VPC endpoint for glue as Im using private subnet. You can check my full post https://github.com/awslabs/aws-glue-schema-registry/issues/118 Do let me know if you able to solve it – Snigdhajyoti Dec 27 '21 at 18:46
  • Fix the issue by opening NAT gateway for private subnets. My assumption is, other than glue MSK connect is trying to connect to some other AWS resources. Because VPC endpoint didn't work. – Snigdhajyoti Dec 29 '21 at 16:29

0 Answers0