1

I've been trying to create an MSK Connector and use Glue schema registry with it. The configuration is as follows.

connector.class=io.confluent.connect.s3.S3SinkConnector
s3.region=eu-west-1
topics.dir=topics/dir
flush.size=200
tasks.max=2
s3.part.size=5242880
timezone=GMT
# value.converter.schema.registry.url=http://someIP:8081
key.converter.schemaName=my-topic-schema
locale=US
format.class=io.confluent.connect.s3.format.parquet.ParquetFormat
value.converter.schemaName=my-topic-schema
value.converter=io.confluent.connect.avro.StringConverter
s3.bucket.name=my-bucket
key.converter=io.confluent.connect.avro.StringConverter
# key.converter.schema.registry.url==http://someIP:8081
partition.duration.ms=3600000
schema.compatibility=BACKWARD
topics=osb
value.converter.registry.name=Glue-Schema-Registry
key.converter.registry.name=Glue-Schema-Registry
key.converter.schemas.enable=true
partitioner.class=io.confluent.connect.storage.partitioner.TimeBasedPartitioner
value.converter.schemas.enable=true
storage.class=io.confluent.connect.s3.storage.S3Storage
rotate.schedule.interval.ms=0
path.format='year'=YYYY/'month'=MM/'day'=dd/'hour'=HH
timestamp.extractor=RecordField
timestamp.field=timestamp

First I was using the confluent schema registry running on an EC2 whose ip I added in "key/value.converter.schema.registry.url" field and it was working fine. Now I'm trying to use Glue Schema Registry. But I don't know how to connect the connector with Glue Schema Registry.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Rohit
  • 35
  • 6

1 Answers1

1

These classes don't exist

key.converter=io.confluent.connect.avro.StringConverter
value.converter=io.confluent.connect.avro.StringConverter

The StringConverter classname starts with org.apache.kafka

Similarly, looks like you've added a bunch of random converter properties, which aren't valid for the String or Confluent Avro converter, apart from the url

To use Glue, you'll need to use AWSKafkaAvroConverter, which is part of this repo,

https://github.com/awslabs/aws-glue-schema-registry/tree/master/avro-kafkaconnect-converter

And is documented here

https://docs.aws.amazon.com/glue/latest/dg/schema-registry-integrations.html#schema-registry-integrations-apache-kafka-connect

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245