Using Confluent Schema Registry with MSK

Question

Is it possible to integrate Confluent Schema Registry with AWS MSK? If you have done this before, can you please provide some pointers / blogs you followed to achieve it?

score 3 · Accepted Answer · answered Dec 29 '21 at 16:18

It is possible. My setup uses ec2 and docker.

Download the IAM auth jar if you are using IAM based auth

mkdir -p /usr/share/java/aws
wget -P /usr/share/java/aws https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
chmod -R 444 /usr/share/java/aws

Use confluent official docker image for schema registry

...

  schema-registry:
    image: confluentinc/cp-schema-registry:5.4.6-1-ubi8
    hostname: schema-registry
    container_name: schema-registry
    ports:
      - "8081:8081"
    volumes:
      - /usr/share/java/aws/aws-msk-iam-auth-1.1.1-all.jar:/usr/share/java/cp-base-new/aws-msk-iam-auth-1.1.1-all.jar
      - /usr/share/java/aws/aws-msk-iam-auth-1.1.1-all.jar:/usr/share/java/rest-utils/aws-msk-iam-auth-1.1.1-all.jar
    environment: # https://docs.confluent.io/platform/current/schema-registry/installation/config.html#schemaregistry-config
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_HOST_NAME: "${HOSTNAME}" # 
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "${BOOTSTRAP_BROKERS_SASL_IAM}"
      SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: "SASL_SSL"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM: "AWS_MSK_IAM"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG: "software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;"
      SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

...

HOSTNAME is your ec2 machine DNS name or IP, example ip-10-0-0-84.ec2.internal
BOOTSTRAP_BROKERS_SASL_IAM is comma separated host1:port,host2:port urls. For port information see this

If you are using PLAINTEXT or SSL auth, last 4 environment variable changes. And you don't have to download iam auth jar

Configure source or sink connector with these property

...
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schemas.enable=true
value.converter.schema.registry.url=http://ip-10-0-0-84.ec2.internal:8081
value.converter.enhanced.avro.schema.support=true

Thats it.
Do open 8081 port in your security group of EC2 instance for MSK cluster

Resource:

Alternate option I have tried is AWS Glue Schema registry But we had to use KSQL, and KSQL does't have 3rd party schema registry integration or custom SerDe Github issue

This is great, thanks! I got it to work. I'm now trying to enable HA (multiple instances of SR) running on EKS. Have you tried this before? — guru, Dec 29 '21 at 19:46
No. I configured this today itself. Don't have need to configure multiple instances of SR. But maybe in near future we need to. Once you done, please do post some blog or resource here. Also I need some help on MSK connect to configure sink connector. If you can join on this chat room https://chat.stackoverflow.com/rooms/240550/aws-msk-config we can discuss more — Snigdhajyoti, Dec 29 '21 at 21:11

score 1 · Answer 2 · answered Dec 28 '21 at 05:49

1

It is possible and is no different than using it with regular Kafka installation.

You point its bootstrap server property at MSK and you point client applications at it.

answered Dec 28 '21 at 05:49

OneCricketeer

179,855
19
132
245

Thanks but I was looking at the Schema Registry service that Confluent Kafka has. MSK by itself is fine. When I try to provide the 'schema.registry.url' property, I get a warning as such: ```2021-12-27 15:18:10.886 WARN 65921 --- [nio-7869-exec-1] o.a.k.clients.producer.ProducerConfig : The configuration 'schema.registry.url' was supplied but isn't a known config.``` So `KafkaProducer` isn't able to understand what schema registry is and will not use the same for saving/registering schemas. I want to know if there's a hard dependency for Schema Registry on Confluent installation. – guru Dec 28 '21 at 16:55
1) "Confluent Kafka" is not a thing. MSK and Confluent.Kafka.IAdminClient.html Platform include the same Apache Kafka 2) You need to use specific serializers for the `schema.registry.url` property to work; it's not a "producer config", it's a serializer one. So, which one are you using? – OneCricketeer Dec 28 '21 at 20:45
1

Yeah, I got it to work with MSK. Thank you. – guru Dec 29 '21 at 19:47

Using Confluent Schema Registry with MSK

2 Answers2