0

I'm working on a project for a large company with millions of users. We are attempting to convert their REST based architecture to an event based architecture. The current architecture involves a service, we'll call it Service-A, that makes 7 REST calls when a user logs in.

Rather than calling out to the 7 services for that data when the user logs in we want to modify those 7 services to produce events when there are updates to the data. Then we will have Service-A listen to 7 different kafka topics and save those updates to the database.

It is a Java Spring Boot application. We are using AWS MSK to host our kafka cluster and we are using AWS Glue for the schema registry. I can configure my consumer in Service-A to listen to 7 topics but I don't know how to get Service-A to check 7 different schemas when consuming a message from one of those 7 topics.

So far, the only configuration I've found for the kafka consumer is one property that takes one schema name.

Here is my config yaml:

spring:
  kafka:
    listener:
      ack-mode: manual_immediate
    consumer:
      enable-auto-commit: false
      group-id: my-group
      key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.json.trusted.packages: com.app.somepackage.domain
        spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
        spring.deserializer.value.delegate.class: com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer
      auto-offset-reset: earliest
    bootstrap-servers: <my-msk-url> 
    properties:
      region: us-west-2
      schemaName: my-first-schema
      registry.name: my-registry-name
      avroRecordType: SPECIFIC_RECORD
user3236794
  • 578
  • 1
  • 6
  • 16
  • It's not entirely clear what you are asking since you didn't state which property you are talking about; because you tagged the question with `spring-kafka`, I assume you are using that; in 2.8 we added a `DelegatingByTopicDeserializer`, which you can configure with a different delegate for each topic. https://docs.spring.io/spring-kafka/docs/current/reference/html/#by-topic – Gary Russell Dec 21 '21 at 20:20
  • added my config @GaryRussell – user3236794 Dec 21 '21 at 21:27
  • 1
    Each _record_ within a topic would be associated with its own schema, not specific topics or specific consumer threads. This is to accommodate schema evolution. The `AWSKafkaAvroDeserializer` should handle all this for you – OneCricketeer Dec 21 '21 at 21:52
  • so the producer includes the schema ID in the message automatically? @OneCricketeer – user3236794 Dec 21 '21 at 21:57
  • At least in the Confluent Serializers, yes. I can only assume AWS copied a similar approach – OneCricketeer Dec 21 '21 at 22:00

0 Answers0