How to set multiple schemas for topic using confluent_kafka python client?

Question

I'm using confluent_kafka==2.2.0 in my python project. I want to set multiple schemas for a topic. I'm reading docs for AvroSerializer, it says that it has config options to set RecordNameStrategy (see docs here https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#avroserializer). Also in the releases of the lib it says it started to support non-default schema strategies (see here https://github.com/confluentinc/confluent-kafka-python/releases/tag/v1.4.0). However, I'm stuck with actually doing it because of 2 reasons:

I have several .avsc files with schemas. How do I provide several schemas to AvroSerializer? It receives only 1 argument for schema.
confluent_kafka.schema_registry.record_subject_name_strategy receives 2 args: ctx and record_name, what should I pass there? My code looks like following:

import asyncio

from confluent_kafka.admin import AdminClient, NewTopic

from confluent_kafka import SerializingProducer, DeserializingConsumer
from confluent_kafka.serialization import StringSerializer, StringDeserializer
from confluent_kafka.schema_registry import record_subject_name_strategy, SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer, AvroDeserializer

from config import Config


# Utility functions
def create_admin(config: dict):
    return AdminClient(config)


async def create_new_topic(admin: AdminClient, topic_name: str):
    if topic_name in admin.list_topics().topics:
        print("Already exist!")
        return

    futures = admin.create_topics([
        NewTopic(topic_name, num_partitions=1, replication_factor=1),
    ])

    await asyncio.wrap_future(futures[topic_name])
    print("Topic created!")


# Main function
async def main():
    admin = create_admin(Config.KAFKA)
    await create_new_topic(admin, "test_multischema")

    schema_registry_client = SchemaRegistryClient(Config.SCHEMA_REGISTRY)
    
    # Reading schema strings
    with open("event1.avsc", "r") as f:
        schema_1_str = f.read()
    with open("event2.avsc", "r") as f:
        schema_2_str = f.read()

    # Here is the problem
    producer_config = Config.KAFKA | {
        "key.serializer": StringSerializer(),
        "value.serializer": AvroSerializer(
            schema_registry_client,
            # This serializer should be able to deserialize messages of both schema_1 and schema_2.
            # How should I write config to pass both to one producer?
            schema_1_str,
            conf={
                # What args should I pass?
                "subject.name.strategy": record_subject_name_strategy()
            }
        )
    }
    producer = SerializingProducer(producer_config)
    # After I will produce msgs and then consume them

Maybe someone could provide code snippets how to configure both SerializingProducer and DeserializingConsumer so that they would be able to read from the topic with these 2 schemas?

Tried finding solution in the docs, but it either states that it is not supported yet (old ones) or states only about the strategy and provides no example on how it actually should look like in code (like here How to change SubjectNameStrategy and use different schemas in confluent-kafka-python's AvroProducer?).

How to set multiple schemas for topic using confluent_kafka python client?

0 Answers0