Schema for json message in aiokafka

Question

How can I add a schema for json message in aiokafka? Kafka Connect cannot work without it.

import asyncio
import json
import random

import aiokafka
from faker import Faker


def serializer(value):
    return json.dumps(value).encode()


async def produce():
    fake = Faker(['ru_RU'])
    producer = aiokafka.AIOKafkaProducer(bootstrap_servers='localhost:9092', value_serializer=serializer)
    await producer.start()

    try:
        while True:
            message = {
                'name': fake.first_name(),
                'surname': fake.last_name(),
                'age': random.randint(20, 30)
            }
            await producer.send("mytopic", message)
    finally:
        await producer.stop()


asyncio.run(produce())

score 1 · Accepted Answer · answered Jul 12 '21 at 21:41

Kafka Connect cannot work without it

It can if you have value.converter.schemas.enable=false. Whether you need a schema depends on specific connectors, not Connect itself.

If you are not able to change your serializer, for example, to the Avro one from the confluent_kafka library, then you need to provide schema and payload as part of each message on your own

e.g.

message = { 'schema': { ... }, 
            'payload': {
              'name': fake.first_name(),
              'surname': fake.last_name(),
              'age': random.randint(20, 30)
            }
        }

More info: Kafka Connect Deep Dive – Converters and Serialization Explained

Schema for json message in aiokafka

1 Answers1