0

I'm working in producer side of kafka to push message in topic. I'm using confluent-kafka avro producer.

Liked issue on github

Below are the my schema .avsc files.

Keys.avsc

{
    "namespace": "io.codebrews.schema.test",
    "type": "record",
    "name": "Keys",
    "fields": [
        {
            "name": "name",
            "type": "string"
        },
        {
            "name": "email",
            "type": "string"
        }
    ]
}

Test.avsc

{
    "namespace": "io.codebrews.schema.test",
    "type": "record",
    "name": "Subscription",
    "fields": [
        {
            "name": "test",
            "type": "string"
        },
        {
            "name": "keys",
            "type": "io.codebrews.schema.test.Keys"
        }
    ]
}

Producer.py

key_schema, value_schema = load_avro_schema_from_file('Subscription.avsc')

try:
    producer = avro.AvroProducer(producer_config, default_key_schema=key_schema, default_value_schema=value_schema)
except Exception as e:
    raise e

def load_avro_schema_from_file(schema_file):
    key_schema_string = """
    {"type": "string"}
    """

    key_schema = avro.loads(key_schema_string)
    value_schema = avro.load("./avro/" + schema_file)

    return key_schema, value_schema

When I try to register Keys.avsc it works fine with no error. But when I try to register Test.avsc after registring Keys.avsc. I get below error.

confluent_kafka.avro.error.ClientError: Schema parse failed: Unknown named schema 'io.codebrews.schema.test.Keys', known names: ['io.codebrews.schema.test.Subscription'].

After registering the schema manually.

{
    "namespace": "io.codebrews.schema.test",
    "type": "record",
    "name": "Subscription",
    "fields": [
        {
            "name": "test",
            "type": "string"
        },
        {
            "name": "keys",
            "type": "Keys"
        }
    ]
}

When push message in my topic I get below error.

ClientError: Incompatible Avro schema:409 message:{'error_code': 409, 'message': 'Schema being registered is incompatible with an earlier schema for subject "test-value".

Am I doing something wrong here?

Also can anyone help me how to stop auto schema registration in python?

Avi
  • 1,424
  • 1
  • 11
  • 32

1 Answers1

0

The error is related to the parser, not the registry registration...

Your AVSC files need to be fully inclusive of all record types. When the parser reads one file, it doesn't have a way to know about the others

If you start with an AVDL file, then convert that into AVSC, the records do properly get embedded in the outer records.

Specifically,

"fields" :[{
  "name": "keys",
  "type": {
       "type": "record", 
       "namespace": "io.codebrews.schema.test.Keys", 
       ... 
  } 
}
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Yeah got your point, I have updated my question after manually registering my schema on confluent-kafka cloud and error I get while pusing message to topic.. Is there any way to use that and make it work with my code, because there are many other applications which are in java are able to use that schema and push message to given topic, As they are already using it, so it won't be possible to change the schema on cloud. – Avi Sep 24 '21 at 10:08
  • `Keys` is not a valid Avro type, so you should expect the same error. I'd need to see the schema already in the registry to tell you what's not compatible – OneCricketeer Sep 24 '21 at 12:03
  • No, I just registered my `Keys.avsc` manually with `io.codebrews.schema.test.Keys` and `Test.avsc` with `-value`, and it gave me schema that I have updated newly in question. Like I didn't change anything in any of my `avsc` file while registering it manually. – Avi Sep 24 '21 at 12:45
  • I stopped using the Registry before support for references were added, and [issues have been posted about it](https://github.com/confluentinc/schema-registry/issues/426) but Python's `avro.load` function has no connection to that – OneCricketeer Sep 24 '21 at 18:05
  • Is there any python reference for the same, above link looks like java lib. – Avi Sep 25 '21 at 09:28
  • Like I said, `avro` or `fastavro` python modules are external from the actual registry functions. If you are getting any error about an incompatible schema, it's not really clear to me in what order you've registered your schemas, but a brand new subject/topic name wouldn't have that error and I don't think Python confluent library supports different naming strategies like the Java library – OneCricketeer Sep 25 '21 at 09:31