0

I want to evolve my AVRO schema with an optional field. I add this on my microservice file (.avsc)

{
  "name": "optionalFieldExample",
  "type": [
    "null",
    "string"
  ]
}

And my topic also have the schema updated with forward compatibility on kafka cluster and the schema-registry is configurated and working on my microservice producer configuration: enter image description here

In another yaml file I have the configuration that remains unchanged across environments: enter image description here

When I send a message I have this error:

Error retrieving Avro schema. Schema not found; error code: 40403

enter image description here Why?

If I change my AVRO schema with this:

{
  "name": "optionalFieldExample",
  "type": [
    "null",
    {
      "type": "string",
      "avro.java.string": "String"
    }
  ]
}

The last AVRO schema works. I don't know the reason. Are these two options valid? What do I need to change for the first option to work?

What producer configuration is recommended when setting up an AVRO optional field?

With "use.latest.version: true" this AVRO snippet works:

{
  "name": "optionalFieldExample",
  "type": [
    "null",
    "string"
  ]
}

But I can't apply "use.latest.version = true" because I have many producers, and this setup would require me to update the (.avsc) file all at once instead of updating the producers progressively. However, with "use.latest.version = false", producers can be updated progressively, allowing both updated and non-updated producers to coexist but with this configuration I have the error: Error retrieving Avro schema. Schema not found; error code: 40403.

davidleongz
  • 155
  • 2
  • 11
  • `Schema not found` is not a compatibility error. What endpoint are you using to register? – OneCricketeer Aug 14 '23 at 23:46
  • I'm using a URL on producer properties, yes, is a valid URL, because it's working if I send a invalid message but my question is why doesn't work with the first way and why work with the second way... – davidleongz Aug 16 '23 at 12:14
  • @OneCricketeer I updated the question, please could you take a look at it whenever you can? Thanks! – davidleongz Aug 16 '23 at 13:43
  • Do you have `cross.optional.field4.test-value` subject in the registry? Where are your key and value serializers defined? This is how you define the registry address https://stackoverflow.com/a/75113916/2308683 (you're missing `spring` and don't need to nest the property strings, therefore may default to access localhost:8081) – OneCricketeer Aug 16 '23 at 14:56
  • Also, you should be able to test the Schema Registry API without Kafka producer, which is why I removed the tags, but you should clarify what version of spring-kafka and registry client +server you're using since both your shown schemas are equivalent and your error is with the API call, not the schema or producer config – OneCricketeer Aug 16 '23 at 15:02
  • 1
    Yes I have the subject in the registry, I can see with Control Center and with this URL: "server:8081/subjects/cross.optional.field.test4-value/versions/" only has 2 versions. – davidleongz Aug 17 '23 at 07:33
  • I updated the yml configuration on the question. I'm using "spring-kafka" version 2.6.4 and "org.apache.avro" version 1.11.0 and "io.confluent.kafka-avro-serializer" version 7.0.1 – davidleongz Aug 17 '23 at 07:54
  • I've configurated my producer with use.latest.version = true, and it works with this optional field: { "name": "textoOld", "type": [ "null", "string" ] }. But I don't know what is the correct form of evolve an AVRO with an optional field... and if is necessary use "use.latest.version = true" producer configuration always... What producer configuration is recommended when setting up an AVRO optional field? – davidleongz Aug 17 '23 at 11:01
  • I can't apply "use.latest.version = true" because I have many producers, and this setup would require me to update the (.avsc) file all at once instead of updating the producers progressively. However, with "use.latest.version = false", producers can be updated progressively, allowing both updated and non-updated producers to coexist but with this configuration I have the error: Error retrieving Avro schema. Schema not found; error code: 40403. – davidleongz Aug 17 '23 at 12:54
  • The only way for it to work for me is by setting the optional field like this: { "name": "textoOld", "type": [ "null", { "avro.java.string": "String", "type": "string" } ] } – davidleongz Aug 17 '23 at 12:54
  • I'd recommend using at least 7.3.1 version of the serializer. Avro 1.8 is used in the old versions... Meanwhile, spring-kafka has had many upgrades as well – OneCricketeer Aug 17 '23 at 12:55
  • I've tried with "kafka-avro-serializer" version 7.4.0, and "org.apache.avro" with version 1.11.2 and it hasn't worked either... – davidleongz Aug 17 '23 at 14:05
  • If you trace the serializer function, you may find the record has no ID. Otherwise the log would be different and include it (I added that feature for debugging). Source - https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/rest/exceptions/Errors.java#L126 so, maybe check the registry server logs to see if the schema is being denied for some other reason – OneCricketeer Aug 18 '23 at 13:29

0 Answers0