How to add an enum value to an AVRO schema in a FULL compatible way?

Question

I have an enum in an AVRO schema like this :

{
    "type": "record",
    "name": "MySchema",
    "namespace": "com.company",
    "fields": [
        {
            "name": "color",
            "type": {
                "type": "enum",
                "name": "Color",
                "symbols": [
                    "UNKNOWN",
                    "GREEN",
                    "RED"
                ]
            },
            "default": "UNKNOWN"
        }
    ]
}

When using FULL (which means BACKWARD and FORWARD) compatibility mode, how am I supposed to add a new symbol to the enum ? Is this impossible ?

I read Avro schema : is adding an enum value to existing schema backward compatible? but it doesn't help.

Whenever I try to add a new value to the symbols it fails the compatibility check in the schema registry even though I have a default value on the enum. After testing a bit it seems that adding a new value is BACKWARD compatible but not FORWARD compatible. However, due to the default value I set I expected it to be also FORWARD compatible. Indeed the old reader schema should be able to read a value written by the new schema and default to the "UNKNOWN" enum value when it doesn't know the new symbol.

score 5 · Accepted Answer · answered Feb 04 '22 at 15:38

5

It appears there is currently a bug in AVRO which affects the versions 1.9.0, 1.10.0, 1.9.1, 1.9.2, 1.11.0, 1.10.1, 1.10.2 and further until it is fixed.

The bug is in avro handling of enum default value.

According to the documentation on the reader side with an old schema, we should be able to deserialize a payload containing an enum value that was generated by the writer side having the new schema. Since the value is unknown to the reader it should be deserialized as the default value.

A default value for this enumeration, used during resolution when the reader encounters a symbol from the writer that isn't defined in the reader's schema

However thats not what happen and the deserializer on the reader side fails with the exception org.apache.avro.AvroTypeException: No match for C.

I have reported the bug here, and a pushed a reproduction test here

Hope it attracts some attention from the maintainers :)

answered Feb 04 '22 at 15:38

singe3

2,065
4
30
48

2

The default value need to be added in the `type` definition, that's why it's not working in your example. – Jouramie Sep 06 '22 at 18:49
@JérémieBolduc The issue is that I'm using Java classes annotated with avro annotations to generate avro schemas. It does not seem to be possible to specifiy an enum's default value at the class level :/ – singe3 Sep 07 '22 at 19:30
1

That's unfortunate... Other technologies like [Avro IDL](https://avro.apache.org/docs/1.10.2/idl.html) support it. Using the right plugins, it allows you to generate the avro schemas and the Java source code. The issue is probably more with your annotation processor and not really with avro itself. – Jouramie Sep 08 '22 at 20:35

score 2 · Answer 2 · answered Aug 26 '22 at 10:23

2

We can use the symbol level defaults to achieve this, (by moving default inside the type definition). Hope this helps

{
"type": "record",
"name": "MySchema",
"namespace": "com.company",
"fields": [
    {
        "name": "color",
        "type": {
            "type": "enum",
            "name": "Color",
            "symbols": [
                "UNKNOWN",
                "GREEN",
                "RED"
            ],
           "default": "UNKNOWN"
        }
    }
 ]
}

answered Aug 26 '22 at 10:23

ChamaraL

351
4
7

The issue is that I'm using Java classes annotated with avro annotations to generate avro schemas. It does not seem to be possible to specifiy an enum's default value at the class level – singe3 Sep 07 '22 at 19:32

score 0 · Answer 3 · answered Jan 18 '22 at 12:33

0

Adding new symbol into an enum is not FULL compatible , not even FORWARD compatible.

see ==> https://github.com/confluentinc/schema-registry/issues/880

answered Jan 18 '22 at 12:33

elfelli

21
4

This issue is old (older than avro 1.9.0 in fact). Since avro 1.9.0 enum have default values, see https://avro.apache.org/docs/1.9.0/spec.html#Enums. "default: A default value for this enumeration, used during resolution when the reader encounters a symbol from the writer that isn't defined in the reader's schema (optional)". So this seems to me that there is a bug. Why am I denied adding a symbol to an enum which has a default value ? The spec explictely allows it and explain how to handle the unknown symbol on the consumer side. – singe3 Jan 18 '22 at 18:58
you are right ! i didn't pay attention that was an old issue ! i agree it look like a bug , should maybe try to create an issue on gitlhub – elfelli Jan 18 '22 at 20:38
1

I have created the issue https://issues.apache.org/jira/browse/AVRO-3313 with a test reproducing the error at https://github.com/idkw/avro/commit/7d36203c137aa6a728d5b85b87969a3f743b45ee – singe3 Jan 19 '22 at 14:02

How to add an enum value to an AVRO schema in a FULL compatible way?

3 Answers3