1

I have an existing avro schema that contains a field with a nested map of map of a record type (let's call it RecordA for now). I'm wondering if it's possible to add a new record type, RecordB, to this nested map of maps while maintaining FULL_TRANSIENT compatibility?

My thinking was that as long as the inner maps gets defaulted to an empty map it still adheres to the schema so it's backwards/forward compatible.

I've tried to redefine the type map<map<RecordA>> maps to map<map<union{RecordA, RecordB}>> maps in an .avdl file, but the schema registry is telling me this is not compatible.

I've also tried to default each map individually to an empty map ({ }) in a generated .avsc file, but schema registry says that's incompatible as well.

I do want to acknowledge that I know map<map<..>> is a bad practice, but what's been done has been done.

Registered Schema (original) .avdl:

record Outer {
    map<map<RecordA>> maps;
}
record RecordA {
    string value;
    string updateTime;
}

Attempt with .avdl:

record Outer {
    map<map<union{RecordA, RecordB}>> maps = {};
}
record RecordA {
    string value;
    string updateTime;
}
record RecordB {
    union{null, array<string>} values = null;
    union{null, string} updateTime = null;
}

Attempt with .avsc:

{
            "name" : "maps",
            "type" : {
              "type" : "map",
              "values" : {
                "type" : "map",
                "values" : [ {
                  "type" : "record",
                  "name" : "RecordA",
                  "fields" : [ {
                    "name" : "value",
                    "type" : "string"
                  }, {
                    "name" : "updateTime",
                    "type" : "string"
                  } ],
                  "default": { }
                }, {
                  "type" : "record",
                  "name" : "RecordB",
                  "fields" : [ {
                    "name" : "value",
                    "type" : [ "null", "string" ],
                    "default" : null
                  }, {
                    "name" : "values",
                    "type" : [ "null", "string" ],
                    "default" : null
                  }, {
                    "name" : "updateTime",
                    "type" : [ "null", "string" ],
                    "default" : null
                  } ],
                  "default": { }
                } ]
              }
            },
            "default" : { }
}

The end goal is to have a map of maps to a record who has a field that can either be a string or array<string>. The original schema was registered to a schema-registry where the field has type string with no union {} with null or a default, so I believe the map needs to be map to a union of types with either version of the field.

Each try has returned the following from the schema-registry compatibility API

{
    "is_compatible": false
}

Any insight would be very much appreciated!

sstannus
  • 23
  • 3

0 Answers0