I have an existing avro schema that contains a field with a nested map of map of a record type (let's call it RecordA for now). I'm wondering if it's possible to add a new record type, RecordB, to this nested map of maps while maintaining FULL_TRANSIENT compatibility?
My thinking was that as long as the inner maps gets defaulted to an empty map it still adheres to the schema so it's backwards/forward compatible.
I've tried to redefine the type map<map<RecordA>> maps
to map<map<union{RecordA, RecordB}>> maps
in an .avdl
file, but the schema registry is telling me this is not compatible.
I've also tried to default each map individually to an empty map ({ }
) in a generated .avsc
file, but schema registry says that's incompatible as well.
I do want to acknowledge that I know map<map<..>>
is a bad practice, but what's been done has been done.
Registered Schema (original) .avdl
:
record Outer {
map<map<RecordA>> maps;
}
record RecordA {
string value;
string updateTime;
}
Attempt with .avdl
:
record Outer {
map<map<union{RecordA, RecordB}>> maps = {};
}
record RecordA {
string value;
string updateTime;
}
record RecordB {
union{null, array<string>} values = null;
union{null, string} updateTime = null;
}
Attempt with .avsc
:
{
"name" : "maps",
"type" : {
"type" : "map",
"values" : {
"type" : "map",
"values" : [ {
"type" : "record",
"name" : "RecordA",
"fields" : [ {
"name" : "value",
"type" : "string"
}, {
"name" : "updateTime",
"type" : "string"
} ],
"default": { }
}, {
"type" : "record",
"name" : "RecordB",
"fields" : [ {
"name" : "value",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "values",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "updateTime",
"type" : [ "null", "string" ],
"default" : null
} ],
"default": { }
} ]
}
},
"default" : { }
}
The end goal is to have a map
of map
s to a record who has a field that can either be a string
or array<string>
. The original schema was registered to a schema-registry where the field has type string
with no union {}
with null
or a default, so I believe the map needs to be map to a union of types with either version of the field.
Each try has returned the following from the schema-registry compatibility API
{
"is_compatible": false
}
Any insight would be very much appreciated!