How to merge multiple datatypes of an union type in avro schema to show one data type in the value field instead of member0 member1

Question

I have the following avro schema

{
  "name": "MyClass",
  "type": "record",
  "namespace": "com.acme.avro",
  "fields": [
    {
      "name": "data",
      "type": 
        {
        "type": "map",
        "values": ["int","string"]
      }
    }
  ]
}

However, when I am streaming some events via kafka to spark with this schema, the streaming data frame depicts the data field as a struct with members with the datatypes specified in the schema as shown in the below image.

Schema format of the dataframe

Is there any possibility to merge the members to only show the value of the key rather than splitting and representing it as multiple members

so like number : -64

rather than number : { member0 : -64 , member1 : null}

score 1 · Accepted Answer · answered Oct 08 '21 at 10:55

1

There isn't any way to merge them and that is how avro works for a union datatype. Might have to live with it.

answered Oct 08 '21 at 10:55

Aiden

41
1

No other way to do it? – Beluga Oct 08 '21 at 11:01
Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Oct 08 '21 at 16:17

How to merge multiple datatypes of an union type in avro schema to show one data type in the value field instead of member0 member1

1 Answers1