Spark streaming Dynamic Schema Evolution from Kafka Eventhub on Microbatch

Question

We are streaming data from the Kafka Eventhub. The records may have a nested structure. The schema is inferred dynamically from the data and the Delta table is formed with the schema of the first incoming batch of data.

Note: The data read from Kafka topic will be a whole JSON string. Hence,

• When we apply schema and convert to a dataframe, we lose the fields' values with mismatch datatype or newly added fields.
• When we do spark.read.json, the entire field values are converted to String.

We encounter a situation where the Source data has some schema changes. Some of the scenarios we faced are :

1. The datatype changes at the parent level
2. The datatype changes at the nested level
3. There are duplicate keys in a different case
4. There are the addition of new fields

A sample Source data with the Actual schema

{
  "Id": "101",
  "Name": "John",
  "Department": {
    "Id": "Dept101",
    "Name": "Technology",
    "EmpId": "10001"
  },
  "Experience": 2,
  "Organization": [
    {
      "Id": "Org101",
      "Name": "Google"
    },
    {
      "Id": "Org102",
      "Name": "Microsoft"
    }
  ]
}

A sample Source data addressing the 4 points mentioned above

{
  "Id": "102",
  "name": "Rambo",   --- Point 3
  "Department": {
    "Id": "Dept101",
    "Name": "Technology",
    "EmpId": 10001         ---- Point 2
  },
  "Experience": "2",   --- Point 1
  "Organization": [
    {
      "Id": "Org101",
      "Name": "Google",
      "Experience": "2",  --- Point 4
    },
    {
      "Id": "Org102",
      "Name": "Microsoft",
      "Experience": "2",
    }
  ]
}

We need a solution to overcome the above issues. Though it's difficult to embed the new schema to the existing delta table, at least we should be able to separate the records with schema changes without losing the original data.