Databricks: autoloader and multiple files with differing schema?

Asked Aug 07 '23 at 03:47

Active Aug 07 '23 at 06:54

Viewed 29 times

I'm following the Databricks Cloud tutorial. I see sample data located at DBFS:

/databricks-datasets/structured-streaming/events
dbfs:/databricks-datasets/structured-streaming/events/file-0.json
dbfs:/databricks-datasets/structured-streaming/events/file-1.json

It looks like these JSON files have the same schema. What happens using spark.readStream.load(path) if above files have different schema? Should they typically have the same schema?

edited Aug 07 '23 at 06:54

Alex Ott

80,552
8
87
132

asked Aug 07 '23 at 03:47

notaorb

1,944
1
8
18

The docs explain what happens. https://docs.databricks.com/en/ingestion/auto-loader/schema.html#how-does-auto-loader-schema-evolution-work – Nick.Mc Aug 07 '23 at 04:16
@notaorb Are you using the azure databricks – DileeprajnarayanThumula Aug 08 '23 at 03:39

Databricks: autoloader and multiple files with differing schema?

0 Answers0