How to read a parquet file in Azure Databricks?

Question

I have few parquet files stored in my storage account, which I am trying to read using the below code. However it fails with error as incorrect syntax. Can someone suggest to me as whats the correct way to read parquet files using azure databricks?

val data = spark.read.parquet("abfss://containername@storagename.dfs.core.windows.net/TestFolder/XYZ/part-00000-1cf0cf7b-6c9f-41-a268-be-c000.snappy.parquet")
display(data)

hi @ZZZSharePoint, can you please share the error image, and also please confirm the above files are in delta format or not ? — B. B. Naga Sai Vamsi, Oct 08 '22 at 10:12

score 1 · Answer 1 · answered Oct 10 '22 at 07:37

abfss://containername@storagename.dfs.core.windows.net/TestFolder/XYZ/part-00000-1cf0cf7b-6c9f-41-a268-be-c000.snappy.parquet

As per the above abfss URL you can use delta or parquet format in the storage account.

Note: If you created delta table, part file creates automatically like this part-00000-1cf0cf7b-6c9f-41-a268-be-c000.snappy.parquet.As per above code it is not possible to read parquet file in delta format .

I have written the datafram df1 and overwrite into a storage account with parquet format.

df1.coalesce(1).write.format('parquet').mode("overwrite").save("abfss://<container>@<stoarge_account>.dfs.core.windows.net/<folder>/<sub_folder>")

Scala

val df11 = spark.read.format("parquet").load("abfss://<container>@<stoarge_account>.dfs.core.windows.net/demo/d121/part-00000-tid-2397072542034942773-def47888-c000.snappy.parquet")
display(df11)

python

df11 = spark.read.format("parquet").load("abfss://<container>@<stoarge_account>.dfs.core.windows.net/demo/d121/part-00000-tid-2397072542034942773-def47888-c000.snappy.parquet")
display(df11)

Output:

enter image description here

How to read a parquet file in Azure Databricks?

1 Answers1