How to write to a folder in a Azure Data Lake container using Delta?

Question

How do I write to a folder in an Azure Data Lake container using Delta?

When I run:

    write_mode = 'overwrite'
    write_format = 'delta'
    save_path = '/mnt/container-name/folder-name'
     
    df.write \
        .mode(write_mode) \
        .format(write_format) \
        .save(save_path)

I get an incompatible format error. This does not happen when I write directly to the container ('/mnt/container-name').

I have also tried writing using the SQL commands but I get the same error.

What am I doing wrong?

it looks like you have some data in the folder... – Alex Ott Apr 26 '22 at 11:46 — Alex Ott, Apr 26 '22 at 11:46

score 0 · Answer 1 · answered Apr 27 '22 at 08:34

Check weather in this path is there any data Available or not /mnt/container-name/folder-name and also check storage connection between the data lake and data bricks .

Otherwise Create mount and Create a storage connection between the data lake and data bricks.Follow below steps.

spark.conf.set("fs.azure.account.key.blobstoragename.dfs.core.windows.net",dbutils.secrets.get(scope="scopename",key="keyvalue"))

Create mount

dbutils.fs.mount(
source = "wasbs://containername@blobstoragename.blob.core.windows.net",
mount_point = "/mnt/iotd/fgh",
extra_configs = {"fs.azure.account.key. blobstoragename.blob.core.windows.net":" past Access key "})

For example, to overwrite the data in a table you can:

df.write.format("delta").mode("overwrite").save("/mnt/container-name/folder-name")

Reference:

https://docs.databricks.com/data/data-sources/azure/adls-gen2/azure-datalake-gen2-get-started.html

Table batch reads and writes - Azure Databricks | Microsoft Docs

Table batch reads and writes — Delta Lake Documentation

https://www.youtube.com/watch?v=cbobqI3ZGuA

score 0 · Accepted Answer · answered Apr 28 '22 at 11:15

It turns out that you cannot save to a folder within a delta table. The mistake that I made was I saved the delta table directly to the container and then tried to save another delta table to a folder within this container.

When I saved to a folder within a container that only had folders in it, I did not get this problem.

How to write to a folder in a Azure Data Lake container using Delta?

2 Answers2