0

Currently, I am trying to copy data from Azure Databricks Delta Lake to my Azure Data Lake through Azure Data Factory. I want to copy to a dynamic directory and a dynamic file name, but I keep receiving this error "Direct copying data from Azure Databricks Delta Lake is only supported when sink is a folder, please enable staging or fix File path" however, when I chose a fix file path and a fix container the pipeline works, but I am not able to copying in different directories. Does anyone face something like this? Who can advise.

However, I tried to enable staging and the error is fixed, but I get another error Databricks is not authorized to perform this operation using this permission xxx 403...

user15532579
  • 1
  • 1
  • Please edit to show all relevant details. As written, there's not much to go on - you haven't shared any details of your data factory pipeline, or anything else. – David Makogon Mar 20 '23 at 12:47

1 Answers1

0

I want to copy to a dynamic directory and a dynamic file name

To achieve this, you can use dataset parameters like below. Create dataset parameters for the folder name and file name and give them in the file path.

enter image description here

In copy activity sink, you can give the folder names and file names dynamically as per your requirement.

enter image description here

If you want to copy the delta table to only a dynamic folder, you can use only one parameter for the folder name.

If you want copy to file name as well, then in this case you need to enable the staging and give the staging location.

In above both cases make sure you give the following in the spark config of the databricks cluster(as you mentioned you are copying to ADLS).

spark.databricks.delta.autoCompact.enabled true
spark.databricks.delta.optimizeWrite.enabled true
spark.databricks.delta.preview.enabled true
spark.hadoop.fs.azure.account.key.rakeshgen2.dfs.core.windows.net <Access key>

enter image description here

My Result when copied to folder:

enter image description here

My Result when copied to single file:

enter image description here

but I get another error Databricks is not authorized to perform this operation using this permission xxx 403..

This error occurs when you don't have permission to read or write the data to your Storage account. Make sure you have the required permissions.

Rakesh Govindula
  • 5,257
  • 1
  • 2
  • 11
  • The OP needs to show their work. I would suggest not writing a custom blog post for questions that only say "it's not working." – David Makogon Mar 20 '23 at 12:46
  • thanks @David Makogon, for your insights. I will try to follow those next time. – Rakesh Govindula Mar 21 '23 at 10:25
  • Thank you @DavidMakogon, for taking the time to answer. I see, I am able to create dynamic containers, the problem is was because of the dynamic folder name. There, I was having the problem. Thanks – user15532579 Mar 21 '23 at 16:58