0

We are trying to implement a new solution where we use a Synapse pipeline to copy data from an Azure Databricks Delta Lake to a Synapse dedicated pool. I'm using a copy activity where the source is a dynamically constructed query to be executed against Databricks. The sink is an existing table in the dedicated pool. Copy method is set to Polybase with staging enabled in the settings.

When I run the pipeline I get the following error:

Operation failed: "The specified filesystem does not exist." 404, HEAD.

However, when I copy/paste the exact same query in the expression field using Preview data it runs and gives me the expected output.

We are using the unity catalog in Databricks and I'm worried this may not yet have been properly implemented in Synapse, but I don't understand why the preview would work but the debug does not. Any help would be much appreciated.

Koen van Wielink
  • 67
  • 1
  • 10

1 Answers1

0

I tried to reproduce the error and got same Operation failed: "The specified filesystem does not exist." 404, HEAD. error.

enter image description here

  • The cause of the error is when you copy data from Azure databricks delta lake table to SQL table you cand do it directly.
  • You have to unable staging in Copy activity and there you need to give linked service to particular ADLS gen 2 account or blob storage account.
  • After that you have to pass correct Path (the error is causing because the path you are providing in not available).

To resolve this:

Go to Copy activities Settings >> Enable staging >> Staging settings >> provide Staging account linked service >> provide correct path.

enter image description here

Pratik Lad
  • 4,343
  • 2
  • 3
  • 11
  • Hi Pratik, Thanks for answering. I forgot to update this because we discovered the cause ourselves last week. We already had staging enabled, but for some reason granting rights to the storage account is not sufficient. We had to explicitly create a storage container and grant the necessary ACL rights to the Synapse managed identity for it to work. I'll mark your comment as the answer as it's pretty close to the root cause. – Koen van Wielink Apr 03 '23 at 11:12