0

I have a copy activity that copy on premise sql data to parquet format in data lake gen2. But I need to copy sql data to delta format in the same data lake. I tried using the data flow to copy from parquet to delta but we have performance issues in this approach. Is there any other way we can achieve this?

Ramkumar
  • 57
  • 1
  • 8
  • Can you explain more about the performance issue. When you use Delta as sink the data factory spins-up a compute which usually takes 3 minutes or more. Did you check if it is taking the time to spin up the compute or actual transform process? If it is taking time to spin up then create and use a custom Azure hosted Integration runtime instead of AutiResolve IR. https://blogs.dhrutara.com/blogs/data_factory_ir_minimize_compute_spinup_times talks more about using Custom integration runtimes to minimize compute spin up ties. – Ramesh Kanjinghat Apr 12 '22 at 23:30

1 Answers1

-1

We have different tools to accomplish the required task.

  1. The Copy Data tool
  2. The Azure portal
  3. The .NET SDK
  4. The Python SDK
  5. Azure PowerShell
  6. The REST API
  7. The Azure Resource Manager template

The major requirement is to make perfect configuration regarding access tokens and authentication.

https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-databricks-delta-lake?tabs=data-factory

Check the above link for the procedure.

Sairam Tadepalli
  • 1,563
  • 1
  • 3
  • 11