Is there any way to copy incremental data from SQL Server to Blob Storage Through Azure Data Factory?

Question

I am trying to copy incremental data from Azure SQL to Blob Storage but it overwrite all the existing data in every file. Can you provide me any work around for this problem

Hello, and welcome. What code are you using to copy this data? Can you include it in your question? — Jeremy Caney, Mar 20 '20 at 08:20

score 0 · Answer 1 · answered Mar 20 '20 at 08:28

0

Your pipeline will need to only capture 'delta' data rather than all data. It will require a number of lookup activities, copy activity which only copies delta data and a stored proc activity for updating the last modified date (to be used for future copying)

Check out: https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-powershell

answered Mar 20 '20 at 08:28

thebernardlim

755
2
10
23

Thanks, But it will create a seperate blob file for every delta load in container...I need only one .txt file which contain its's existing as well as delta data in it. – Somya Gupta Mar 20 '20 at 09:28

score 0 · Answer 2 · answered Apr 07 '20 at 07:24

0

Create new file in each delta load and after that append delta file data to original file. For append file data you use this link How to Append a Text File in an Azure Blob with a Azure Function.

answered Apr 07 '20 at 07:24

NP007

659
8
21

Is there any way to copy incremental data from SQL Server to Blob Storage Through Azure Data Factory?

2 Answers2