0

I need to move my delta lake files to a new blobstore on a different subscription. Any ideas whats the best way to do this?

Im moving them to an ADLS Gen2 Storage, I think the previous storage was just blob storage. This delta lake is updated on an hourly basis by databricks jobs (but I can pause those if necessary). Size is around 3TB-5TB, I'm initially thinking of pausing all jobs and using azcopy to move the files and point the jobs there afterwards. But I want to check other options that may be better in terms of speed of transfer and cost.

Jong
  • 1
  • 2
  • Delta stores a bunch of Parquet and JSON files in a filesystem. Moving Delta files from one filesystem / cloud to another for Delta is the same as moving other files, say CSV files. You could improve this question by adding the current storage system, the file destination, and what you've tried to accomplish the task so far – Powers Jul 12 '20 at 14:46
  • We decided to use azcopy. another requirement we have is we wanted the owner of the destination files to be under a service principal. – Jong Sep 16 '20 at 11:47

1 Answers1

1

The best way would just to use Azure Data Factory. There you can point to your different locations and move the files really quick.

Alex-msi
  • 49
  • 5
  • thanks, going to look at adf and check how it performs. will accept this as an answer once done. – Jong Jul 17 '20 at 10:26