1

I have three different Azure Storage blob containers that are used to serve website content, one for production, one for staging and one for development.

My goal is to sync staging and development on a daily or weekly basis so they match production.

I put together something in PowerShell that works in principal but it's slow and heavy handed. It involves deleting the staging and dev containers, then copying the entire production container twice. For just 20k items, this whole process takes over an hour.

Considering only a couple hundred items may change in a week, a delta operation would likely be done much faster.

Has anyone seen a tool or method that can do a delta copy between Azure Storage blob containers? I'll likely end up writing my own tool but wanted to see if there are any out there currently.

Kevin Denham
  • 131
  • 1
  • 6
  • had the same demand before and end up scripting a PowerShell runbook for this just like you, but yea, the drawback is that it's pretty slow when working with many files. I couldn't find any solution to do the job, but if you do, please post the answer here. – Bruno Faria Feb 13 '18 at 02:45
  • We used a variant of this: http://toolheaven.net/post/How-to-do-a-fast-recursive-local-folder-tofrom-azure-blob-storage-synchronization.aspx – Arithmomaniac Jan 07 '19 at 23:15

3 Answers3

1

Unfortunately there isn't a good way to do this. As you have already seen tools like AZCopy will move the files for you, but they won't do a delta copy or sync.

You could do something yourself in PowerShell that ran through the blobs and checked if they exist in the destination and then compare something like the modified date or file hash, but I can't imagine that will be particularly quick either unfortunately.

Sam Cogan
  • 38,736
  • 6
  • 78
  • 114
1

I revisited the AzCopy tool today and found it's come quite a long way and serves this purpose perfectly. The flag is AzCopy Sync and there are a number of control options as well, such as deleting destination files which don't exist on the target.

Probably due to improvements in the Azure architecture or infrastructure, this transfer is also incredibly fast. It did an initial sync of 10GB in less than 60 seconds. Previously this would take an hour or more.

Kevin Denham
  • 131
  • 1
  • 6
0

I know it's a bit of an old post but I just came across it and thought I'd throw an idea into the pot, is there a reason it needs to run daily/weekly? You could use something like a Logic App with a blob storage trigger or a function with a blob trigger? That would trigger instantly so the storage accounts are always in sync.

The other option would be to use the triggers to insert a delta into a storage account/database and then process the changes en-mass daily/weekly

Manatherin
  • 101
  • 2