3

I have a requirement where I need to migrate data from one azure table storage , basically from one table to other table (Both table could be either in same subscription or different subscription).

Is there any way in Azure table storage to do above requirement like in SQL storage where user can generate scripts or backup of entire database or individual table.

David Makogon
  • 69,407
  • 21
  • 141
  • 189
JARVIS
  • 765
  • 1
  • 8
  • 28
  • Does this answer your question? [How Do I Backup Azure Tables and Blobs](https://stackoverflow.com/questions/35835993/how-do-i-backup-azure-tables-and-blobs) – Paul Roub Jun 18 '20 at 15:29

2 Answers2

2

Short answer: there's no built-in backup. Longer answer:

  • There's no built-in backup service for table storage, or "snapshot" feature (which blobs have): You'll need to do your own copy operation, from one table to another. There's a REST API available along with SDKs, PowerShell cmdlets, CLI commands, and AzCopy (all built upon the API). There are also a bunch of 3rd-party tools you can search for. How you accomplish your table-copying is up to you.

  • Table storage is durable storage - triple-replicated within a region (and optionally geo-replicated to another region). Even if storage became unavailable in the primary region, you'd have the option of reading from the paired region (assuming you enabled your storage account to be geo-redundant). Note: This is not the same as backup - if you delete an entity in a table, that deletion is replicated everywhere

  • Storage-copying (e.g. copying entities out of a table, to another table) will be the same, regardless of subscription. Storage accounts are keyed on account namespace + access key (and optionally SAS).

David Makogon
  • 69,407
  • 21
  • 141
  • 189
1

I realize this is an old question, but I thought I'd leave this response for those who are still looking for ways to back up Azure table data in 2018.

I've seen a lot of suggestions for using AzCopy, which looks like a great way to do it.

However, if using C# works better for you, I wrote a tool (that my workplace allowed me to open source) which is on github: https://github.com/Watts-Energy/Watts.Azure#azure-data-factory

The main objective of the project is not backups, but it can be used to do just that, and we have backups running in Azure Web Jobs using the functionality therein. We open sourced it because we figured it could prove useful to others, besides us, since it allows you to do 'incremental' backups, which I don't know if you can accomplish with AzCopy. I'm not saying you can't, only that I haven't a clue whether that's possible.

The idea is that you create a small console application (to be hosted as an Azure Web Job for example) in .NET and you can e.g. do something like this:

DataCopyBuilder
.InDataFactoryEnvironment(environment)
.UsingDataFactorySetup(environment.DataFactorySetup)
.UsingDefaultCopySetup()
.WithTimeoutInMinutes(numberOfMinutes)
.AuthenticateUsing(authentication)
.CopyFromTable(sourceTable)
.WithSourceQuery(null)
.ToTable(targetTable)
.ReportProgressToConsole()
.StartCopy();

If you, when the job runs, store the time (in UTC) when you started your last copy operation, you can supply a 'source query' (example: 'Timestamp gt datetime'{lastBackupStarted.ToIso8601()}') rather than null like in the example above, and it will only take data modified since that date. It's explained in greater detail in the project README.txt .

Again, not sure if it's useful to anyone, but it does solve some challenges we have had.

kkirk
  • 352
  • 1
  • 7
  • does this library allow different columns per row when backing up an Azure Table or do all rows in the table need to have the same columns? – Francesco Mar 29 '19 at 00:40
  • It does not guarantee it. When backing up a table, it queries the first 1000 rows and determines the columns from that. If all columns appear in the first 1000 rows, everything should be fine, but if not, those columns that did not, will not get copied. As you probably know, Azure table does not have a 'schema', so it's not possible to guarantee all columns will be copied, unless you examine every row in the table. However, if you know that all columns will exist somewhere in the first X rows, clone the Watts.Azure repo and modify the number of rows it examines from 1000 to X. – kkirk Mar 30 '19 at 19:19
  • That's done by modifying the value used in https://github.com/Watts-Energy/Watts.Azure/blob/development/Watts.Azure.Common/Storage/Objects/AzureTableStorage.cs#L244 – kkirk Mar 30 '19 at 19:20
  • I should also add that functionality has since been added which makes running backups a lot easier than described in my original post. You can now use the built-in backup functionality, where you specify a management table (which keeps track of backups) and specify how long they should be retained, how often incremental backups should be run, and when to switch to a new storage account. It's documented (very briefly) in the README. Take a look at https://github.com/Watts-Energy/Watts.Azure/blob/development/Watts.Azure.Tests/IntegrationTests/BackupIntegrationTests.cs if you're interested. – kkirk Mar 30 '19 at 19:28
  • thanks for clarifying how the backup determines which columns to select and for building an awesome library. I'm looking for a solution like yours but one that can backup an Azure Table with a per row schema like azcopy does. – Francesco Mar 30 '19 at 23:20
  • You’re welcome and thank you for that. Not too familiar with azcopy, but sounds interesting that azcopy can actually do that. Would be great to implement something similar in Watts.Azure when time allows. – kkirk Mar 31 '19 at 07:29