Azure Table Storage sync between 2 different storages

Question

I have a list of storage accounts and I would like to copy the exact table content from source_table to destination_table exactly how it is. Which mean if I add an entry to source_table that will be moved to the destination_table same think if I delete the entry from source I want it to be deleted from destination.

So far I have in place this code:

source_table = TableService(account_name="sourcestorageaccount",
                                     account_key="source key")
destination_storage = TableService(account_name="destination storage",
                                    account_key="destinationKey")

query_size = 1000

# save data to storage2 and check if there is lefted data in current table，if yes recurrence
def queryAndSaveAllDataBySize(source_table_name, target_table_name, resp_data: ListGenerator,
                              table_out: TableService, table_in: TableService, query_size: int):
    for item in resp_data:
        tb_name = source_table_name
        del item.etag
        del item.Timestamp
        print("INSERT data:" + str(item) + "into TABLE:" + tb_name)
        table_in.insert_or_replace_entity(target_table_name, item)
    if resp_data.next_marker:
        data = table_out.query_entities(table_name=source_table_name, num_results=query_size,
                                        marker=resp_data.next_marker)
        queryAndSaveAllDataBySize(source_table_name, target_table_name, data, table_out, table_in, query_size)

tbs_out = table_service_out.list_tables()
print(tbs_out)

for tb in tbs_out:
    table = tb.name
    # create table with same name in storage2
    table_service_in.create_table(table_name=table, fail_on_exist=False)

    # first query
    data = table_service_out.query_entities(tb.name, num_results=query_size)
    queryAndSaveAllDataBySize(tb.name, table, data, table_service_out, table_service_in, query_size)

As you can see this block of code up runs just perfectly, it loops over the source storage account table and creates the same table and its content in destination storage account. but I am missing the part of how I can check if a record has been deleted from the source storage and remove the same record from the destination table.

I hope my question/issue is clear enough, and if not please just ask me for more informations.

Thank you so much for any help you can provide

UPDATE: The more a think about this the more the logic get messy. One of the solution that I thought about and tried is to have 2 lists to store every single table entity:

Source_table_entries
Destination_table_entries

Once I have populated the lists for each run I can compare the partition keys and if a partition key is present in Destination_table_entries but on in source, that will me promoted to be deleted.

But this logic will work flawless as long as I have a small table, unfortunately some table contains hundreds of thousands of entities (and I have hundreds of storages) which sooner or later will become a mess to managed.

So one of the solution that I thought about. Is to keep the same code I have above and just create a new table every week and delete the older one (from the destination storage). For example

Table week 1
Table week 2
Table week 3 (this will be deleted)

I read around that I could potentially add a metadata to the table for date and leverage that to decide which table should be deleted based on date time. But I cannot find anything in the documentation.

Can anyone please direct me on the best approach for this. Thank you so much, I am loosing my mind on this last bit

Azure Table Storage sync between 2 different storages

0 Answers0