Automatically Delete/Expire Azure Blobs after a time period

Question

With Azure Blob storage is it possible to either have an individual blob or all blobs within a container delete themselves after a certain period of time similar to Amazon AWS S3's Object Expiration Feature? Or does Azure storage not provide such functionality?

Answer 1

It is possible with the Azure Blob storage lifecycle. Please take a look here

https://learn.microsoft.com/en-us/azure/storage/blobs/storage-lifecycle-management-concepts?tabs=azure-portal

and Configure a lifecycle management policy

Answer 2

Because I've missed the feature for years I wrote a small project with a nice 'Deploy to Azure button'. Not yet perfect but works https://github.com/nulllogicone/ExpireBlobFunction

And now I see that Microsoft has released this as a feature on March 27, 2019.

Excerpt from that article:

Azure Blob storage lifecycle management offers a rich, rule-based policy for GPv2 and Blob storage accounts. Use the policy to transition your data to the appropriate access tiers or expire at the end of the data's lifecycle.

The lifecycle management policy lets you:

• Transition blobs to a cooler storage tier (hot to cool, hot to archive, or cool to archive) to optimize for performance and cost
• Delete blobs at the end of their lifecycles
• Define rules to be run once per day at the storage account level Apply rules to containers or a subset of blobs (using prefixes as filters)

Answer 3

You can auto-delete in various ways. Since long time you can do it even with Logical App but, sometimes, it is not so clear. Today you have it directly available in your storage account:

and there you have an easy and specific task generator (template) to delete old blobs.

Answer 4

Looks like the feature is planned now at least: https://feedback.azure.com/forums/217298-storage/suggestions/2474308-provide-time-to-live-feature-for-blobs

Answer 5

The Azure Storage Team recently posted (Oct 5, 2017) an update on expiring blobs. It seems that this is now possible using an Azure Logic App template and they will have a native blob storage solution later this year.

Link: Provide Time to live feature for Blobs

We are pleased to announce that we have made an Azure Logic Apps template available to expire old blobs. To set up this automated solution in your environment: Create a new Logic Apps instance, select the “Delete old Azure blobs” template, customize and run. We will release a blog post detailing instructions and providing more templates in the coming weeks.

Allowing users to define expiration policies on blobs natively from storage is still planned for the coming year. As soon as we have progress to share, we will do so. We will continue to provide updates at least once per quarter.

For any further questions, or to discuss your specific scenario, send us an email at azurestoragefeedback@microsoft.com.

Answer 6

Azure Storage does not have an expiration feature; you must delete blobs via your app. How you do this is up to you; you'll need to store your expiration date target somewhere (whether in a database or in blob properties).

You can effectively create TTL on blob access, via Shared Access Signatures (by setting an end-date on the SAS). This would let you have an effective way of removing access when it's time to remove access, and then have a follow-on process remove the now-expired blobs.

Answer 7

Yes, it's possible. Refer these two, it was hard finding out the sample code.

Rules reference: https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview?tabs=azure-portal

Python sample code reference: https://github.com/Azure-Samples/azure-samples-python-management/blob/master/samples/storage/manage_management_policy.py

Code snippet I used:

def add_expiry_rule(self):
        token_credential = ClientSecretCredential(
            tenant_id=tenant_id,
            client_id=client_id,
            client_secret=client_secret,
        )
        storage_client = StorageManagementClient(
            credential=token_credential, subscription_id=subscription_id
        )
        rule = {
            "id": "test",
            "prefix": "test/",
            "expiration": 91,
        }
        azure_rule = {
            "enabled": True,
            "name": rule.get("id"),
            "type": "Lifecycle",
            "definition": {
                "filters": {"blob_types": ["blockBlob"], "prefix_match": [rule.get("prefix")]},
                "actions": {
                    "base_blob": {
                        "delete": {
                            "days_after_modification_greater_than": str(rule.get("expiration"))
                        }
                    }
                },
            },
        }
        try:
            management_policy = storage_client.management_policies.get(
                group_name, storage_account, "default"
            )
            existing_rules = management_policy.policy.as_dict()
            existing_rules.get("rules").append(azure_rule)
            management_policy_rules = existing_rules
        except Exception as e:
            management_policy_rules = {"rules": [azure_rule]}
        try:
            management_policy = storage_client.management_policies.create_or_update(
                group_name,
                storage_account,
                "default",
                {"policy": management_policy_rules},
            )
            print("Azure: Added rule {} successfully".format(rule.get("id")))
        except Exception as e:
            if e.message.endswith("conflicting rule name."):
                print("Azure: Rule ID: {} exists".format(rule.get("id")))
            else:
                raise Exception("Azure: Error adding rule. {}".format(e.message))