How to remove all files in Azure BLOB storage

Question

Background: I have a system which works with a database where I keep metadata of files and Azure Blob storage where I keep files. The database and Azure Blob Storage work together via web-services.

To check that all parts of the system work I created unit tests to web-services which download, upload and remove files. After testing, the database and Azure Blob storage keep a lot of data and I need to remove all of them. I have a script to remove all data from the database (Drop all the tables, stored procedures, triggers, constraints and all the dependencies in one sql statement).

Now I need to write a sctipt (power shell) or code (C#) to remove all files from Azure Blob storage, but I do not remove containers, only files in the containers.

My questions: Which of these ways (power shell or С#) are the best ? If I use C# and tasks(System.Threading.Tasks) to remove files it will be faster?

Is there a reason why you don't want to delete the container? IMHO, it would be much-much faster and inexpensive to remove the container and recreate it in a little while. — Gaurav Mantri, May 17 '16 at 11:36
When I remove containers and try recreate them in a few seconds I usually get exception, that container exists, so I decide to remove only files. — Bushuev, May 17 '16 at 11:40
There is no "best" way. But then again, we have barely any info regarding the extent of your tests (10's of blobs? 100's? Millions?). And as @GauravMantri questioned: Why **not** delete your container? Containers are easy to re-create later (and by deleting, pretty much guarantees you are starting from a clean test setup when creating a new container later). — David Makogon, May 17 '16 at 11:42
I agree that it takes some time to recreate a container by the same name as the deletes are not synchronous. However there's no such functionality which will clear the container (i.e. delete all files in a single call). You would need to list all files in a container and delete them one by one. This is time consuming, expensive + error prone as you're making multiple network calls to list and delete the files. — Gaurav Mantri, May 17 '16 at 11:43
In the future I think to extand this method. Imgae when users remove some files that they are removed by using tasks, because of containers can be used by other users. — Bushuev, May 17 '16 at 11:52

score 5 · Answer 1 · answered Sep 20 '18 at 10:09

5

I am not sure but, i landed here just to see how come i can delete all the file in blob container in one shot. From azure portal UI, they dont offer any feature to selected all for delete.

Just use Azure Storage Explorer, it has select all functionality for delete. I worked for me.

I know it may not be relevant for the exactly for this question but people like me who wanted to delete manually will find this helpful.

answered Sep 20 '18 at 10:09

Indrajeet Gour

4,020
5
43
70

1

It gives you lists of 1000 files at a time. If you have many more than that, you need the code approach. – João Pedro 'jota' Martins May 18 '20 at 08:43
Yes, this given approach can be used only for testing purpose which is again manual – Indrajeet Gour Dec 14 '20 at 11:11
good enough for what I need (which is testing). thanks for this! – randolfarevalo Feb 17 '22 at 05:11

score 2 · Accepted Answer · answered May 18 '16 at 05:39

The best solution to the problem, if you save titles of the containers, remove them and try to recreate them in a few seconds (if errors ocurr, you need to wait and try again), but if you have to remove only files you can use it:

CloudStorageAccount storageAccount;
CloudBlobClient cloudBlobClient;

//connection is kept in app.config
storageAccount =
    CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
cloudBlobClient = storageAccount.CreateCloudBlobClient();

Parallel.ForEach(cloudBlobClient.ListContainers(), x =>
    {
        Parallel.ForEach(x.ListBlobs(),y=>
            {
                ((CloudBlockBlob)y).DeleteIfExists();
            });
    });

score 0 · Answer 3 · answered Mar 22 '22 at 15:41

Based on Bushuev answer.

2022 UPDATE

Here is my complete class to delete all blobs in all containers (Except the containers in the list on the top of the class)

The unique needed parameter is the "connection string"

public class BlobStorageService : IBlobStorageService
{
    private readonly List<string> _systemContainerNames = new List<string>()
    {
        "azure-webjobs-hosts"
    };

    public async Task CleanAllBlobsInAllContainers(string connectionString)
    {
        CloudBlobClient cloudBlobClient = CloudStorageAccount.Parse(connectionString)
                                                            .CreateCloudBlobClient();

        ContainerResultSegment allContainers = await cloudBlobClient.ListContainersSegmentedAsync(default);

        foreach (CloudBlobContainer container in allContainers.Results)
        {
            if (_systemContainerNames.Any(name => name.Equals(container.Name)))
                continue;

            BlobResultSegment allBlobs = await container.ListBlobsSegmentedAsync(default);

            foreach (CloudBlockBlob blob in allBlobs.Results.OfType<CloudBlockBlob>())
            {
                await blob.DeleteIfExistsAsync();
            }
        }
    }
}

How to remove all files in Azure BLOB storage

3 Answers3