2

I'm trying to list all the blobs in my container. There are more than 250,000 blobs in the container. Is it possible to list all those blobs inside the logic app ?

This is the layout I tried

I was able to get 100,000 using the pagination threshold in one but with the other one I had the duplicate blobs. The next page marker didn't help. The next page marker worked with listing 5000 blobs. So I had 10,000 unique blobs.

Max pagination threshold is 100,000

Is there a way to list all those files in logic app ?

Gaurav Mantri
  • 128,066
  • 12
  • 206
  • 241
Rahul Shokeen
  • 23
  • 3

1 Answers1

3

Is there a way to list all those files in logic app ?

No, it's impossible.

You could set the Threshold as 10000 for which the maximum page size for List blobs is 5000 and the marker default maximum result is 5000.

Next page marker: Used to page container results when not all results were returned by the Storage service either due to the default maximum results allowed (the current default is 5000) or because you specify the maxresults parameter in the URI. The marker prefix is opaque to the client application.

So you can create loops that repeat workflow actions in logic app.

Joey Cai
  • 18,968
  • 1
  • 20
  • 30
  • 1
    I ran list blobs until where the Next Marker returned null. I still had duplicate blobs returned. Not sure what I'm missing here. The main objective is to parse those files as JSON and inserting them into a database. Logic App limits to 100,000 blobs. I already have those in the database now need to get other blobs too. – Rahul Shokeen Jul 20 '20 at 06:06
  • Yes, when next marker return null, the blob is all listed. – Joey Cai Jul 20 '20 at 06:08
  • I checked again, the next marker disappears when I do pagination on any of the List Blob actions. So the loop didn't work because of this. I still can't find a way to list these blobs. I found that sharepoint list also have similar limits. But it has filter query which helps in getting the next items. This works in getting the rows from sql database as well. https://alextofan.com/2019/08/22/how-to-get-more-than-5000-item-from-sharepoint-online-in-flow/ Is there a similar solution like this which can work for blob? – Rahul Shokeen Jul 21 '20 at 00:00
  • It worked seems like there was an issue with coalesce, using null expression worked. Thank you. – Rahul Shokeen Jul 21 '20 at 05:16
  • Have a similar issue. Can you update your question/answer of exactly what you did that worked (how did you get that `Next page marker`)? – Prathamesh dhanawade Feb 10 '21 at 17:37
  • I have the same issue -- when I call list blobs with pagination and a threshold of 5000, I do not get a Next page marker in the output. – Rajesh Aug 27 '21 at 02:48