2

Need to calculate size of specific containers and folders at ADLS Gen2. Started with command az storage fs file list. However don't understand how to grab next_marker ? It appears in stdout as warning but not in output of command:

WARNING: Next Marker:
WARNING: VBbYvMrBhcCCqHEYSAAAA=

So how to get this next_marker:

$files=$(az storage fs file list --file-system <container name>\
 --auth-mode login --account-name <account name> \
 --query "[*].[contentLength]" --num-results 1000 -o tsv)

$files.next_marker is empty.

UPDATE1: Created issues https://github.com/Azure/azure-cli/issues/16893

Alezis
  • 2,659
  • 3
  • 27
  • 34

1 Answers1

3

If you're using this azure cli command: az storage fs file list, the next_marker is not returned to the variable $files, it's always printed out in the console. You need to copy and paste it.

As a workaround, you can use this azure cli command: az storage blob list(Most of the azure blob storage cli commands are also available in ADLS Gen2). This command has a parameter --show-next-marker, you can use it to return next_marker to a variable.

I write an azure cli scripts and it can work well for ADLS Gen2:

 $next_token = ""
 $blobs=""
 $response = & az storage blob list --container-name your_file_system_in_ADLS_Gen2 --account-name your_ADLS_Gen2_account --account-key your_ADLS_Gen2_key --num-results 5 --show-next-marker | ConvertFrom-Json
 $blobs += $response.properties.contentLength
 $next_token = $response.nextMarker 

  while ($next_token -ne $null){
    
  $response = & az storage blob list --container-name your_file_system_in_ADLS_Gen2 --account-name your_ADLS_Gen2_account --account-key your_ADLS_Gen2_key --num-results 5 --marker $next_token --show-next-marker | ConvertFrom-Json
  
  $blobs = $blobs + " " + $response.properties.contentLength

  $next_token = $response.nextMarker 
  }

$blobs

The test result:

enter image description here

Please note that upgrade your azure cli to the latest version, the --show-next-marker parameter may not work in the old versions as per this issue.

Ivan Glasenberg
  • 29,865
  • 2
  • 44
  • 60
  • Yes, thanks. From my experience `az storage fs file list` works much faster. Another thing that `az storage blob list` will be deprecated in new releases. Looking into code found that next marke is logged as a warning. It looks like nonsense for me. https://github.com/Azure/azure-cli/blob/8aedd2d631699b422a1a525fb3101284c37ad942/src/azure-cli/azure/cli/command_modules/storage/operations/fs_file.py#L79 – Alezis Feb 10 '21 at 09:10
  • @Alezis how do you know if the `az storage blob list` will be deprecated in new releases? – Ivan Glasenberg Feb 10 '21 at 09:18
  • I execute it and see warning: "This command has been deprecated and will be removed in future release. Use 'az storage fs file list' instead. For more information go to https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/azure/cli/command_modules/storage/docs/ADLS%20Gen2.md The behavior of this command has been altered by the following extension: storage-preview" `{ "azure-cli": "2.18.0", "azure-cli-core": "2.18.0", "azure-cli-telemetry": "1.0.6", "extensions": { "interactive": "0.4.4", "storage-preview": "0.7.0" } }` – Alezis Feb 10 '21 at 10:38
  • @Alezis First, I didn't see such warning in the latest version of azure cli `2.19.0` and `2.19.1`. Second, I've already ping the owner of azure cli, the similar parameter `--show-next-marker` [will be added soon](https://github.com/Azure/azure-cli/issues/13704#issuecomment-776509621). Currently, you can just use it as a workaround, or you can send the output(the nextMarker) to a variable, and fetch it by yourself. If the answer is helpful, could you please accept it as answer? Thanks. – Ivan Glasenberg Feb 11 '21 at 06:39