0

I'm trying to retrieve list of files and folders form ADLS Gen2. I can get the first 5000 items, but when I use continuation to get the rest (about 17,000 items or so), I get Error 403 (Forbidden). According to documentation, I add the continuation token to URI and to Canonicalized Resource in signature string. However, I cannot get it to work.

I've read the documentation on ADLS Gen2 REST calls and whatever I could find on this, and I can't figure out the issue.

var date = System.DateTime.UtcNow.ToString("R");
string toSign = DefaultSignatureString(date);
toSign +=
    $"/{storageaccountname}/{filesystemname}" + "\n" +
    $"directory:{dir}" +"\n" +
    "recursive:true" + "\n" +
    "resource:filesystem";

var signedSignature = SignData(accessKey, toSign);
var uri = $"https://{storageaccountname}.dfs.core.windows.net/{filesystemname}?directory={dir}&recursive=true&resource=filesystem";

HttpWebResponse response = GetWebResponse(storageaccountname, date, signedSignature, uri);
var token_continuation = response.Headers["x-ms-continuation"];
//I get the token_continuation and repeat the previous steps, adding the continuation part:
while (token_continuation != null)
{
    date = System.DateTime.UtcNow.ToString("R");
    toSign = DefaultSignatureString(date);
    toSign +=
        $"/{storageaccountname}/{filesystemname}" + "\n" +
        $"continuation:{token_continuation}" + "\n" +
        $"directory:{dir}" + "\n" +
        "recursive:true" + "\n" +
        "resource:filesystem";

    signedSignature = SignData(accessKey, toSign);
    uri = $"https://{storageaccountname}.dfs.core.windows.net/{filesystemname}?directory={dir}&recursive=true&resource=filesystem&continuation={token_continuation}";
    response = GetWebResponse(storageaccountname, date, signedSignature, uri);
    token_continuation = response.Headers["x-ms-continuation"];
}


//this is my GetWebResponse method
private static HttpWebResponse GetWebResponse(string storageaccountname, string date, string signedSignature, string uri, string continuation = null)
    {
        WebRequest request = WebRequest.Create(uri);
        if (continuation != null)
        {
            request.Headers.Add($"x-ms-continuation:{continuation}");
        }
        request.Headers.Add($"x-ms-date:{date}");
        request.Headers.Add($"x-ms-version:2018-11-09");
        request.Headers.Add($"Authorization:SharedKey {storageaccountname}:{signedSignature}");
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        return response;
    }

As I said, I get the first response OK; when I get into while loop, I get the error. What am I doing wrong?

gunta
  • 71
  • 6

1 Answers1

1

In most of the cases (if not all) continuation token returns with == at the end, which messes up the URI.

For URIs, escape them by replacing == with %3D%3D.

For canonicalized resources, leave the string as is.

Besworks
  • 4,123
  • 1
  • 18
  • 34
gunta
  • 71
  • 6