Azure Data Factory only returns the first 100 pipeline runs

Question

I have an Azure Data Factory instance and I am trying to read all pipeline runs from the last day or so. It's working, but I can't seem to get more than 100 results back from the api. I believe from what I've googled that I am reading the AsyncPageable correctly, but it seems like I am only getting one page back from the api. Here's the code:

ResourceIdentifier dataFactoryResourceId = DataFactoryResource.CreateResourceIdentifier(subscriptionId, resourceGroupName, factoryName);
DataFactoryResource dataFactory = _armClient.GetDataFactoryResource(dataFactoryResourceId);     

RunFilterContent content = new RunFilterContent(DateTimeOffset.UtcNow.AddDays(-1).AddHours(-1), DateTimeOffset.MaxValue);
content.OrderBy.Add(new RunQueryOrderBy(RunQueryOrderByField.RunStart, RunQueryOrder.Desc));

List<FactoryPipelineRunInfo> runs = new List<FactoryPipelineRunInfo>();
await foreach (FactoryPipelineRunInfo item in dataFactory.GetPipelineRunsAsync(content))
{
    runs.Add(item);
}

score 1 · Answer 1 · answered Mar 09 '23 at 16:58

I found a workaround, so sort of an answer. I guess the SDK is still prerelease(1.0.0-beta2) at the time I'm writing this, so it's probably something that will be addressed in the future.

I ended up using an HttpClient and making the calls directly to the API. Each response contains a continuation token that when passed back in subsequent calls gets the next page of data, until you get to the last page and the continuation token comes back null. I'm guessing the SDK doesn't handle the token correctly.

In case anyone is interested, here's the code I wrote to do that:

 string subscriptionId = _config.GetValue<string>("Azure:SubscriptionId");
 string resourceGroupName = _config.GetValue<string>("Azure:DataFactoryResourceGroupName");
 string factoryName = _config.GetValue<string>("Azure:DataFactoryName");
 var http = _httpFactory.CreateClient("AzureManagement");
 string url = $"https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01";
 JsonObject body = new JsonObject
 {
   { "lastUpdatedAfter", DateTimeOffset.UtcNow.AddDays(-1).AddHours(-1).ToString() },
   { "lastUpdatedBefore", DateTimeOffset.MaxValue }
 };
 PipelineRunResponseDTO? runInfoResponse = null;
 List<PipelineRunDTO> runs = new List<PipelineRunDTO>();
 do
 {
   if (!string.IsNullOrWhiteSpace(runInfoResponse?.ContinuationToken))
   {
     body["continuationToken"] = runInfoResponse.ContinuationToken;
   }
   var postBody = new StringContent(body.ToString(), Encoding.UTF8, "application/json");
   var response = await http.PostAsync(url, postBody);
   if (response.IsSuccessStatusCode)
   {
     string responseStr = await response.Content.ReadAsStringAsync();
     runInfoResponse = JsonSerializer.Deserialize<PipelineRunResponseDTO>(responseStr);
     if (runInfoResponse != null)
     {
       runs.AddRange(runInfoResponse.Value);
     }
   }
 } while (!string.IsNullOrWhiteSpace(runInfoResponse?.ContinuationToken));
 return runs;

PipelineRunResponseDTO is a class I wrote that has the same properties as FactoryPipelineRunInfo, but I couldn't use FactoryPipelineRunInfo because all it's constructors are marked as internal.

Azure Data Factory only returns the first 100 pipeline runs

1 Answers1