0

I have created a copy data activity in azure data factory and this data pipeline pulls the data from an API (via REST activity source) and writes the response body (json) on a file kept in the azure blob storage. The API which I am fetching the response from, is paginated and the link to next page is sent in the response headers in response->headers->link.

This URL to next page is in the following general format: <https%3A%2F%2FsomeAPI.com%2Fv2%2FgetRequest%3FperPage%3D80%26sortOrder%3DDESCENDING%26nextPageToken%3DVAdjkjklfjjgkl>; rel="next"

I want to fetch the next page token present in the above URL and use it in the pagination rule.

I have tried using some pagination rules:

> AbsoluteURL = Headers.link But, this did not work as the entire encoded link shown above, is getting appended directly and hence, the pipeline throws an error.

> Query Parameters I have also tried to use the query parameters but could not get any result.

I have followed questions over stackoverflow and have read the documentations:

Please help me with how can I access this next page token or what can be the pagination rule to support the scenario.

Pasting postman output and ADF data pipeline, for reference. Postman Response Headers Output Pagination Rules, I need help on

Rubin Shah
  • 1
  • 1
  • [Referred the official documentation here](https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support) I have tried following this official documentation but couldn't get success – Rubin Shah Feb 01 '23 at 08:44
  • Did you ever figure this out? I have a similar issue with an API that uses RFC5998. I have it configured per the documentation (example 7), however, pagnation is not working. – Josh Feb 22 '23 at 21:59
  • @Josh No, so RFC5998 did not work for me. Probably, because the link which I am using is encoded. For now, I am doing the entire pipeline in python rather than using Azure Data Factory. The support for such small issues is not clearly found in the Azure Data Factory. – Rubin Shah Feb 24 '23 at 19:44

1 Answers1

0

AFAIK, there is no direct option to decode the URL in ADF paginating rule.

You can follow below approach:

  • First take a set variable and create temp variable as string and value as 1 enter image description here

  • Then take until activity and give the condition as @equals(variables('temp1'),10) so it will iterate over it till condition matches. enter image description here

  • Under the until loop take another set variable and create variable URL1 as string and value as URL of rest api. enter image description here

  • After this pass variable value to copy activity and copy the first page to destination. enter image description here

  • After this take web activity and pass that URL variable to it and get response headers. enter image description here

  • After this take set variable and update the URL to next page url usnig dynamic expression.

@uriComponentToString(split(split(activity('Web1').output.ADFWebActivityResponseHeaders["link"],'>')[0],'<')[1])

enter image description here

  • In last create a variable called increment as string and increment initial temp variable for every iteration by @add(int(variables('temp1')),1) enter image description here

By this process you can copy as many pages as you want or you set in Until loop as we don't have any specific condition to terminate loop.

Pratik Lad
  • 4,343
  • 2
  • 3
  • 11