1

I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to our SQL Data Warehouse. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.

The API I am trying to pull data from is Cisco Meraki (https://developer.cisco.com/meraki/api-v1/#!introduction/meraki-dashboard-api) which is a REST API that uses the RFC 5988 pagination method. This means it passes the pagination rules for the next page in a response header.

I run the following API call

{{baseUrl}}/networks/:networkId/events?productType=wireless&perPage=1000

which passes the following as a link header response to dictate the next page

<{{baseUrl}}/networks/:networkId/events?endingBefore=2021-10-19T13%3A03%3A06.769680Z&perPage=1000&productType=wireless>; rel=prev, <{{baseUrl}}/networks/:networkId/events?perPage=1000&productType=wireless&startingAfter=2021-10-19T13%3A44%3A01.586776Z>; rel=next

How can I implement the above in the pagination rules in ADF to get the next page of info? I've currently defined the rules as here: ADF Pagination Rules but it's not accessing any new information. No error message just still only the first page.

Thanks for any pointers,

user17192499
  • 11
  • 2

1 Answers1

0

I know this post is a bit old, but I thought I would comment for others.

I have found that when processing a response that conforms to , datafactory will process the pagination if you do nothing. i.e. leave the pagination rules empty. This will then add "paginationRules": {"supportRFC5988": "true"} to the source Json.

I have tested this and verified it reads all pages.

Roman Sinyakov
  • 559
  • 1
  • 6
  • 24
gbarr42
  • 1