0

I am trying to read the data from a paginated API using ADF copy activity. While accessing the API at first time, only the base URL is used(without any query params). When API returns the data, it will have a key called cursor with some token. This has to be passed as query string to get the next page. How to implement this using ADF?. How to append "?cursor=" in the base URL and then add cursor value from API response

First Page : {{Base URL}} #returns a JSON with a key called cursor
From Second Page : {{Base URL}}?cursor = {{value of cursor key in JSON Response}}

I have tried using Query Parameters inside the Copy Activity of pagination. But it is not working

Rathesh
  • 1
  • 3

1 Answers1

0

You can use web activity to the cursor value and pass it to the relative URL in copy activity.

  1. You can first make a call with the base URL using web activity.

enter image description here

  1. Parameterize the relative URL in the dataset.

enter image description here

  1. Connect web activity to copy data activity and pass the web activity output required value to the copy activity dataset parameters to run the URL with cursor value.

Ex: @concat('?cursor=',activity('Web1').output.cursor)

enter image description here

NiharikaMoola-MT
  • 4,700
  • 1
  • 3
  • 15
  • Thanks for the answer. But the Web Activity has a limitation to fetch only less than 4 MB of data. Data I am trying to read for each page goes beyond that. So this approach won't work – Rathesh Jun 18 '22 at 11:23