0

I am having a below API

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=0

Have to extract more data by increasing the offset into 2000

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=2000

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=4000

http://example.com/?mdule=API&method=Live.getLastVisitsDetails&filter_limit=2000&filter_offset=6000

I have a maximum of 6000 records. I don't know how to pass the offset value of every 2000 in the data factory

I can able to extract individually with the above links but wanted to do it automatically for all 6000 records.

Can anyone show me some documentation or advise how to execute this in the datafactory?

I saw the pagination documentation, but no success

sthambi
  • 197
  • 2
  • 17

1 Answers1

0

Extracting data using copy activity from rest API

Step1: Create a new pipeline and add a Copy Data activity.

Step2: Configure the source of copy activity, adding a pagination rule configured as below

Ref1

Follow this sample URL and also make sure ={offset} at the end of URL.

Ref2

Pagination rule either select option1 or option2 .In my case I selected Range:0:8:2

Ref3

In your scenario you can follow the range as below :

  • Option1: QueryParameter.{offset}: Range:0:6000:2000

  • Option2: AbsoluteUrl.{offset}:Range:0:6000:2000

Range option are using 0 as the start value,6000 as the max value, and increasing by 2000 each time.

Ref4

Ref5

Storage account

Ref6

For more detail information refer this official article.

B. B. Naga Sai Vamsi
  • 2,386
  • 2
  • 3
  • 11