0

I'm working on a data integration pipeline in Azure Data Factory, and I need to implement offset pagination for a copy activity. I have an API that returns a response with pagination information, and I want to use that information to perform offset pagination in the copy activity.

Here's an example of the API response:

{
    "log_id": "3133A719-AB38-4846-1111-9D37AD9D682F",
    "pagination": {
        "next_page": 2,
        "last_page": 2428,
        "total_records": 24274,
        "prev_page": 1,
        "current_page": 1
    },
    "data": {
        "ws_fin_out_header": [
            {
                "serial": "TST1400002"
            },
            {
                "serial": "TST1400001"
            }
        ]
    },
    "auth": true,
    "success": true,
    "message": null,
    "uuid": "E0EB1293-1111-4505-A02D-950A6477CF53",
    "version": "4",
    "sqlResult": null,
    "token_succes": true,
    "benchmark": "request finished in 219 ms",
    "token": false
}

I use QueryParameters {pageNumber} = Range(1, 2428,1) but the 2428 is now hardcoded. How do dynamically set the end of the range so that the copy activity stops when all is processed.

I already tried to work with "EndCondition": $.data.ws_fin_out_header = EMPTY. This did not work.

Any ideas how to set the offSet correctly and stop when done?

Anouar
  • 85
  • 5

1 Answers1

1

To achieve your scenario, you can directly pass the value to end condition and stop pagination in range as below:

In your condition it can be like $pagination.total_records

enter image description here

Also, you can refer this SO thread.

Pratik Lad
  • 4,343
  • 2
  • 3
  • 11
  • Thank you. This works fine as long as the API returns data. But the Copy Activity fails when there is no data returned: "data": { "ws_fin_out_header": [] How do I cope with this? – Anouar May 21 '23 at 14:41