Azure Synapse mapping dataflow - REST source dataset not paginating from ITGlue API

Question

This is my first post here so forgive me if I'm in the wrong place.

I'm running a mapping data flow in Azure Synapse to query ITGlue's REST API. It is only returning one page of results, vs all of them.

Here's the endpoint I'm querying: https://api.itglue.com/configurations

Here's the response body for that endpoint (truncated to keep it brief):

{
    "data": [
        {
            "id": "1234567",
            "type": "configurations",
            "attributes": {
               ......
            },
            "relationships": {
                "adapters-resources": {
                    "data": []
                }
            }
        }
    ],
    "meta": {
        "current-page": 1,
        "next-page": 2,
        "prev-page": null,
        "total-pages": 1000,
        "total-count": 1000,
        "filters": {}
    },
    "links": {
        "self": "https://api.itglue.com/configurations?page%5Bnumber%5D=1&page%5Bsize%5D=1",
        "next": "https://api.itglue.com/configurations?page%5Bnumber%5D=2&page%5Bsize%5D=1",
        "last": "https://api.itglue.com/configurations?page%5Bnumber%5D=1714&page%5Bsize%5D=1"
    }
}

Here's what I think is the relevant configuration for ITGlue:

Dataset source options

As far as I can tell, this is the correct syntax for the pagination rule. The only thing I can think that is messing this up is the characters in the 'next' link, which are http encoded [ and ] characters.

The IT Glue API docs for this endpoint here confirm this - with page[number] instead.

Has anyone had this issue before?

Here's what I've tried with pagination rules in Azure Synapse - all to no success (dataflow only returns one page of data)

1. AbsoluteUrl - Body - {links.next} (pictured)
2. AbsoluteUrl - Body - links.next
3. AbsoluteUrl - Body - $.{links.next}
4. AbsoluteUrl - Body - ['links']['next']
5. AbsoluteUrl - None - body.{links.next}
6. AbsoluteUrl - None - body.links.next
7. Query - page%5Bnumber%5D - Body - {meta.next-page}
8. Query - page[number] - Body - {meta.next-page}

When testing this behavior with Postman or Powershell Invoke-RestMethod, it seems to work correctly.

Answer 1

I tried to reproduce your scenario in my environment with your API I am getting some error so tried with my sample API.

My sample API returns the response as,

{{"_embedded":
.
.
.},
"_links":{
"first":{"href":"https://www.ebi.ac.uk/ols/api/ontologies?page=0&size=20"},
"last":{"href":"https://www.ebi.ac.uk/ols/api/ontologies?page=13&size=20"},
"next":{"href":"https://www.ebi.ac.uk/ols/api/ontologies?page=1&size=20"},
"self":{"href":"https://www.ebi.ac.uk/ols/api/ontologies"}},
"page":{"number":"0","size":"20","totalElements":"280","totalPages":"14"}},
"headers":{.......}

In Pagination rules I gave AbsoluteUrl value as body._links.next.href and in Document form I gave Document per line.

With the above settings I am able to retrieve all the pages from My sample API.

In your API, the pagination should look like

Output