How can I get the Next Link from response header in Okta's API List Users when paginating?

Question

I'm trying to implement Okta's pagination for their API endpoint List Users. It looks like in order to paginate one must get the next link via their incoming headers from their response. When executing their List Users API endpoint via the command line's cUrl or Postman, everything looks great in the header, but the issue is when running it from a PHP script using cUrl or guzzle, the Link html tags are stripped from the header as shown below:

HTTP/1.1 200 OK
Date: Thu, 03 Nov 2016 19:36:34 GMT
Server: nginx
Content-Type: application/json;charset=UTF-8
Vary: Accept-Encoding
X-Okta-Request-Id: WBuTwqhxlYz3iu5PY1jqHQZZBMU
X-Rate-Limit-Limit: 1200
X-Rate-Limit-Remaining: 1198
X-Rate-Limit-Reset: 1478201841
Cache-Control: no-cache, no-store
Pragma: no-cache
Expires: 0
Link: ; rel="self"
Strict-Transport-Security: max-age=315360000

The header should instead looks as:

HTTP/1.1 200 OK
Content-Type: application/json
Link: <https://your-domain.okta.com/api/v1/users?limit=200>; rel="self"
Link: <https://your-domain.okta.com/api/v1/users?   after=00ud4tVDDXYVKPXKVLCO&limit=200>; rel="next"

I searched for a while and can't find a solution. Has anyone come across this issue before? Thanks in advance.

score 2 · Accepted Answer · answered Nov 03 '16 at 22:26

Make sure that you are including the Accept: application/json header in your requests. My guess is that PHP's cURL or Guzzle are using a different MIME type, like text/plain.

I was able to reproduce your issue with the following curl command, which gives no results:

 curl --verbose \
       --header "Authorization: SSWS ${API_KEY}" \
       --header "Content-Type: application/json" \
       --header "Accept: text/plain" \
       "https://${ORG}/api/v1/users?limit=1" 2>&1 | grep 'Link: '

However, if I change the Accept: header to application/json I do get the Link: header:

  curl --verbose \
       --header "Authorization: SSWS ${API_KEY}" \
       --header "Content-Type: application/json" \
       --header "Accept: application/json" \
       "https://${ORG}/api/v1/users?limit=1" 2>&1 | grep 'Link: '

< Link: <https://example.okta.com/api/v1/users?limit=1>; rel="self"
< Link: <https://example.okta.com/api/v1/users?after=012a3b456cdefgHijK7l8&limit=1>; rel="next"

Thank you for the reply. So looks like in my case, we were using php's print_r() to print the header, which for some reason this function strips the link's content from the header. You were right about the MIME type in the header though. Thanks again! — user1370897, Nov 04 '16 at 03:51

score 0 · Answer 2 · answered Nov 15 '16 at 13:46

I found your post while searching for something else, I recently tackled this problem and this is my solution to it. All of this is written in PowerShell.

#Function to automatically get all listings by pagination, this function will use the default Okta Limit parameter. Which is 1000 as the time of this making.
#Invoke-OktaPagedMethod is based on the _oktaRecGet() function from https://github.com/mbegan/Okta-PSModule/blob/master/Okta.psm1
function Invoke-OktaPagedMethod {
    param
    (
        [string]$Uri,
        [array]$col,
        [int]$loopcount = 0
    )

    try {
        #[System.Net.HttpWebResponse]$response = $request.GetResponse()
        $OktaResponse = Invoke-WebRequest -Method Get -UseBasicParsing -Uri $Uri -Headers $OktaHeaders -TimeoutSec 300

        #Build an Hashtable to store the links
        $link = @{}
        if ($OktaResponse.Headers.Link) { # Some searches (eg List Users with Search) do not support pagination.
            foreach ($header in $OktaResponse.Headers.Link.split(",")) {
                if ($header -match '<(.*)>; rel="(.*)"') {
                    $link[$matches[2]] = $matches[1]
                }
            }
        }

        $link = @{
            next = $link.next
        }

        try {
            $psobj = ConvertFrom-Json -InputObject $OktaResponse.Content
            $col = $col + $psobj
        } catch {
            throw "Json Exception : " + $OktaResponse
        }
    } catch { 
        throw $_
    }

    if ($link.next) {
        $loopcount++
        if ($oktaVerbose) { Write-Host "fetching next page $loopcount : " -ForegroundColor Cyan}
        Invoke-OktaPagedMethod -Uri $link.next -col $col -loopcount $loopcount   
    } else {
        return $col
    }
}

How can I get the Next Link from response header in Okta's API List Users when paginating?

2 Answers2