Invoke-WebRequest Canvas LMS API pagination

Question

I cannot to seem to find any examples on this topic and I would like to know how to do it. Can anyone show me an example or point me to a link on how to do pagination in powershell with Invoke web-request? The challenge I am facing is that I am making API calls to a server that only returns 100 rows at a time. In order to get any more rows, I would have to make a second call to the server. I have no clue how to do it. If it helps, here is the link provided by the Canvas LMS and my code that I have so far.

Pagination:

Pagination

Requests that return multiple items will be paginated to 10 items by default. You can set a custom per-page amount with the ?per_page parameter. There is an unspecified limit to how big you can set per_page to, so be sure to always check for the Link header.

To retrieve additional pages, the returned Link headers should be used. These links should be treated as opaque. They will be absolute urls that include all parameters necessary to retrieve the desired current, next, previous, first, or last page. The one exception is that if an access_token parameter is sent for authentication, it will not be included in the returned links, and must be re-appended.

Pagination information is provided in the Link header:
Link:
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueA>; rel="current",
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueB>;> rel="next",
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueC>;> rel="first",
<https://<canvas>/api/v1/courses/:id/discussion_topics.json?opaqueD>;> rel="last" 
The possible rel values are:

current - link to the current page of results. next - link to the next page of results. prev - link to the previous page of results. first - link to the first page of results. last - link to the last page of results. These will only be included if they are relevant. For example, the first page of results will not contain a rel="prev" link. rel="last" may also be excluded if the total count is too expensive to compute on each request.

The beginning product

$curlly=""
$url_main="https://[instance].instructure.com/api/v1/accounts/1/courses?per_page=1"
$security_token="imhungry"
$header = @{"Authorization"="Bearer "+ $security_token; "rel"="last"}
$curlly=Invoke-WebRequest -Headers $header   -Method Get   -Uri $url_main   
$curlly = ConvertFrom-Json $curlly.Content
foreach($course in $curlly)
{
    $course.name
}
$curlly.Count

The final product

 ##This is an example on how to use pagination in powershell
$url_main="https://[instance].instructure.com/api/v1/accounts/1/courses?per_page=100"
$security_token="boyimhungry"
$header = @{"Authorization"="Bearer "+ $security_token}
$purlly=Invoke-WebRequest -Headers $header   -Method Get   -Uri $url_main   
$curlly = ConvertFrom-Json $purlly.Content
$url_main = $purlly.Headers.Link.Split(",")[1].Replace("<","").Replace(">","") ## you can get away with just doing one replace("<","") but it looks neater this way
while( !$url_main.Contains("prev"))
{
$purlly=Invoke-WebRequest -Headers $header   -Method Get   -Uri $url_main   
$curlly += ConvertFrom-Json $purlly.Content
$url_main = $purlly.Headers.Link.Split(",")[1].Replace("<","").Replace(">","")
cls
$curlly.Count
$url_main
}
foreach($course in $curlly)
{
    $course.name
}
$curlly.Count

score 4 · Answer 1 · answered Nov 14 '16 at 13:53

I know you've accepted an answer, but I thought I'd give my code example just in case anybody needs one. This example is getting a list of all of our Canvas users. Not a terrible process -- most of the work is done with just a 4-line do..while loop.

$token = "YOUR_ACCESS_TOKEN"
$headers = @{"Authorization"="Bearer "+$token}
$allCanvasUsers = New-Object System.Collections.ArrayList @()
$pageNumber = 1

Function RequestPageOfUsers($page) {
    $script:resultsPage = Invoke-WebRequest -Method GET -Headers $headers -Uri "https://$domain/api/v1/accounts/self/users?per_page=100&search_term=P00&page=$page"
    $usersPage = ConvertFrom-Json $resultsPage.Content

    foreach ($user in $usersPage) {
        $script:allCanvasUsers.Add($user)
    }
}

do {
    RequestPageOfUsers $pageNumber
    $pageNumber++
} while ($resultsPage.Headers.Link.Contains('rel="next"'))

score 2 · Answer 2 · answered Mar 02 '20 at 23:56

David Baker's answer is fantastic, and nearly worked for me, but was only returning a single page. Not sure why, but the CONTAINS('...') test was always returning FALSE.

In order to get this to work, I had to make a minor change to the WHILE condition. For me, the ResultsPage object was returned with a property called RelationLink, which is a dictionary containing key-value pairs. The last page in a paginated set of results has no 'Next' key. So, my DO-WHILE loop runs until the 'Next' value is found to be null.

$Token = '<YOUR-TOKEN>'
$headers = @{"Authorization"="Bearer "+$token}
$allCanvasUsers = New-Object System.Collections.ArrayList @()
$pageNumber = 1

Function RequestPageOfUsers($page) {

    $script:resultsPage = Invoke-WebRequest `
        -Method GET `
        -Headers $headers `
        -Uri "https://<DOMAIN>:443/api/v1/accounts/self/users?per_page=100&page=$page"
    $usersPage = ConvertFrom-Json $resultsPage.Content

    foreach ($user in $usersPage) { 
        $script:allCanvasUsers.Add($user)
    }
}

do {
    RequestPageOfUsers $pageNumber
    $pageNumber++
} while ($resultsPage.RelationLink.Next -NE $NULL)

score 1 · Accepted Answer · answered Jun 09 '16 at 18:30

This looks like a real pain.

On each paginated request, you're going to get back a Link header that contains one or more of the links they describe.

For your purposes (a sequential read through of every result), you only really need to be concerned with the link rel=next link. You would keep calling that one until there is no rel=next anymore, which is how you'll know that you're on the last page.

You don't provide the rel part; it's not a header. It's something that lets you identify which link to use, and then use that link as is.

So the basic overview of what you need to do:

Make the first request.
Read through the returned Link header, and find the one that corresponds to rel=next (if it's not there, you're done).
Make the next request directly to the link you found.
Parse the Link header again, repeat.

Invoke-WebRequest Canvas LMS API pagination

3 Answers3