How to correct paging by using top and skip in Azure Search?

Question

According to this document Total hits and Page Counts, if I want to implements paging, I should combine skip and top in my query.

The following is my POST query

{
  "count": true ,
  "search": "apple",
  "searchFields": "content",
  "select": "itemID, content",
  "searchMode": "all",
  "queryType":"full",
  "skip": 100,
  "top":100

}

It should return something like from 100 --> 200

{
"@odata.context": "https://example.search.windows.net/indexes('example-index-')/$metadata#docs(*)",
"@odata.count": 1611,
"@search.nextPageParameters": {
    "count": true,
    "search": "apple",
    "searchFields": "content",
    "select": "itemID, content",
    "searchMode": "all",
    "queryType": "full",
    "skip": 200
},

But it just return top 100 , not paging offset to 100

 "@odata.context": "https://example.search.windows.net/indexes('example-index')/$metadata#docs(*)",
"@odata.count": 1611,

Is there any thing I should setup?

Can you please clarify -- are you receiving @search.nextPageParameters in the response? Or is that just what you expect to see? — Bruce Johnston, Mar 20 '19 at 17:50
I don't receive @search.nextPageParameters in the response, when I add "top":100. — howie, Mar 20 '19 at 23:26
@search.nextPageParameters isn't supposed to be returned in this case. It is not supposed to be used for client-controlled paging. Please see this related question and let me know if it helps: https://stackoverflow.com/questions/46448766/why-cant-continuationtoken-be-used-for-paging-in-azure-search-api — Bruce Johnston, Mar 21 '19 at 02:38
Right now I use a for loop to iterate get nextPageParameters if reply.Next != nil { skip = reply.Next.Skip}. But each skip (offset) is 50. I only want ro increase skip (offset) size. I not understand why It is not supposed to be used for client-controlled paging. If I have search result 10,000 docs , what should I do to retrive all the document? — howie, Mar 21 '19 at 03:24
There are explanations of how to do this in the documentation. I've included a few of them, along with links to the documentation, in my answer. If you're still having trouble, please leave a comment on the answer and I'll see what I can clarify. Hope this helps! — Bruce Johnston, Mar 21 '19 at 06:58

score 5 · Answer 1 · answered Mar 21 '19 at 06:58

The short answer of how to implement paging (from this article): Set top to your desired page size, then increment skip by that page size on every request. Here is an example of how that would look with the REST API using GET:

GET /indexes/onlineCatalog/docs?search=*$top=15&$skip=0
GET /indexes/onlineCatalog/docs?search=*$top=15&$skip=15
GET /indexes/onlineCatalog/docs?search=*$top=15&$skip=30

The REST API documentation also covers how to implement paging, as well as the true purpose of @odata.nextLink and @search.nextPageParameters (also covered in this related question).

Quoting from the API reference:

Continuation of Partial Search Responses

Sometimes Azure Search can't return all the requested results in a single Search response. This can happen for different reasons, such as when the query requests too many documents by not specifying $top or specifying a value for $top that is too large. In such cases, Azure Search will include the @odata.nextLink annotation in the response body, and also @search.nextPageParameters if it was a POST request. You can use the values of these annotations to formulate another Search request to get the next part of the search response. This is called a continuation of the original Search request, and the annotations are generally called continuation tokens. See the example in Response below for details on the syntax of these annotations and where they appear in the response body.

The reasons why Azure Search might return continuation tokens are implementation-specific and subject to change. Robust clients should always be ready to handle cases where fewer documents than expected are returned and a continuation token is included to continue retrieving documents. Also note that you must use the same HTTP method as the original request in order to continue. For example, if you sent a GET request, any continuation requests you send must also use GET (and likewise for POST).

Note

The purpose of @odata.nextLink and @search.nextPageParameters is to protect the service from queries that request too many results, not to provide a general mechanism for paging. If you want to page through results, use $top and $skip together. For example, if you want pages of size 10, your first request should have $top=10 and $skip=0, the second request should have $top=10 and $skip=10, the third request should have $top=10 and $skip=20, and so on.

I use post method, and also follow this document. But what I got is when I add "top" , the "@odata.nextLink" and "@search.nextPageParameters" will not appeal. If I remove "top" , and only keep skip , the paging work just fine. — howie, Mar 21 '19 at 07:09
No, it’s by design, as described by the documentation and my answer above. — Bruce Johnston, Mar 21 '19 at 14:57

score -1 · Accepted Answer · answered Mar 22 '19 at 00:11

The follwing example is Get method without top.

GET Method

https://example.search.windows.net/indexes('example-content-index-zh')/docs?api-version=2017-11-11&search=2018&$count=true&$skip=150&select=itemID

The Result:

{
"@odata.context": "https://example.search.windows.net/indexes('example-index-zh')/$metadata#docs(*)",
"@odata.count": 133063,
"value": [ //skip value ],
"@odata.nextLink": "https://example.search.windows.net/indexes('example-content-index-zh')/docs?api-version=2017-11-11&search=2018&$count=true&$skip=150"
}

No matter I use post or get, onces I add $top=# the @odata.nextLink will not show in result.

Finally, I figure out although @odata.nextLink or @search.nextPageParameters will not show. But the paging is working, I have to count myself.

How to correct paging by using top and skip in Azure Search?

2 Answers2