How to paginate (get all) Appwrite using listDocuments?

Question

I can list documents from Appwrite database using: https://appwrite.io/docs/client/database#databaseListDocuments

const sdk = new Appwrite();

sdk
    .setEndpoint('https://[HOSTNAME_OR_IP]/v1') // Your API Endpoint
    .setProject('5df5acd0d48c2') // Your project ID
;

let promise = sdk.database.listDocuments('[COLLECTION_ID]');

promise.then(function (response) {
    console.log(response); // Success
}, function (error) {
    console.log(error); // Failure
});

This function supports limit, but it is capped at a maximum 100. What if I have 500 documents? How can I get all documents using this method?

score 1 · Answer 1 · answered Jan 03 '22 at 22:54

Include an offset parameter with your request, too (keep reading for details).

Rather than return an entire database (which could bring all network traffic to a crawl), AppWrite, being paginated, returns only a single page of results with each request. The limit parameter indicates the maximum amount of data allowed on each page (rather than the maximum number of items available in all).

Including a limit of 100 tells the API server you want a single page of results with 100 documents, and by default, AppWrite will always return the first page -- unless you also specify the page number you want, too. Fortunately, the listDocuments() function allows you to do just that via its offset parameter, which you can think of as a "page number."

To retrieve more items, simply move to the next page:

For example, begin with a limit of 100 (or less) and an offset of 0 (the first page). If the server returns a full page of results (eg, 100 items in this case), then increase the offset by 1 and make another request (This second request will return the next 100 to 199 documents). Continue increasing the offset and making requests until the server returns fewer results than the limit allows. At this point, you'll know you've reached the end of the documents, without forcing the server to choke on all of the results at once.

This is a common procedure with any system returning paginated results, although the parameter names can vary (eg, "page" vs "offset").

score 0 · Answer 2 · answered Jul 08 '22 at 21:57

As of Appwrite 0.12.0, Appwrite offers 2 forms of pagination: offset and cursor.

Offset pagination is only supported up to 5000 documents. This is because the higher the offset, the longer it takes to get the data. This is an example of offset based pagination:

import { Databases } from "appwrite";
const databases = new Databases(client, "[DATABASE_ID]");  // 'client' comes from setup
                
// Page 1
const page1 = await databases.listDocuments('movies', [], 25, 0);

// Page 2
const page2 = await databases.listDocuments('movies', [], 25, 25);

Cursor based pagination is much more efficient and, as such, does not have a limit. This is an example of cursor based pagination:

import { Databases } from "appwrite";
const databases = new Databases(client, "[DATABASE_ID]");  // 'client' comes from setup

// Page 1
const page1 = await databases.listDocuments('movies', [], 25, 0);
const lastId = results.documents[results.documents.length - 1].$id;

// Page 2
const page2 = await databases.listDocuments('movies', [], 25, 0, lastId);

For more information on offset vs cursor based pagination, refer to this article. For more information on pagination in Appwrite, refer to the Appwrite Pagination docs.

How to paginate (get all) Appwrite using listDocuments?

2 Answers2