0

I'm writing an app leveraging Notion APIs to fetch all pages under my account but it couldn't get all Page IDs. I'm aware of pagination in the POST https://api.notion.com/v1/search endpoint and implemented a recursion function to fetch all of them until there is no next page, but the strange part is the numbers of returned results varying from day to day when hit the endpoint. For example, last month I got around 250 page IDs as the blocklist variable in the main, but yesterday I got only 71 results. I'm sure there are more pages in the same folder hierarchy than that. (See Edit#1)One guess is that the endpoint doesn't allow me to find pages created a specific time ago, like it can search pages only created in the last three months, but just a guess. Does anyone have idea what the root cause could be?

Edit #1:
Ruled out the possibility as these past days I ran the code again and it returned more than 200 results again. I have no idea now...

index.js

import axios from 'axios';
import dotenv from 'dotenv';
import os from 'os';

const notion = new Client({
    auth: NOTION_TOKEN,
});

// get the list of UIDs (recursion)
const fetchListOfUids = async (nextPageId, blockList = []) => {
    const payload = nextPageId ? { start_cursor: nextPageId } : {};

    try {
        const { results, has_more, next_cursor } = await notion.search({
            ...payload,
        });
        const accumulatedBlockList = [...blockList.concat(results)];

        // Continue fetching next page if any
        if (has_more) {
            return fetchListOfUids(next_cursor, accumulatedBlockList);
        }
        return accumulatedBlockList;
    } catch (err) {
        console.log({ err });
    }
};

// main
(async () => {
    createDownloadDirectory();
    try {
        const blockList = await fetchListOfUids();
        blockList.map(async (block) => {
            setTimeout(async () => {
                await downloadAsMarkdown(block);
            }, 1000);
        });
    } catch (err) {
        console.log({ err });
    }
})();
Andy Lu
  • 29
  • 1
  • 7

1 Answers1

1

Have you noticed this documented limitation of the search endpoint of the Notion API? I have not yet encountered a use case where I tried to do something like this and I'm interested to know why you would need that functionality.

Nel Prinsloo
  • 81
  • 2
  • 6
  • Hi @NelPrinsloo, yes I did notice the limitation but don't know if there is any better endpoint that I could use. My use case is actually back up pages under my account to my local storage. Just for peace of mind – Andy Lu Sep 17 '22 at 15:31
  • Interesting. A few days ago, I noticed even searching for a particular text string by using the Search functionality on the notion desktop app also did not return all the results. I was looking for a particular page and I found it a short while later using another search term, but when I read the page, I clearly saw the first string I had searched for right there bold and beautiful on the page. I should have then searched for the first string again, but my mind was already on something else. – Nel Prinsloo Jun 06 '23 at 22:27