How save request to JSON with Puppeteer

Question

I'm trying to collect and save all the api calls from a website, using puppeteer.

At the moment I can print the request that include /api/

    let page = await browser.newPage();
    
    page.on("request", async (request) => {
        const url = await request.url();
        if (url.includes("/api"))
        console.log(request.url())
    })
    

    // Go to the URL
    await page.goto('http://www.twitter.com', {});

But whenever I try to save the request.url() into JSON

page.on("request", async (request) => {
        const url = await request.url();
        if (url.includes("/api"))
        fs.writeFile('./data.json', JSON.stringify(url), err => err ? console.log(err): null);
        console.log(request.url())
    })

It only logs the last request.url.

I would like to know if there is a way to save all the request :)

Did you mean to add `{}`s around your `if` block so that it includes both lines after it? Are you trying to append to the JSON rather than overwrite it? What do you mean by only "logs"? Don't you mean "only writes to file"? Are you looking for [How do I add to an existing json file in node.js](https://stackoverflow.com/questions/36093042/how-do-i-add-to-an-existing-json-file-in-node-js)? — ggorlen, Oct 04 '22 at 06:24

score 0 · Accepted Answer · answered Oct 04 '22 at 06:10

0

The request event is when the request is made, not when the response is received.

You want to listen for the page.on(“response”, handler)

answered Oct 04 '22 at 06:10

Yisheng Jiang

110
1
5

It could be an improvement in my code, but that's not answering my question console.log --> print all the request made by the website fs.writefFile --> Only write the last request I would like them both write all the request :/ – DGulian Oct 04 '22 at 06:20
Oh, you want to fs.appendFile function – Yisheng Jiang Oct 04 '22 at 06:25
Yeah ! That's it :D – DGulian Oct 04 '22 at 06:31

How save request to JSON with Puppeteer

1 Answers1