How to get ALL request headers in Puppeteer

Question

I'm trying to get ALL request headers to properly inspect the request, but it only returns headers like the User-Agent and Origin, while the original request contains a lot more headers.

Is there a way of actually getting all the headers?

For reference, here's the code:

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({
    headless: false
});

const page = await browser.newPage();
page.on('request', req => {
   console.log(req.headers());
});
await page.goto('https://reddit.com');

Thanks in advance, iLinked

There might be a way by listening for the cdp request events as noted here https://stackoverflow.com/a/62232903 — Gavin Haynes, Oct 09 '20 at 23:51
Also check out [Headers in Puppeteer are not same as in browser](https://stackoverflow.com/questions/62336825/headers-in-puppeteer-are-not-same-as-in-browser) — Sergey Geron, Jan 28 '22 at 06:16

Elia Weiss · Answer 1 · 2021-03-04T13:37:15.017

4

U can use the url https://headers.cloxy.net/request.php to see your headers

await page.goto('https://headers.cloxy.net/request.php');

U can also print to log

  console.log((await page.goto('https://example.org/')).request().headers());

edited Mar 04 '21 at 13:37

answered Mar 04 '21 at 13:28

Elia Weiss

8,324
13
70
110

score 1 · Answer 2 · answered Mar 08 '21 at 10:42

You can switch from puppeteer to playwright, and then using Firefox (but not Chromium or WebKit) you'll get more headers:

import playwright from 'playwright';

(async () => {
    const browser = await playwright['firefox'].launch();
    const page = await browser.newPage();

    page.on('request', req => {
        console.log(req.headers());
    });
    await page.goto("https://example.com/");

    await browser.close();
})();

playwright['firefox'] output (on other sites I've seen cookies too):

{
  host: 'example.com',
  'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0',
  accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
  'accept-language': 'en-US,en;q=0.5',
  'accept-encoding': 'gzip, deflate, br',
  connection: 'keep-alive',
  'upgrade-insecure-requests': '1'
}

vs. playwright['chromium'] output:

{
  'upgrade-insecure-requests': '1',
  'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4421.0 Safari/537.36'
}

I don't know who downvoted but you deserve an upvote because this works too well!! Thanks for sharing a code! I wonder how would you use `playwright` to get the body's data as well? — Jugert Mucoimaj, Jul 27 '23 at 11:54
[`page.content()`](https://playwright.dev/docs/api/class-page#page-content) — Boris Verkhovskiy, Jul 27 '23 at 15:51
it gives me an error: `Promise { } node:internal/process/promises:288 triggerUncaughtException(err, true /* fromPromise */); ^ page.content: Target closed` — Jugert Mucoimaj, Jul 27 '23 at 19:51

How to get ALL request headers in Puppeteer

2 Answers2