Read DOMStorage / LocalStorage via Chrome DevTools Protocol

Question

I know the following code snippet allows me to read all current cookies:

let Main = await import('./main/main.js');
await Main.MainImpl.sendOverProtocol('Network.getCookies');

Now, for testing purposes, I need to get all available LocalStorage entries. I tried a few things but unfortunately I did not get it. Chrome DevTools itself does only query it with the already available securityOrigin variable, and I do not know where it is taken from:

I also found Page.getFrameTree but there are missing some entries - so I think this does not relate to the entries of LocalStorage:

Is there any other method in the docs I am missing?

Edit #1

As mentioned in the comments I got it to work with Page.getResourceTree. Unfortunately the next issue I get is the following:

Protocol error (DOMStorage.getDOMStorageItems): Frame not found for the given security origin.

`securityOrigin` is simply the `protocol://hostname:port` part of a URL so all you need is to enumerate all frames in the page via `Page.getResourceTree` (this is what devtools uses). — wOxxOm, Jun 13 '20 at 08:11
@wOxxOm I just tried `Page.getResourceTree`, unfortunately I never get `https://vars.hotjar.com`. The cause is that a script of `vars.hotjar.com` is initiated through another script of `https://static.hotjar.com`. — Matthias Günter, Jun 13 '20 at 08:59
It's possible the page doesn't create that frame when executed in puppeteer. Maybe you need to wait a bit or scroll the page. Also note that although a script can create a frame but frames aren't related to scripts. Anyway this is the way devtools works. — wOxxOm, Jun 13 '20 at 09:26
I got it to work but I got a new error. I have edited the question. — Matthias Günter, Jun 13 '20 at 11:59

score 2 · Answer 1 · answered Jun 13 '20 at 08:10

Does this suffice?

'use strict';

const puppeteer = require('puppeteer');

(async function main() {
  try {
    const browser = await puppeteer.launch();
    const [page] = await browser.pages();

    await page.goto('http://example.org/');

    await page.evaluate(() => { window.localStorage.setItem('foo', 42); });

    const cdp = await page.target().createCDPSession();

    const data = await cdp.send('DOMStorage.getDOMStorageItems', {
      storageId: {
        securityOrigin: await page.evaluate(() => window.origin),
        isLocalStorage: true,
      },
    });
    console.log(data);

    await browser.close();
  } catch (err) {
    console.error(err);
  }
})();

Output:

{ entries: [ [ 'foo', '42' ] ] }

I got it to work, unfortunately I get this error message when I try to get the `LocalStorage` of another domain: See edited question (Edit #1) — Matthias Günter, Jun 13 '20 at 12:01
can you update your post with the solution pls? same issue there — MichelDelpech, Sep 02 '20 at 09:59

rriemann · Answer 2 · 2020-08-26T11:04:47.063

I use the following code:

getLocalStorage = async function(page, data = {}) {
  // based on https://stackoverflow.com/a/54355801/1407622
  const client = page._client;
  for (const frame of page.frames()) {
    // it is unclear when the following url values occur:
    // potentially about:blank is the frame before the very first page is browsed
    if(!frame.url().startsWith('http')) continue; // filters chrome-error://, about:blank and empty url

    const securityOrigin = new url.URL(frame.url()).origin;
    const response = await client.send('DOMStorage.getDOMStorageItems',
      { storageId: { isLocalStorage: true, securityOrigin } },
    );
    if (response.entries.length > 0) {
      let entries = {};
      for (const [key,val] of response.entries) {
        entries[key] = {
          value: safeJSONParse(val),
        };
      }
      // console.log(response.entries);
      data[securityOrigin] = Object.assign({}, data[securityOrigin], entries);
    }
  }
  return data;
};

source: https://github.com/EU-EDPS/website-evidence-collector/blob/master/lib/tools.js
original SA source: Get all values from localStorage using Puppeteer

However, sometimes I get the same error: "Frame not found for the given security origin"

score 0 · Answer 3 · answered Sep 02 '20 at 10:09

Puppeteer may opens a blank chromium tab so you may get bad frame urls like about:blank or empty urls.

I managed to bypass this error by adding two simple check :

      if (!frame.url()) {
        continue;
      }

      if (frame.url() === 'about:blank') {
        continue;
      }

it seems like there are some bad urls too like :

chrome-error://chromewebdata/

Read DOMStorage / LocalStorage via Chrome DevTools Protocol

Edit #1

3 Answers3