Scrape a page and get html content, after javascript updates the dom on that page

Asked May 24 '23 at 09:18

Active May 24 '23 at 09:18

Viewed 29 times

Is this possible? Scrape a page and get html content, after javascript updates the dom on that page? after 10 seconds or so...

The javascript on the page looks something like:

function renderStatisticsDiv() {
        const container = document.createElement('div');
        container.setAttribute('id', 'statisticsContainer');

        const failureCountElement = document.createElement('p');
        failureCountElement.textContent = `Failure Count: ${statistics.failureCount}`;
        const successCountElement = document.createElement('p');
        successCountElement.textContent = `Success Count: ${statistics.successCount}`;
}

Preferable in python, but in nodejs is fine too!

PS: I tried pyppeteer and selenium driver, but it didn't update the dom when scraping

asked May 24 '23 at 09:18

Oba Api

2

Can you confirm the page url, is it publicly accessible? – Barry the Platipus May 24 '23 at 09:20
Your JavaScript code sample creates objects, but does not add them to the DOM. No scraper will pick that up. – Ruud Helderman May 24 '23 at 09:24
1

Does this answer your question? [How to scrape dynamic content from a website?](https://stackoverflow.com/questions/55709463/how-to-scrape-dynamic-content-from-a-website) – Ruud Helderman May 24 '23 at 09:26
@ruud I left out code that adds the elements to the dom – Oba Api May 24 '23 at 10:58

Scrape a page and get html content, after javascript updates the dom on that page

0 Answers0