3

A little info:

When 'inspected' (Google Chrome), the website displays the information I need (namely, a simple link to a .pdf).

When I cURL the website, only a part of it gets saved. This coupled with the fact that there are functions and <script> tags leads me to believe that javascript is the culprit (I'm honestly not 100% sure, as I'm pretty new at this).

I need to pull this link periodically, and it changes each time.

The question:

Is there a way for me, in bash, to run this javascript and save the new HTML code it generates to a file?

Community
  • 1
  • 1
Óndenbi
  • 31
  • 2
  • 1
    First, I would make absolutely sure that the information you want is only generated via javascript -- you can disable javascript in your chrome console, refresh the page and see if the info is there still – TKoL Jan 13 '20 at 11:49
  • 1
    If you are absolutely sure that javascript is required for your information to appear in the page, then you should consider using the npm package `puppeteer`. This runs its own instance of Chrome to run the javascript on a page, do some optional interactions if you so wish, and output whatever you might need from the rendered page – TKoL Jan 13 '20 at 11:49
  • @TKoL Thank you for the quick response! I've done what you suggested and have now confirmed that the rest of the HTML is generated via javascript. I'll try puppeteer, thank you :) – Óndenbi Jan 13 '20 at 11:52

2 Answers2

1

Not trivially.

Typically, for that approach, you need to:

  • Construct a DOM from the HTML
  • Execute the JavaScript in the context of that DOM while resolving URLs relative to the URL you fetched the HTML from

There are tools which can help with this, such as Puppeteer, PhantomJS, and Selenium, but they generally lend themselves to being driven with beefier programming languages than bash.

As an alternative, you can look at reverse engineering the page. It gets the data from somewhere. You can probably work out the URLs (the Network tab of a browser's developer tools is helpful there) and access them directly.

Quentin
  • 914,110
  • 126
  • 1,211
  • 1,335
1

If you want to download a web page that generates itself with JavaScript, you'll need to execute this JavaScript in order to load the page. To achieve this you can use libraries that do this like puppeteer with NodeJS. There's a lot of other libraries, but that's the most popular.

If you're wondering why does this happens, it's because web developers often use frameworks like React, Vue or Angular to quote the most popular ones which only generates a JavaScript output that's not executed by common HTTP requesting libraries.

Nathanael Demacon
  • 1,169
  • 7
  • 19