Splash won't render all contents of page

Question

I am using Splash v2.3.2 and I am trying to render a page but it is not rendering everything. It won't render images or dynamically loaded content.

I am using my http://localhost:8050/ with script:

function main(splash)
  local url = splash.args.url
  assert(splash:go(url))
  assert(splash:wait(10))
  return {
    html = splash:html(),
    png = splash:png(),
    har = splash:har(),
  }
end

Here is a browser rendering:

Here is a screenshot of the Splash rendering:

I have tried to change the wait time and also tried to allow plugins. None of this will work. I am assuming that the dynamically loaded content is being restricted but I am unsure. Any help is appreciated.

score 5 · Accepted Answer · answered Jul 20 '17 at 20:41

The problem is with localStorage - site uses it, but Splash uses Private Mode by default, and this disabls localStorage. To fix it, disable private mode (see here). This script works for me (Splash 3.0):

function main(splash)
  splash.private_mode_enabled = false
  local url = splash.args.url
  assert(splash:go(url))
  assert(splash:wait(10))
  return {
    html = splash:html(),
    png = splash:png(),
    har = splash:har(),
  }
end

See also: http://splash.readthedocs.io/en/stable/faq.html#website-is-not-rendered-correctly

score 0 · Answer 2 · answered Jul 05 '17 at 08:21

I'm assuming that you are trying to scrape property description text. In your code you have just added splash:wait(10), My suggestion is you should try and implement wait for a specific css element. In you case, span#listingpropertydescription. You can write a function to wait for this particular element and then return the html page.

Note you can find a sample wait-for-element code in http://localhost:8050/

Hope this will help you

Splash won't render all contents of page

2 Answers2

Linked