I think in the script below, Splash fetches twice the same url from distant server.
In first case, it sends the html rendered since the endpoint is 'render.html' and for another request, we execute lua script to click a button and then send request from 'execute' endpoint.
url = response.url + '#OddsComparison'
meta_data['type'] = 'odds'
meta_data['original_url'] = url
meta_data['splash'] = {}
meta_data['splash']['args'] = {}
meta_data['splash']['args']['url'] = url
meta_data['bet_type'] = 'Win'
meta_data['splash']['endpoint'] = 'render.html'
meta_data['splash']['splash_url'] = getSplashURL()
meta_data['splash']['wait'] = '10'
yield Request(url, self.parse_odds, meta=meta_data, headers=headers)
lua_script = """
function main(splash, args)
assert(splash:go(args.url))
assert(splash:wait(10))
js = 'document.getElementsByClassName("bettype__button")[1].click()'
splash:runjs(js)
function is_complete(splash, condition)
while not condition() do
splash:wait(0.05)
end
end
is_complete(splash, function()
return splash:evaljs("document.getElementsByClassName('bettype__button')[1].className.includes('eventActive')")
end)
return {
html = splash:html()
}
end"""
meta_data['bet_type'] = 'Place'
meta_data['splash']['args']['lua_source'] = lua_script
meta_data['splash']['endpoint'] = 'execute'
meta_data['splash']['splash_url'] = getSplashURL()
meta_data['splash']['wait'] = '10'
yield Request(url, self.parse_odds, meta=meta_data, headers=headers)
Is the same url not being fetched twice? Can I optimize this in anyway ? Can I make the request to remote server just once and send one request and using the same fetched page, run some javascript and yield the request again? Is this possible ?
