We have a Perl script that emulates a browser using LWP::UserAgent to login to a website and download some data. Recently that website was changed so that you have to have JavaScript enabled in order to log in. How can we tell what the website is doing to determine whether JavaScript is enabled, and/or how can we use LWP::UserAgent to spoof the site into thinking it is a JavaScript enabled browser?
Asked
Active
Viewed 70 times
0
JasonMArcher
- 14,195
- 22
- 56
- 52
Dave
- 1
-
1could you post some code, and/or the website URL? – HaloWebMaster May 13 '11 at 18:41
2 Answers
0
Most websites only require javascript to be enabled to log in if it's actually needed for the log in to succeed. They might use javascript to set a cookie or display the log in form. Maybe the log in is done using an XMLHTTPRequest. In most of these cases, you won't be able to spoof it; you'll actually need a javascript-enabled browser. Get an embed-able version of webkit or something.
lazycs
- 1,586
- 1
- 13
- 11
0
It turns out that the problem we were seeing was not with the Perl script at all...it was an error in a configuration file.
While researching the problem, someone suggested a great tool called wsp Web Scraping Proxy. That was a great suggestion to find out exactly what was going on between the server and the browser.
Thanks for taking the time to answer the question.
Dave
- 1