2

I'm using selenium to open a webpage. When the page is opened through firefox it downloads a script and send few requests which is visible through the network console of the browser. I want to check in my program what other requests are made when all the scripts are rendered. Is it possible to do it in selenium ?

Here's my code:

proxy = Proxy({
        'proxyType': ProxyType.MANUAL,
        'httpProxy': 192.168.1.1:8080,
        'ftpProxy':192.168.1.1:8080,
        'sslProxy': 192.168.1.1:8080,
        'noProxy': ''
        })
display = Display(visible=0, size=(800, 600))
display.start()
driver = webdriver.Firefox(proxy=proxy)
driver.get("http://docs.python-requests.org/en/master/user/advanced/")
driver.quit()
display.stop()
Rahul
  • 3,208
  • 8
  • 38
  • 68
  • `"http://docs.python-requests.org/en/master/user/advanced/"` is a target page or you use it just as example? Additional requests are triggered by `JavaScript` execution? – Andersson Dec 23 '16 at 08:33
  • It's just an example. – Rahul Dec 23 '16 at 08:34
  • Take this website http://www.forumer.com/ . It's downloading flurry.js and the sending request to https://data.flurry.com/aah.do? – Rahul Dec 23 '16 at 08:37
  • Downloading of `flurry.js` and downloading of script from `https://data.flurry.com` seem not to be anyhow correlated. What exact information you want to get? `URL` of request? – Andersson Dec 23 '16 at 08:47
  • What about overriding ajax requests with something like: http://stackoverflow.com/questions/26481212/capture-ajax-response-with-selenium-and-python – jmunsch Dec 23 '16 at 09:12
  • So I disabled javascript and now no requests are made to `https://data.flurry.com`. Basically I want to capture all the network traffic(only url) where the requests are made whenever I open a website. – Rahul Dec 23 '16 at 09:12
  • @jmunsch I'm not interested in scraping the page. I only need to know if a request is made or not. Also the urls are dynamic so it cannot be automated unless the script is downloaded. – Rahul Dec 23 '16 at 09:24
  • By overwriting the ajax within the browser as shown in the example you can intercept all the requests that are made? http://stackoverflow.com/questions/6884616/intercept-all-ajax-calls#answer-10796951 you'll need to add a bit more logic though to check the url path and such. – jmunsch Dec 23 '16 at 09:30
  • @Rahul, you can use `Wireshark` tool or `tcpdump` to get network traffic as `.pcap` file and then to parse it with something like https://pypi.python.org/pypi/pcap-parser. – Andersson Dec 23 '16 at 10:36

0 Answers0