How do I save a webpage to the disk for later scraping using splinter library?

Question

I have the following python code using splinter library for searching a specific term in a website:

from splinter import Browser
browser = Browser()

browser.visit("http://decs.bvs.br/cgi-bin/wxis1660.exe/decsserver/?IsisScript=../cgi-bin/decsserver/decsserver.xis&interface_language=p&previous_page=homepage&previous_task=NULL&task=start")
browser.choose('search_language','p')
browser.fill('search_exp','costas')
element = browser.find_by_name("consult_button")
element.click()

And it works, the firefox page opens the page with the results. However I have not found a way to save those results as a html file to disk in orde to scrape them for terms. How do you save the webpage to disk using splinter?

Thanks in advance

Has the answer helped to solve the problem? – alecxe May 06 '15 at 09:46 — alecxe, May 06 '15 at 09:46

score 4 · Answer 1 · answered Jul 30 '14 at 14:43

You can open a file for writing and write browser.html to it:

with open('output.html', 'w') as f:
    f.write(browser.html.encode('utf-8'))

Note that I don't see the legitimate reason why you need to save the html for post-scraping. splinter (as is selenium) is powerful in terms of locating elements. The library is not only for programmatic web-browsing, it can navigate, search, extract etc. See Finding elements.

How do I save a webpage to the disk for later scraping using splinter library?

1 Answers1