A GUI-Less Browser to scrape with

Question

I'm writing a tool to get info from several websites, all require me to log in. My normal approach is to follow the requests and responses in Fiddler or alike, and follow that direct path.

However, that feels a bit strict. A minimal change in the website could break my code. So I'm looking for something like a UI-less browser that I can use the following way:

Browser.Load("https://sourceforge.net/account/login.php");
Browser.Document.ElementById("form_loginname").Value = "login";
Browser.Document.ElementById("form_pw").Value = "password";
(Browser.Document.ElementById("login") As WebButton).Click(); // the login button is named "login".

After this code finishes I'd like to see the page I would get in a regular browser.

Does something like I described exist?

EDIT - C# support is preferred

score 2 · Accepted Answer · edited May 23 '17 at 11:43

2

This is a lot like automated user testing for web applications, with the key difference being you don't own the tested application.

Selenium is a popular library to automate driving a browser. If you want to run a program using Selenium in a headless fashion, you can use a headless X server such as Xvfb. Regarding not using Xvfb, there is a older question on alternatives.

Since Selenium can drive many browsers from a variety of programming languages, I encourage you to explore the Selenium tag on StackOverflow.

Of particular interest for you would be something like SimpleBrowser.WebDriver: Selenium bindings for an in-memory light weight browser for .Net. Actually, SimpleBrowser directly might address your needs, without the added complexity of Selenium.

edited May 23 '17 at 11:43

Community

1
1

answered Nov 08 '12 at 23:38

Dwight Holman

1,570
14
15

Does selenium drive a browser? Or does it "browse" by itself? – Nitay Nov 09 '12 at 00:15
Drives a browser, so it has full support for pages with Javascript, hidden fields, etc. – Dwight Holman Nov 09 '12 at 00:30
This means I need to actually use a browser which is installed on the target machine. It's a good fall-back, but I'd rather find something lighter... Though Selenium gives the most complete solution – Nitay Nov 09 '12 at 00:35
I've added a link to a lightweight browser library to my answer. – Dwight Holman Nov 09 '12 at 00:42
This looks perfect. I'm checking it out now – Nitay Nov 09 '12 at 01:05
SimpleBrowser is just what i needed. Thanks! – Nitay Nov 09 '12 at 01:38

BeniBela · Answer 2 · 2012-11-08T23:51:22.407

1

My Xidel can do something like that:

e.g.:

 xidel https://sourceforge.net/account/login.php -f 'form((//form)[2], "form_loginname=login&form_pw=password&login=Log in")' --download site.html

(the 'login=Log in' is value of the submit button)

It does not support JavaScript through. I heard there is something called Selenium that does...

edited Nov 08 '12 at 23:51

answered Nov 08 '12 at 23:39

BeniBela

16,412
4
45
52

A GUI-Less Browser to scrape with

2 Answers2