Stack Overflow
Stack Overflow

Questions tagged [mechanize-python]

Stateful programmatic web browsing in Python, after Andy Lester’s Perl module WWW::Mechanize.

Stateful programmatic web browsing in Python, after Andy Lester’s Perl module WWW::Mechanize.

Some of the key features:

  • The browser class mechanize.Browser implements the interface of urllib2.OpenerDirector, so any URL can be opened not just http.
  • Easy HTML form filling.
  • Convenient link parsing and following.
  • Browser history (.back() and .reload() methods).
  • The Referer HTTP header is added properly (optional).
  • Automatic observance of robots.txt (can be disabled if desired).
  • Automatic handling of HTTP-Equiv and Refresh.

python-mechanize was created by John J. Lee. Maintenance was taken over by Kovid Goyal.

HomePage | Documentation | Github | FAQ

235 questions
3
votes
5 answers

Python scraper mechanize/javascript

I have to scrape all info for former US governors from this site. However, to read out the results and then follow the links, I need to access the different results pages, or, preferably, simply set the results limit shown per page to the maximum of…
ilprincipe
  • 856
  • 6
  • 23
3
votes
1 answer

Connect to IIS server using .pfx certificate

Hi I am new to SSL and I am trying to connect to an IIS ASP.NET web server which has issued to me a unique .pfx certificate to verify me to the server. Using a browser where I have installed the certificate I am able to connect the site where I have…
Hemant
  • 225
  • 4
  • 17
3
votes
1 answer

Find all links within a div using lxml

I'm writing a tool that needs a collect all urls within a div on a web page but no urls outside that div. Simplified the page it looks something like this:
Teletha
  • 603
  • 1
  • 11
  • 21
2
votes
3 answers

Use Mechanize (Python) to get redirection log

I want to get the url redirection log using Mechanize written in Python. For example, www.google.com --> www.google.co.in. The exact question has been asked before in SO but it is for Ruby How to get redirect log in Mechanize? The answer explains…
anu.agg
  • 197
  • 13
2
votes
3 answers

Python Mechanize can't find login.live.com form

I'm trying to login to a login.live.com, but Mechanize will not recognize that there is a form on that page. Does anyone have any suggestions? br=mechanize.Browser() br.open('https://login.live.com/') br.select_form(nr=0) This results…
Zachary Orr
  • 1,724
  • 1
  • 15
  • 25
2
votes
3 answers

Python web-scraping threaded performance

I have a web app that needs both functionality and performance tested, and part of the test suite that we plan on using is already written in Python. When I first wrote this, I used mechanize as my means of web-scraping, but it seems to be too bulky…
Andy
  • 8,749
  • 5
  • 34
  • 59
2
votes
2 answers

Proxy seems to be ignored by Mechanize?

I am using an http proxy and the Mechanize module. I initialize the mechanize object and set the proxy like so: self.br = mechanize.Browser() self.br.set_proxies({"http": proxyAddress}) #proxy address is like 1.1.1.1:8080 Then I open the site…
Wes
  • 41
  • 5
2
votes
3 answers

How do I delete an instance of a class in python (by force)?

I'm using a script to test if a website runs smoothly, basically I open the site every 20 minutes or so and check the response time and so on. Like this: while True: MechBrowser = mechanize.Browser() Response =…
Shane
  • 4,875
  • 12
  • 49
  • 87
2
votes
0 answers

how to login on paypal using mechanize in python?

I have to login on paypal using mechanize in python.I tried many codes but invent.Same code is applicable for login on gmail but not work for paypal login.I tried code below.. import mechanize from bs4 import BeautifulSoup from mechanize import…
Kirti Chichmalkar
  • 41
  • 3
2
votes
0 answers

RoboBrowser submit form response status 500

I'm trying to login in Quora using robobrowser and i wrote this script from robobrowser import RoboBrowser browser = RoboBrowser( history=True, user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu…
Wendy
  • 1,523
  • 3
  • 12
  • 16
2
votes
0 answers

How to login with mechanize when no password control

i'm new to python and mechanize, and i try to login to a site where the password control doesn't seem to appear. browser = mechanize.Browser() browser.open(url) browser.select_form(nr = 0) print(browser.form) the print is :
Andro Iglesia
  • 21
  • 1
2
votes
2 answers

Mechanize Python and addheader method - how do I know the newest headers?

Currently, I'm using mechanize like this: browser = mechanize.Browser() browser.set_handle_robots(False) browser.set_handle_equiv(False) browser.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US;…
mirx
  • 606
  • 1
  • 9
  • 21
2
votes
1 answer

Python transfer mechanize browser session

I'm having a little difficulty trying to navigate a website past the login screen. I've done this using mechanize. However once I navigate past the login page I want to interact with the page, click attributes, etc. which mechanize cannot do. I also…
Alexander
  • 147
  • 1
  • 4
  • 11
2
votes
1 answer

Alternative to using mechanize to fetch data?

Currently, I am using mechanize to fill up a form and send a POST request, then I am doing a regex search on the response to get the data ( a floating point number). Is there any way I can do this by just sending a POST request? I know this is…
SanketDG
  • 550
  • 5
  • 12
2
votes
2 answers

Python mechanize form submitting doesn't work

I am trying to write a simple bot that would login to my account on a page and then comment other users' images. However I am not able to get the comment form submitting work correctly. The comment form looks like this:
Parzival
  • 302
  • 1
  • 2
  • 11
1 2
3
…
15 16