Questions tagged [mechanize]

Mechanize is a library for automated web browsing originally developed for Perl, there are now also Python and Ruby implementations.

Mechanize is a Ruby library for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. It is adapted from the Perl module. There is also a for Python.

2512 questions
0
votes
1 answer

Throttle Mechanize gem

Is there any built-in way to throttle Mechanize gem? I'm searching something like a callback on making an HTTP request. Later edit: I would like to implement bandwith throttling, to avoid flooding parsed sites. EG: Only allow one request per…
Vlad Zloteanu
  • 8,464
  • 3
  • 41
  • 58
0
votes
1 answer

Web-scraping with Ruby Mechanize

I have tried scraping a web page with ruby mechanize but is not working. Basically that website have some products and i need the links of the products. HTML I was test this code below, and I expected the links to the products but the output…
0
votes
1 answer

Automated Web Scraping Issues

I am developing a rather large automation application to scrape various abandoned property information from various state databases, in order to find specific properties. I have already developed search scripts for about 8 state websites, using…
Anthony
  • 25
  • 5
0
votes
0 answers

How to use the mechanize Browser cookiejar for POST requests?

I have the following scenario: I have logged in successful with with mechanize.Browser(): url_login = "http://my_login_page.com/login" br = mechanize.Browser() br.set_handle_robots(False) br.addheaders = [ ( "User-agent", …
MaKaNu
  • 762
  • 8
  • 25
0
votes
1 answer

Issue in parsing nokogiri element

I am using Mechanize gem to parse html content. Firing this on terminal, agent.get("http://www.example.com/").search(".sidebar-deal-excerpt").first gives me this:
Mohit Jain
  • 43,139
  • 57
  • 169
  • 274
0
votes
0 answers

what's wrong with this form submission (python-mechanize)

I am trying to automate data uploads to a private website using python mechanize. I successfully login and navigate to the uploading page, which offers 3 possible ways of providing data: database connection (source-sql), file upload (source-file),…
abu
  • 422
  • 7
  • 14
0
votes
0 answers

Passing cookies from mechanize to requests-HTML in Python

I'm attempting to write a Python script that logs in to a website that runs JavaScript and scrape an element from the dashboard page. I'm using mechanize to login to the website and Requests-HTML to scape the data. I can successfully login to the…
0
votes
1 answer

How to check if remote file exits behind a proxy

I writing app that connect to a web server (I am the owner of he server) sends information provided by the user, process that information and send result back to the application. The time needed to process the results depends on the user request…
aloctavodia
  • 2,040
  • 21
  • 28
0
votes
1 answer

uninitialized constant Scrubyt::Navigation::Mechanize::WWW (NameError)

I am using the Scrubyt gem, and when I use it, I get this error: /home/bitnami/.rvm/gems/ruby-1.8.7-p352/gems/scrubyt-0.4.06/lib/scrubyt/core/navigation/agents/mechanize.rb:17:in `included': uninitialized constant Scrubyt::Navigation::Mechanize::WWW…
Hanxue
  • 12,243
  • 18
  • 88
  • 130
0
votes
1 answer

Automating Web form Filling using Selenium or Mechanize Python with Pandas Dataframe

Automating Web form Filling using Selenium or Mechanize Python with Pandas Dataframe. I am looping through a three columns of pandas dataframe to auto-complete the form for me and submit. Below is the code I came up with. Should I use Selenium or…
user19671534
0
votes
1 answer

WWW::Mechanize and yellowpages.com

I'm trying to search yellowpages.com via the Perl module WWW::Mechanize. $mech->get( "http://www.yellowpages.com" ); $mech->form_name( "standard-searchform" ); $mech->field( "search-terms, "schneider" ); $mech->field( "search-location", "CA"…
manuel
  • 165
  • 1
  • 1
  • 8
0
votes
1 answer

Using twill's builtin mechanize distribution throws AttributeError on _debug?

I use twill (0.9) and I want to also access its builtin mechanize distribution (on Python 2.6.6). I read all the documentation and I'm not clear whether they even support or recommend that, if so they certainly don't publicize it. It seems you can…
smci
  • 32,567
  • 20
  • 113
  • 146
0
votes
1 answer

Unable to click download button with web scraper

for whatever reason mechanize is unable to click this export button to automate downloading a public government csv file, i'm trying to automate downloading a fishing report, does anyone have any ideas on how to get it to work? agent =…
alilland
  • 2,039
  • 1
  • 21
  • 42
0
votes
0 answers

How can I get a contain part of an HTML element by ID or NAME - Python - browser or BeautifulSoup

from gettext import find import mechanize browser = mechanize.Browser() browser.set_handle_robots(False) cookies = mechanize.CookieJar() browser.set_cookiejar(cookies) browser.addheaders = [('user_agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X…
DeziLuv
  • 69
  • 6
0
votes
1 answer

Structure of HTML Form Data

I know this is stupid question but how does payload of this form data be formatted. I can't make request more than one time so i can't see real request send. I have tried same on local html file and then tries to make request but i don't know if…
Adarsh Raj
  • 325
  • 4
  • 17
1 2 3
99
100