Stack Overflow
Stack Overflow

Questions tagged [mechanize]

Mechanize is a library for automated web browsing originally developed for Perl, there are now also Python and Ruby implementations.

Mechanize is a Ruby library for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. It is adapted from the Perl module. There is also a for Python.

2512 questions
0
votes
3 answers

recaptcha ruby script

I'm writing an application, in which the user needs to solve a recaptcha image outside of a browser, basically they would see the image from a page like this http://www.google.com/recaptcha/api/noscript?k=6Lf5YAcAAAAAAILdm73fp007vvmaaDpFb6A5HLJP,…
HRÓÐÓLFR
  • 5,842
  • 5
  • 32
  • 35
0
votes
1 answer

Error writing scraped data to csv file

I'm scraping some information off a site and one of the fields were stored in my list like this: [u'Dover Park', u'30 \u2013 38 Dover Rise'] The \2013 should be a –. When trying to write to a .csv file, I get the following…
super9
  • 29,181
  • 39
  • 119
  • 172
0
votes
2 answers

Python: How can I fix this code so it works on Windows?

import lxml.html import mechanize, cookielib import multiprocessing browser = None def download(i): link = 'www.google.com' response = browser.open(link) tree = lxml.html.parse(response) print tree return 0 if __name__ ==…
user975982
  • 37
  • 4
0
votes
3 answers

mechanize could not retrieve all forms(including some generated by js)

it is a html including two forms. One of them is generated dynamic by js when the page is loaded So, if I try to fetch them, only one form could be return, and the form generated dynamic not found. the question is how to fetch all forms even if…
user496894
  • 51
  • 1
  • 6
0
votes
1 answer

How to select a option of this selectlist in Mechanize

So I have this code running pretty well.. lately.. and I need to select an option of a select list (only one that has GET method) here is the code I am using require 'mechanize' require 'logger' agent = Mechanize.new{|a| a.log = Logger.new(STDERR)…
ingalcala
  • 1,785
  • 3
  • 14
  • 24
0
votes
1 answer

Using Python mechanize on websites that use DHTML, AJAX, etc.?

So, let's say I'm trying to create something that replies to tweets of a certain "hashtag keyword" on twitter (for example "#FirstWorldProblems") I have a script that looks like this: # apply settings, create a mechanize.Browser, etc. login() # log…
Bhaxy
  • 5,346
  • 10
  • 39
  • 41
0
votes
1 answer

Mechanize doesn't parse fields properly

I start using mechanize with Rails, but seems it doesn't work correctly, it won't find fields inside form. I've tried to use example from Mechanize web site, but it doesn't work. I did: agent = Mechanize.new page =…
Dzmitry
  • 749
  • 2
  • 11
  • 25
0
votes
1 answer

Create dynamic variables from th class name in tables, move td values into that row's array or hash?

I'm an amateur programmer wanting to scrape data from a site that is similar to this site: http://www.highschoolsports.net/massey/ (I have permission to scrape the site, by the way.) The target site has 'th' classes for each 'th' in row[0] but I…
user1010100
  • 47
  • 4
0
votes
1 answer

Perl - Mechanize? - How to get all links in a page up to a specific "delimiter" text

These are the modules I have installed. use WWW::Mechanize; use XML::Simple; use LWP::Simple; use Data::Dumper; use Web::Scraper; #use HTML::Grabber; I am trying to get all links that end in '.com' up to an html tag: '
Jim_Bo
  • 317
  • 5
  • 17
0
votes
2 answers

Entering login information for a site that doesn't use forms with Mechanize/Ruby

I need to log in to a site to run some scripts. Mechanize with Nokogiri looks perfect for the task, but I'm having a hard time inputting the login information since the input fields aren't located in forms (agent.page.forms.first yields nil): …
jengman cd
  • 25
  • 2
0
votes
1 answer

Python Mechanize: Session has expired

Trying out mechanize to scrape some content off an https asp site, it looks as if the login page submission works as I get returned a 200. But when I try and open a url presumably using the cookie captured after the login, I get re-directed back to…
adam0101
  • 1,198
  • 2
  • 16
  • 24
0
votes
1 answer

Using Nokogiri to scrape element from a web page with ruby

Description Have a web page which loading into Mechanize ruby gem. Problem is that I can see the HTML fragment in developer tools but not having any luck extracting the element and the associated text. The HTML fragment on the page, nested in a…
Grant Sayer
  • 2,330
  • 1
  • 23
  • 34
0
votes
1 answer

How resolve a 403 Forbidden message Mechanize Ruby

403 => Net::HTTPForbidden for https://www.state.gov/countries-areas-archive/tunisia/page/2/ -- unhandled response (Mechanize::ResponseCodeError) This is what i read in console , i look to scrape a 9-pages US state Department statements about…
Karim Bn Abdlaziz
  • 57
  • 7
0
votes
0 answers

Why can't I scrape this particular website with Ruby Mechanize?

I ideally want to access the API from this website, but since I am struggling to do that, I have decided to try and scrape the page instead. I am starting at this page: https://fantasy.sixnationsrugby.com/#/welcome/login Where I plan to log in and…
ldthompson
  • 29
  • 6
0
votes
0 answers

Python, cant use mechanize - No module named 'mechanize'

hope u can help me. i have the problem that i cant use mechanize in Python. i really dont know why. My python version is: Python 3.10.9 i just wrote this in a python file: import mechanize browser = mechanize.Browser() and the Error is: Traceback…
C.O.D.E
  • 1
  • 2
1 2 3
…
99
100