Questions tagged [mechanize-ruby]

The Ruby library for automating interaction with websites.

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

193 questions
4
votes
0 answers

Ruby Mechanize referencing pop up/new window

I'm using Ruby with mechanize gem to try to scrape data from a website. my question is, after login on the referenced page, the website launches a new page where the actual data lives. How to I reference that new page in mechanize? script so…
Dieter
  • 83
  • 1
  • 5
4
votes
1 answer

How to set the body of a POST request using Ruby Mechanize?

How can you set the body of a POST request using the Ruby Mechanize gem. I know you can do mechanize.post(url, query, headers) but I want to set the body of the POST request with a JSON string. Is that possible? So, similar to something like this…
RTF
  • 6,214
  • 12
  • 64
  • 132
4
votes
1 answer

Mechanize cookie_jar not saving session tokens

I'm using Mechanize to scrape a password-protected website upon user's request. I'm trying to decouple login and search functionality by running a Rake task that logs into the site and saves the cookies into a database, which is reused by subsequent…
Arman H
  • 5,488
  • 10
  • 51
  • 76
4
votes
1 answer

How can I perform a Head request using mechanize in Ruby

I can perform a HEAD request with Faraday (Faraday.head url), but I am using Mechanize on my current project. I would like to grab a value from the header (filename) without downloading the file. Does the Mechanize gem provide such an option? I am…
JP.
  • 5,536
  • 7
  • 58
  • 100
4
votes
1 answer

Scraping rake task seemingly suffering from unwanted caching

I'm stumped! I have a rake task which is cron'd to run every minute. It's logs in, it finds the JSON that I'm interested in but can take up to 30 runs of the task before any changes in the JSON are noticed in the rake task. During which time I've…
iOSDevil
  • 1,786
  • 3
  • 16
  • 29
3
votes
3 answers

Mechanize pre/post connect hooks

Using Mechanize, I need to find some way to limit requests to 1 per second(or 1 every 5 seconds, or 2 every minute, etc the point is find some way to rate limit requests). Searching, this seems to be the way to begin to approach the issue: pre/post…
blueblank
  • 4,724
  • 9
  • 48
  • 73
3
votes
4 answers

Anyone know of a caching plugin for Ruby Mechanize?

I have a Mechanize based Ruby script to scrape a website. I am hoping to speed it up by caching the downloaded HTML pages locally to make the whole "tweak output -> run -> tweak output" cycle quicker. I would prefer not to have to install an…
David Tinker
  • 9,383
  • 9
  • 66
  • 98
3
votes
1 answer

Ruby Mechanize Page to String

I have captured a Mechanize page. How can I get that item into a string? Pretty Print is used to output that object, however I'd like to get that into a string for further instructions. I can't seem to find any method. Any advice…
Rich_F
  • 1,830
  • 3
  • 24
  • 45
3
votes
1 answer

Mechanize getting "Errno::ECONNRESET: Connection reset by peer - SSL_connect"

I'm unable to get Mechanize to load a page that used to work -- it's reliably failing with a Errno: ECONNRESET: Connection reset by peer - SSL_connect message. Any suggestions as to what I should try or details I should look at? (Please see "what…
fearless_fool
  • 33,645
  • 23
  • 135
  • 217
3
votes
1 answer

Ruby - Socks4 proxy with WWW::Mechanize and NET::HTTP::GET

I searched on google and read in the ruby manuals, but I couldn't find a way to use WWW::Mechanize and NET::HTTP::GET over a socks4 proxy. I read, that WWW::Mechanize is a subclass of the UserAgent module, and that therefore the ->proxy() method…
maxedmelon
  • 251
  • 4
  • 17
3
votes
1 answer

Ruby mechanize clicking divs

I'm trying to search a page for divs containing a specific class and then click on them. Each of these divs containing the class have a onclick event on them. My current code looks like so, home_page =…
randy newfield
  • 1,221
  • 3
  • 25
  • 38
3
votes
4 answers

Running multiple commands in irb

I am using Mechanize to crawl a site that requires login. The following code logs me in. require 'mechanize' agent = Mechanize.new agent.get 'http://www.specialsite.com' agent.page.form.txtEmail = 'myemail@email.com' agent.page.form.txtPassword =…
user1077851
  • 111
  • 1
  • 5
3
votes
3 answers

click on xpath link with Mechanize

I want to click a link with Mechanize that I select with xpath (nokogiri). How is that possible? next_page = page.search "//div[@class='grid-dataset-pager']/span[@class='currentPage']/following-sibling::a[starts-with(@class, 'page')][1]" …
all jazz
  • 2,007
  • 2
  • 21
  • 37
3
votes
2 answers

How to block visits from Ruby Mechanize Gem?

I'm starting to use Mechanize gem for Ruby and I wonder if there is anyway a web server can detect and block activities from Mechanize agent? If yes, what's the code or steps to block Mechanize to scrap or visit a site?
2
votes
3 answers

is mechanize one of the most popular libraries for automating web site interactions in Ruby?

is mechanize one of the most popular libraries for automating web site interactions in Ruby? It's been a while since I've needed to automate a bit of web site interaction and just wanted to check there isn't now some more highly used/newer…
Greg
  • 34,042
  • 79
  • 253
  • 454
1
2
3
12 13