Stack Overflow
Stack Overflow

Questions tagged [mechanize-ruby]

The Ruby library for automating interaction with websites.

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

193 questions
2
votes
1 answer

ruby mechanize + https link causes ruby segfault (with rvm installed)

I followed the instructions specified in the GUIDE to use mechanize, but when I try to get an HTTPS page, I get a segfault. require 'rubygems' require 'mechanize' agent = Mechanize.new ruby-1.9.3-p0 :005 >…
Waynn Lue
  • 11,344
  • 8
  • 51
  • 76
2
votes
0 answers

Upgrading to Heroku-20, OpenSSL “dh key too small” workaround with Mechanize

Tried recently to upgrade from Heroku-18 to Heroku-20, and encounted the error described here: https://devcenter.heroku.com/articles/heroku-20-stack#upgrading-to-heroku-20 I contacted the website that I use Mechanize to automate interaction with,…
Cody
  • 436
  • 5
  • 13
2
votes
2 answers

Iconv::IllegalSequence when using www::mechanize

I'm trying to do a little bit of webscraping, but the WWW:Mechanize gem doesn't seem to like the encoding and crashes. The post request results in a 302 redirect (which mechanize follows, so far so good) and the resulting page seems to crash it. I…
Marc Seeger
  • 2,717
  • 4
  • 28
  • 32
2
votes
1 answer

Mechanize Rails - Web Scraping - Server responds with JSON - How to Parse URL from to Download CSV

I am new to Mechanize and trying to overcome this probably very obvious answer. I put together a short script to auth on an external site, then click a link that generates a CSV file dynamically. I have finally got it to click on the export button,…
GottaTinker
  • 21
  • 5
2
votes
2 answers

Crawl page which requires login with Anemone

I'm using Anemone gem in following way: Visit first url (seed), save page content to database and save all links from this page to database as well (all links which are not in database yet) Load next link from database, save its content and any…
kmaci
  • 3,094
  • 2
  • 18
  • 28
2
votes
3 answers

What is proper way to test error handling?

I've been working with the Mechanize gem a lot recently and would like to incorporate some tests to ensure I am catching the proper errors. What is the proper way to test error handling? This is my basic method: def get(str) url =…
binarymason
  • 1,351
  • 1
  • 14
  • 31
2
votes
0 answers

how to click a link with scraper/crawler

So i'm using mechanize/nokogiri with a rails project and need to be able to click a link on a web page. the webpage is this: https://www.fanduel.com/p/Home (must be signed in) so what I want to do is click a link that matches "NBA Salary Cap 60k Mon…
meowmixplzdeliver
  • 197
  • 1
  • 3
  • 9
2
votes
0 answers

Rotating proxies mechanize for Rails?

Is there a way to get Mechanize to use multiple proxies or rotate proxies for each request? agent.set_proxy(addr, port, user = nil, pass = nil) sets up a proxy, is there a way I could set this to change during each request?
Kwestion
  • 464
  • 5
  • 19
2
votes
2 answers

Mechanize::ResponseReadError - Content-Length does not match response body length

I have encountered this error while requesting a page with Mechanize: Mechanize::ResponseReadErrorContent-Length (17317) does not match response body length (17070) (Mechanize::ResponseReadError) Any thoughts on why this occurs and how I could…
Severin
  • 8,508
  • 14
  • 68
  • 117
2
votes
0 answers

Streaming a file using Mechanize in ruby

I'm trying to use Mechanize::Download so I can work with FLV video streams on the fly without buffering the whole video into memory, however Mechanize isn't following the pluggable parser for 'video/flv' and keeps returning a Mechanize::File…
Kyo
  • 277
  • 3
  • 10
2
votes
2 answers

Name Field Blank for Ruby Mechanize

Im working on submitting the search form for yellowpages.com, but when I do pretty print page I notice in the name field it is blank. How would I tell mechanize to locate that form if the name field is blank? require 'mechanize' agent =…
Seal
  • 1,060
  • 1
  • 12
  • 31
2
votes
1 answer

Can't load 'mechanize' in rails 4.0.2

In the rails interactive shell (rails c), I get a error that says: No such file to load -- Mechanize' when I run require 'mechanize'. Curiously enough I don't get any error when I enter require 'mechanize' on the ruby shell (irb). A sample output is…
user3255878
  • 23
  • 3
2
votes
2 answers

Randomness Timeout:Error Exception in Ruby with Mechanize Gem

I'm building an application in Ruby 1.9.3-p327 that fetch-parse some pages(scrapping) and then according some values insert/update some columns into the database. In order to fetch-parse, the app use Mechanize gem, and the access to the…
Rodrigo Gauzmanf
  • 2,517
  • 1
  • 23
  • 26
2
votes
1 answer

Selecting "" links with mechanize in ruby

I made a script in ruby that uses mechanize. It goes to google.com, logs you in and the does an image search for cats. Next i want to select one of the results links from the page and then save the image. My problem is that the links for all of the…
Cody H
  • 23
  • 1
  • 3
2
votes
1 answer

Ruby mechanize over SSH tunnel Socks5 proxy?

I'm trying to get mechanize to go over a SSH tunnel SOCKS5 proxy. Some reason it keeps getting an error during the HTTP requests. Like it's getting an EOF before it should. If I turn off the proxy setting it works, I'm not sure why it would be…
krizzo
  • 1,823
  • 5
  • 30
  • 52
1 2
3
…
12 13