Stack Overflow
Stack Overflow

Questions tagged [mechanize-ruby]

The Ruby library for automating interaction with websites.

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

193 questions
0
votes
1 answer

Getting Mechanize::UnauthorizedError: 401 => Net::HTTPUnauthorized when accessing API with Basic Auth

I'm trying to access an API using Basic Auth. It works with HTTParty, but not with 2.7.6 Mechanize. This is what I tried: agent = Mechanize.new agent.log = Logger.new(STDERR) agent.add_auth("https://website.net/listingapi", "user", "pass") page =…
Niels Kristian
  • 8,661
  • 11
  • 59
  • 117
0
votes
1 answer

cURL works but mechanize doesn't

I am trying to scrape my university web site using ruby mechanize. This my ruby script; require 'mechanize' agent = Mechanize.new agent.get('https://kampus.izu.edu.tr') This script doesn't return response. I need to see login page but the response…
Bilal Ünal
  • 3
  • 2
0
votes
1 answer

Ruby Mechanize scraping ResponseCodeError

I'm trying to scrape all results page of a website, it works but sometimes the script stops and display this error: 502 => Net::HTTPBadGateway for https://website.com/id/12/ -- unhandled response (Mechanize::ResponseCodeError) I would like to…
Rubyx
  • 708
  • 1
  • 11
  • 32
0
votes
1 answer

Ruby-Mechanize: undefined method `button_with'

My ruby script is throwing an error : undefined method `button_with'. The ruby version is 1.9.3-p484 and the Mechanize version is 2.5.1. I checked Mechanize help and v2.5.1 is compatible with Ruby v1.9.3. form = a.page.form_with(:id =>…
Anoorag Saxena
  • 57
  • 1
  • 8
0
votes
1 answer

RUBY -Mechanize::ResponseCodeError: 503 => Net::HTTPServiceUnavailable

I tried looking for answers on other threads but didn't feel like the responses were relevant to my situation (I apologize if that's not the case). That said, I am trying to run a simple ruby script in irb > require 'mechanize' > > …
javiwins
  • 1
0
votes
0 answers

Ruby Mechanize Script fails on PHP execution, but works on SSH with Putty

I am trying to run a Ruby script that scrapes some URL, and execute it from PHP. This is my PHP code: PHP: $url = 'https://www.example.com'; $result= shell_exec("ruby script.rb $url"); echo 'result is:'; echo $result; I have two different codes,…
rockyraw
  • 1,125
  • 2
  • 15
  • 36
0
votes
0 answers

Using Ruby with Nokogiri/Mechanize to scrape data from a webforms params asp

I am currently trying to scrape data using ruby from a webpage with Nokogiri and Mechanize. I want to scrape data for to get a list of tenders from next link: http://www.panamacompra.gob.pa/ambientepublico/AP_BusquedaAvanzada.aspx --Follow this…
Javeds
  • 1
  • 1
0
votes
1 answer

Call onclick Ruby Mechanize

How can I call onclick method instead of submit using ruby Mechanize?
Mohamed Rafiq P
  • 758
  • 1
  • 7
  • 17
0
votes
1 answer

Mechanize Ruby: Print Cookies

When I do agent.get(URL) I get blank results while when I access the URL directly I can see the results. Is it because some cookies are passed to the result page? Can you please help me to print cookies before agent.get(URL) so that I will know what…
Mohamed Rafiq P
  • 758
  • 1
  • 7
  • 17
0
votes
0 answers

Can't use Mechanize to parse the website after deploy to Linode

Here is the web address which I want to parse it http://e-service.cwb.gov.tw/HistoryDataQuery/MonthDataController.do?command=viewMain&station=467300&stname=%25E6%259D%25B1%25E5%2590%2589%25E5%25B3%25B6&datepicker=2016-09 I use mechanize to parse…
Sheng-Wei Ho
  • 1
  • 1
0
votes
2 answers

Mechanize won't conect to site

Welcome, I got a problem, gem mechanize won't connect to a site. Gem is installed. Code: require 'mechanize' agent = Mechanize.new main_page = agent.get 'https://imbd.com' main_page.link_with(text: "Top 250").click rows =…
Ioo
  • 15
  • 1
  • 7
0
votes
0 answers

Running multiple threads in tandem to scrape search engines

I want to create a multi threaded web crawler, however after doing some research I've discovered that the gem mechanize is not multi thread safe. So my question is, is it possible to write a multi thread crawler to scrape multiple search engines at…
JasonBorne
  • 87
  • 2
  • 9
0
votes
0 answers

Why there is different response in python and ruby mechanize for the same HTTP GET request?

Why there is different response in python and ruby mechanize for the same HTTP GET request? Python: def send(cls, url, values = None, method = 'GET', header={}): print(url) c = cls() data = None context =…
owgitt
  • 313
  • 5
  • 22
0
votes
1 answer

Net::HTTPNotFound for https://dashboard.stripe.com/login -- unhandled response

We need to retrieve access to the logs for our Stripe instance for a specific time period. There isn't an endpoint in there API (grrrr) so we are trying a quick screen scrape, because the dashboard structures them quite nicely. At this point though…
Nick Erdenberger
  • 3
  • 1
0
votes
1 answer

"Unsupported content-encoding: gzip,gzip" when submitting form with mechanize

I'm trying to scrape some data using the mechanize library in ruby and I have to first get past a "Terms and Conditions" page. To that end I'm clicking an "I agree" button. require 'mechanize' agent = Mechanize.new agent.agent.http.verify_mode =…
Hugh Stimson
  • 33
  • 1
  • 11
1 2 3
…
12 13