Stack Overflow
Stack Overflow

Questions tagged [mechanize-ruby]

The Ruby library for automating interaction with websites.

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

193 questions
1
vote
2 answers

Submitting a form via an element with Ruby Mechanize

I've got a set of pages that I'm trying to scrape with Mechanize in Ruby. On some of the pages, they redirect to a page that wants me to fill out a select-list form and then submit it. The problem is with the button that submits the form, which is…
GDP2
  • 1,948
  • 2
  • 22
  • 38
1
vote
1 answer

Net/HTTPS not getting all the content

I need to login into Jenkins through a crawler to collect some data, but Net/HTTPS gets an incomplete page in comparison to Jenkins' source, here are both sources: Net/HTTPS' HTML
Filipe Gorges Reuwsaat
  • 856
  • 1
  • 9
  • 23
1
vote
1 answer

Mechanize memory leak despite setting history size to 0 and clearing history

Here is an example script that reproduces the issue require 'mechanize' agent = Mechanize.new agent.history.max_size = 0 5000.times do |i| agent.get('http://www.yahoo.com') agent.history.clear p `ps -o rss -p #{$$}`.strip.split.last.to_i…
Cheenu Madan
  • 129
  • 12
1
vote
3 answers

Automated website interaction - Mechanzie - Rails

I'm using the Mechanize gem to automate interaction with a website form. The site i'm trying to interact with is http://www.tastekid.com/like/books I'm trying to automatically submit a string to query in the form and return the suggested books in an…
Katie H
  • 2,283
  • 5
  • 30
  • 51
1
vote
2 answers

403 Error with Mechanize on Heroku

When using mechanize to pull some data from craigslist I keep getting the following error on Heroku: status: Net::HTTPForbidden 1.1 403 Forbidden I am wondering what are some ways to prevent this from happening, my setup is below: agent =…
barnett
  • 1,572
  • 2
  • 13
  • 25
1
vote
1 answer

Click on a button outside a form (Ruby Mechanize)

I am trying to update this page by clicking a button that is outside of any forms using Ruby Mechanize. Does anyone know if this is possible? I know that I can't use the standard button submit. Below is a snippet of the html surrounding the …
arc
  • 477
  • 2
  • 8
  • 14
1
vote
2 answers

Upload multiple files in form with ruby mechanize

I can successfully upload a single file using a Mechanize form like this: def add_attachment(form, attachments) attachments.each_with_index do |attachment, i| form.file_uploads.first.file_name = attachment[:path] end end where form is a…
CHsurfer
  • 1,304
  • 1
  • 15
  • 34
1
vote
2 answers

Mechanize returning 500 on valid URL?

I'm trying to fetch a specific page with Mechanize: require 'mechanize' agent = Mechanize.new p agent.get("http://formitas.si") but I get this: `fetch': 500 => Net::HTTPInternalServerError for http://formitas.si/ -- unhandled response…
davidhq
  • 4,660
  • 6
  • 30
  • 40
1
vote
1 answer

Clicking on a div using mechanize

Im using mechanize to automate interaction with websites, everything works fine. I m logging in to the site (using username and password), navigating through the pages and everything. The problem is that I want to simulate a click on an item that…
Ran
  • 3,455
  • 12
  • 47
  • 60
1
vote
1 answer

Rails ruby-mechanize how to get a page after redirection

I want to collect manufacturers and their medicine details from http://www.mims.com/India/Browse/Alphabet/All?cat=Company&tab=company. Mechanize gem is used to extract content from html page with help of ryan Tutorial I can login successfully but…
Manivannan Jeganathan
  • 513
  • 1
  • 7
  • 22
1
vote
1 answer

Scraping pages that do not seem to have URLs

I'm trying to scrape these listings and provide more exposure for these job listings on a site that belongs to a client of mine. The issue is that I need to be able to link to the specific job listing in order for the job seeker to apply. This is…
Nick Res
  • 2,154
  • 5
  • 30
  • 48
1
vote
1 answer

Problems with text/csv Content-Encoding = UTF-8 in Ruby Mechanize

When attempting to load a page which is a CSV that has encoding of UTF-8, using Mechanize V2.5.1, I used the following code: a.content_encoding_hooks << lambda{|httpagent, uri, response, body_io| response['Content-Encoding'] = 'none' if…
Jack R-G
  • 1,778
  • 2
  • 19
  • 25
1
vote
1 answer

How do I scrape data through Mechanize and Nokogiri?

I am working on an application which gets the HTML from http://www.screener.in/. I can enter a company name like "Atul Auto Ltd" and submit it and, from the next page, scrape the following details: "CMP/BV" and "CMP". I am using this code: require…
Deepender Singla
  • 999
  • 9
  • 28
1
vote
1 answer

Mechanize throwing "bad argument (expected URI object or URI string)" using user_agent_alias

I'm trying to fetch images from Twitter but it crashes if I use user_agent_alias. I need to specify the user agent. But why it happens using agent alias? Is that normal? Try yourself with and without the { |agent| agent.user_agent_alias = 'Mac…
Luccas
  • 4,078
  • 6
  • 42
  • 72
1
vote
2 answers

different page from ruby's mechanize

My friend wrote a script, which uses ruby's mechanize to get a cookie off a page after login. start mechanize go through login get cookie value from page delivered by form The script seems to work for him, but doesn't for me. I've tried doing…
voodoogiant
  • 2,118
  • 6
  • 29
  • 49
1 2 3
…
12 13