OpenURI is a Ruby module included in the standard library, which provides an easy-to-use wrapper for net/http, net/https and net/ftp.
Questions tagged [open-uri]
311 questions
0
votes
1 answer
Ruby - Opening absolute url from relative
I've built a web scraper ruby script using open-uri and nokogiri, I'm pretty new to it all but it's all working for a couple of the websites I need to extract data from that have full URL's in the source, apart from one which uses relative URL's.…
James
- 109
- 1
- 1
- 8
0
votes
0 answers
How to present credentials to s3 using open-uri
I need to download assets from Amazon s3.I am not currently able to test my code, because I don't have the credentials. I think the following should work:
open(url, http_basic_authentication: [username, password])
...assuming that url, username,…
Morgan Laco
- 160
- 2
- 10
0
votes
0 answers
Different results between Nokogiri and real browser
The target url is: http://courts.delaware.gov/opinions/List.aspx?ag=all+courts
It seems to only retrieve the first 10 links, while a real browser retrieves 50 links.
Here's some sample code to reproduce the error:
require 'open-uri'
require…
John
- 485
- 2
- 5
- 15
0
votes
1 answer
Receiving NoMethodError when attempting to retrieve HTML source for webpage using open
As part of my webpage I need to use Open-URI in order to grab the source for a webpage. For some reason whenever I try to grab the source for the webpage found at…
Saba
- 401
- 5
- 13
0
votes
2 answers
How to send form data with a POST request?
I want to view and scrape job listings on the https://www.akzonobel.com/nl/careers/vacatures/ website. The country has to be "The Netherlands" and the Job level is "Entry level".
I'm using httparty to send a POST request but it keeps returning the…
narzero
- 2,199
- 5
- 40
- 73
0
votes
2 answers
Open URI - Opening multiple pages
Using Node's open-uri module, is it possible to open multiple pages such as this? For example, a user's blog might have 10 pages.
ruby
index = 1
downloads = []
page = open("http://#{user_name}.tumblr.com/page/#{index.to_s}") do |page|
…
theGrayFox
- 921
- 1
- 10
- 22
0
votes
1 answer
Ruby Madness Downloading same file with Nokogiri, Mechanize and OpenUri to get different information
Ok,
I am writing the ubiquitous crawler and have run into some problems. ~Not surprising being a total noob at Ruby.
I use Nokogiri to get the html of a page - find all the links in it that I am interested in and then download the files associated…
Dave Gordon
- 1,815
- 4
- 30
- 52
0
votes
2 answers
How to mock open when is used in another method?
I have a method that looks like this:
def set_container_bg
open(self.page.permanent_screenshot) do |file|
# some logic with file
end
end
And I am trying to create a rspec test, but I want to mock the open method inside set_container_bg, so…
Hommer Smith
- 26,772
- 56
- 167
- 296
0
votes
1 answer
Net::FTP and open-uri hanging while trying to put binary file to from URL to FTP server
I have a scenario where I have an S3 URL with binary content that needs to be transferred to an FTP server. I'm using Net::FTP, open-uri, and Ruby 2.0.0-p353.
require 'net/ftp'
require 'open-uri'
Net::FTP.open(x,y,z) do |ftp|
…
Sam Halicke
- 194
- 2
- 9
0
votes
1 answer
How to handle a json file return by the server with ruby?
I have a json file return by a web radio
require 'open-uri'
rquiire 'json'
songlist=open('http://douban.fm/j/mine/playlist?type=n&channel=0')
##this will return a json file:
##{"r":0,"song"…
Ziu
- 649
- 1
- 8
- 20
0
votes
0 answers
Ruby open_uri always 404. (allow https redirects git version)
I'm using the open-uri module which allows https redirects.
What I'm trying to do is open every page from a domain. I do this by first crawling it through anemone:
require 'anemone'
require "./open_uri"
class Query
def initialize()
fs =…
Bula
- 2,398
- 5
- 28
- 54
0
votes
0 answers
Multiple http request at a time getting slower in ruby
I am trying to maximize the amount of data I can scrape per second with my parser.
The parser sends a get requests with open-uri and parses the response with nokogiri.
I tried put my code in threads and to fork it. But I still cant get it…
nohayeye
- 1,975
- 3
- 15
- 15
0
votes
1 answer
Read a Zip::Entry object after unzipping an xml file
I have an external xml file download that needs unzipped and parsed. I have downloaded and unzipped it but now it is stuck as an Zip::Entry object and I am unable to parse it with Nokogiri.
require 'open-uri'
require 'zip'
require 'nokogiri'
url =…
macoughl
- 591
- 2
- 5
- 17
0
votes
1 answer
File open - with open socket - ruby - reading file continuously
require 'open-uri'
file_contents = open('local-file.txt') { |f| f.read }
taking the file open method forward.. how can we open and read a local file - with a continuously live or changing data???
Some thing similar to live feed.. etc - other than…
Ajmal
- 51
- 1
- 11
0
votes
2 answers
Ruby - open-uri doesn't download a file itself, but just the HTML code of the website
I am trying to use this snippet:
open("data.csv", "wb") do |file|
file << open("https://website.com/data.php", http_basic_authentication: ["username", "password"]).read
end
But instead of the desired CSV file, I get just downloaded the HTML…
user984621
- 46,344
- 73
- 224
- 412