Trying to parse all URLs from a file or in this case a remote URL

Question

I'm trying to parse 3 arguments (this works) and request a file that has a bunch of txt and URLs in it. I'm trying to put each URL into an array I have try with some regexp with no luck and also with scan(URI.regexp) et.c and cant see to figure what I'm doing wrong.

#!/usr/bin/env ruby
require 'uri'
require 'open-uri'

HOST=ARGV[0]
ID=ARGV[1]
VERSION=ARGV[2]
MYLINKS = Array.new

file = open("http://#{HOST}/v1/dc/manifest/#{ID}/#{VERSION}").read

file.each_line do |line|
    #puts "doing #{line}" # this works..
    MYLINKS << URI.extract(line, ['http', 'https'])
end

PS: the file is a JSON file. This is all working on a Bash script but I'm migrating it to Ruby. In the Bash script I download the file to /tmp then I parse it with awk/tr etc.

Have you considered an API client like HTTParty, which will parse the JSON for you? — Mark Thomas, Mar 20 '12 at 02:23
I did but then I though that since I just need to strip out urls's I may not need it.. and I have no idea of json.. how will you go about it? — cfernandezlinux, Mar 20 '12 at 17:39

score 0 · Answer 1 · answered Mar 20 '12 at 04:27

0

You should start by replacing the constant MYLINKS by a variable, e.g. my_links. This will allow you to add elements to the my_links array. Ruby will assume that any name starting with an uppercase letter (e.g. MYLINKS, MyLinks) is a constant.

#!/usr/bin/env ruby
require 'uri'
require 'open-uri'

HOST=ARGV[0]
ID=ARGV[1]
VERSION=ARGV[2]
my_links = []

file.each_line do |line|
    my_links << URI.extract(line, ['http', 'https'])
end

answered Mar 20 '12 at 04:27

user2398029

6,699
8
48
80

@luiism hello, thanks for the reply makes sense but still not stripping the URL's from each line :-( – cfernandezlinux Mar 20 '12 at 17:38

Mark Thomas · Accepted Answer · 2012-03-22T00:50:12.643

0

With an API client like HTTParty you don't really have to know JSON, because responses are parsed into a data structure you can extract. You can start with a simple class:

require 'httparty'

class MyApi
  include HTTParty
  format :json
end

Then you can do the following:

response = MyApi.get("http://#{HOST}/v1/dc/manifest/#{ID}/#{VERSION}")

and the response will be parsed into a hash structure that you'll be able to pull your URLs from in a robust way.

If your API requires authentication or posting data, it is easy to add. There are several examples in the gem itself, so you can see how various things are done.

edited Mar 22 '12 at 00:50

answered Mar 21 '12 at 00:15

Mark Thomas

37,131
11
74
101

ok what im doing is something like file = MyApi.get("http://#{HOST}/v1/dc/manifest/#{ID}/#{VERSION}") then I file.each do |item| puts item['manifest']['files']['url'] end but is not working when I add the 'url' also on irb I can get all the sub hashes inside ['manifest']['files'] and I can only get the url if I expecify which one.. like for example puts file['manifest']['files']['1']['url'] <-- this does work on irb but since I need to get ALL url's this is why I use .each but it gives me a cant convert to string error or similar – cfernandezlinux Mar 21 '12 at 22:02
Extracting values from a hash is a different topic. It could easily be another StackOverflow question. In fact if you post a sample hash and your dilemma, I'm sure you'd get answers quickly. – Mark Thomas Mar 21 '12 at 22:43
ok I created this one, take a look since I added more examples maybe it rings a bell.http://stackoverflow.com/questions/9814456/extracting-values-from-a-hash – cfernandezlinux Mar 21 '12 at 23:42
Sorry I realized that it will only automatically parse as JSON if the URL ends in '.json'; otherwise, you have to explicitly mention the format. I have updated my answer. – Mark Thomas Mar 22 '12 at 00:49

Trying to parse all URLs from a file or in this case a remote URL

2 Answers2