Use Hash value on url with Nokogiri or RestClient

Question

I have a url like :

http://172.0.0.1:22230/test.action?data={"foo":"bar","joe":"doe"}&sign=x6das

In my browser I can get data from that url, but if I'm use nokogiri

Nokogiri::HTML(open('http://172.0.0.1:22230/test.action?data={"foo":"bar","joe":"doe"}&sign=x6das'))

I get

URI::InvalidURIError: bad URI(is not URI?): http://172.0.0.1:22230/test.action?data={"foo":"bar","joe":"doe"}&sign=x6das
from /home/worka/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:176:in `split'

Also with RestClient

RestClient.get 'http://172.0.0.1:22230/test.action?data={"foo":"bar","joe":"doe"}&sign=x6das'

I got same an error.

`Nokogiri::HTML(open('http://google.com/test?data={"foo":"bar","joe":"doe"}&sign=x6das'))` => `OpenURI::HTTPError: 404 Not Found`. Can you write your URI? — Ilya, Apr 12 '16 at 10:31
@Ilya I think write original uri in stackoverflow is bad idea, but I will edit common with original url in my quetion. — itx, Apr 12 '16 at 10:39
Nokogiri has nothing to do with a bad URI; It's not aware of that level of the code and instead only reads the stream if one is passed to it, and parses the resulting string. OpenURI or RestClient are the layers that would be returning that error. It's important to understand what components in your code do what. Please remove the references and tag for Nokogiri and replace them with OpenURI and your question will make more sense. — the Tin Man, Apr 12 '16 at 18:57
Where did you get the sample URI? Is it generated from your code? If so, your question seems like an XY problem where you're asking about Y but should ask about X, which is how to generate the URI. — the Tin Man, Apr 12 '16 at 19:11
@theTinMan URI from a third party, they provide a URI for my purposes site, so I can't do anything to change it. Oh, About wrong question, yeah that's my question, unclear what you're asking?, At least there are some people who understand, although my english is not good. But thanks for suggestion and double down vote from you. — itx, Apr 14 '16 at 16:31

score 0 · Answer 1 · answered Apr 12 '16 at 10:58

0

Encode your url first then use it.

url = 'http://172.0.0:22230/test.action?data={"foo":"bar","joe":"doe"}&sign=x6das'
encoded_url = CGI::escape(url)
Nokogiri::HTML(open(encoded_url))

answered Apr 12 '16 at 10:58

Asad Ali

640
6
18

I got `Errno::ENAMETOOLONG: File name too long @ rb_sysopen` – itx Apr 12 '16 at 11:12

score 0 · Answer 2 · answered Apr 12 '16 at 19:19

When dealing with URIs, it's a good idea to use the tools designed for them such as URI, which comes with Ruby.

The URI can't be

http://172.0.0.1:22230/test.action?data={"foo":"bar","joe":"doe"}&sign=x6das

because the data component is invalid. If you are adding data then I'd start with:

require 'uri'

uri = URI.parse('http://172.0.0.1:22230/test.action?sign=x6das')
query = URI.decode_www_form(uri.query).to_h # => {"sign"=>"x6das"}
data = {"foo" => "bar","joe" => "doe"}
uri.query = URI.encode_www_form(query.merge(data)) # => "sign=x6das&foo=bar&joe=doe"
uri.to_s # => "http://172.0.0.1:22230/test.action?sign=x6das&foo=bar&joe=doe"

Your initial example using {"foo":"bar","joe":"doe"} is JSON serialized data, which usually isn't passed in a URL like that. If you need to create JSON, start with the initial hash:

require 'json'

data = {"foo" => "bar","joe" => "doe"}
data.to_json # => "{\"foo\":\"bar\",\"joe\":\"doe\"}"

to_json serializes the hash into a string, which could then be encoded into the URI:

data = {"foo" => "bar","joe" => "doe"}

uri = URI.parse('http://172.0.0.1:22230/test.action?sign=x6das')
query = URI.decode_www_form(uri.query).to_h # => {"sign"=>"x6das"}
uri.query = URI.encode_www_form(query.merge('data' => data.to_json)) # => "sign=x6das&data=%7B%22foo%22%3A%22bar%22%2C%22joe%22%3A%22doe%22%7D"

But again, sending encoded JSON as a query parameter in the URI is not very common or standard since data payload is smaller without the JSON encoding.

score -1 · Answer 3 · answered Apr 12 '16 at 11:32

-1

Ok I got solved my problem

url = http://172.0.0.1:22230/test.action?data={"foo":"bar","joe":"doe"}&sign=x6das
RestClient.get(URI.encode(url.strip))

answered Apr 12 '16 at 11:32

itx

1,327
1
15
38

1

Unfortunately your code isn't going to run. You didn't declare `url` correctly. – the Tin Man Apr 13 '16 at 01:41
It isn't about declare url, but how "I" can solve to deal with url like that and I've found useful solution for my problem from [this question](http://stackoverflow.com/questions/15700784/how-to-fix-bad-uri-is-not-uri), and that's working for me. – itx Apr 14 '16 at 16:38

Use Hash value on url with Nokogiri or RestClient

3 Answers3