Ruby chunked encoding client

Question

I'm trying to make little ruby script which receives streamed data with a HTTP Chunked Transfer encoding. So I'm using ruby at the client end of this. I can find lots of questions and info about ruby (and rails) serving streams, but not much about consuming them. Maybe because it's trivial, and should just work as a low level feature of HTTP. So can someone give me a little sample script?

I tried a couple of things. This one using open-uri:

require 'open-uri'

streamURL = 'http://localhost/sample-stream.php'

puts "Opening"

open(streamURL) do |f| 
   puts "Opening steam " + streamURL + " (never gets here)"
   p  f.meta
   p "Content-Type: " + f.content_type
   p "last modified" + f.last_modified.to_s

   no = 1
   # print the first three lines
   f.each do |line|
      print "#{no}: #{line}"
      no += 1
      break if no > 4
   end
end

...and this one using net http :

require 'net/http'
require 'uri'

streamURL = 'http://localhost/sample-stream.php'

url = URI.parse(streamURL)
full_path = (url.query.empty?) ? url.path : "#{url.path}?#{url.query}"

the_request = Net::HTTP::Get.new(full_path, {'Transfer-Encoding' => 'chunked', 'content-type' => 'text/plain'})
#(Side question: Does it even make sense to specify 'chunked' in the request?)

puts "Opening stream " + streamURL

the_response = Net::HTTP.start(url.host, url.port) { |http|
  http.request(the_request)
}

puts the_response.body #never gets here

In both cases it never outputs anything, and I can see in apache my php script is busily spewing more and more data. I think it's clear what's going wrong here. In both cases I think my script is waiting to get the whole response (which will never happen) before processing it.

I should mention that there may be something wrong with my php script (the server), but I can see it blowing its chunks quite nicely into firefox.

So is there an easy ruby way to stream in the data? or should I expect to need special gems/libraries to do this? (I came across this library for example)

Answer 1

I think the following code should work.

require 'net/http'

streamURL = 'http://localhost/sample-stream.php'

uri = URI.parse(streamURL)

Net::HTTP.start(uri.host, uri.port) do |http|
  request = Net::HTTP::Get.new uri.request_uri

  http.request request do |response|
    response.read_body do |chunk|
       #We get the data here chunk-by-chunk
       puts chunk
    end
  end
end

Actually this doesn't work for my sample-stream.php, but I have had luck with something similar to this script, pointing at another server. My working code was more complicated because it also required authentication, but I've stripped it down a bit here.

Pointing at my test php script, this code results in a somewhat helpful error message:

`read_chunked': wrong chunk size line: (Net::HTTPBadResponse)

...which clearly indicates I'm doing something wrong in the php server end of things. Not sure what's wrong, but that's another question.

Answer 2

I tried the approaches in previous answers (Net::HTTP.start and wget), but didn't get any output. What worked for me was this:

url = <...>
IO.popen "curl -s #{url}" do |io|
  io.sync = true
  io.each do |line|
    break if !(process_line line)
  end
end

Answer 3

Another approach calling 'wget' system command.

wget_command = "wget #{streamURL} --user=myusername --password=mypassword -qO-"

puts "Opening stream: #{wget_command}"
f = open("|" + wget_command)
  while (line = f.gets)
    puts "LINE:" + line  
  end #(loops forever)
end

This seemed more reliable than using ruby's net http at first, and it's certainly more compact code. As an added bonus I can easily get the data line-by-line (something else I was having to fudge from the chunk reading approach) Not really sure what the disadvantages are. There are maybe inefficiencies in piping stdout data in this way. Also this code wont work on a platform without the wget command (windows) But for me this seemd to works a bit better than net http...

However it's still running into problems after three or four minutes. I'm imagining this is either because the stream is coming in too quick and some buffer somewhere is getting filled, or because this particular stream slows down after a few minutes (Don't ask me why. It just does) and so maybe the errors come when it finally catches up.