Streaming data from Sinatra/Rack application

Question

I am trying to stream textual data (XML/JSON) from a Ruby (1.9.1p378) Sinatra (1.0) Rack (1.2.1) application. The suggested solutions (e.g. Is there a way to flush html to the wire in Sinatra) do not seem to work - the server just blocks when I yield elements of some infinite stream (e.g. from %w(foo bar).cycle). I tried webrick and thin as servers.

Any suggestions on getting this done? Should I use http://sinatra.rubyforge.org/api/classes/Sinatra/Streaming.html and if so how would I use it in my application?

score 31 · Answer 1 · answered Oct 03 '11 at 18:32

31

Starting with Sinatra 1.3, you could also use the new streaming API:

get '/evented' do
  stream(:keep_open) do |out|
    EventMachine::PeriodicTimer.new(1) { out << "#{Time.now}\n" }
  end
end

answered Oct 03 '11 at 18:32

Konstantin Haase

25,687
2
57
59

Konstantin - does this work with ALL web servers or just evented web servers like Thin? I'm having trouble tracking down example code for the new streaming API that use a standard server like Passenger... – wchrisjohnson Aug 07 '12 at 13:20
1

Only with evented web servers, like Thin, Rainbows! or Ebb - but Puma is planning to support this. – Konstantin Haase Aug 13 '12 at 13:20
Streaming without `:keep_open` works on all servers, but will not stream on all (namely not on Webrick). – Konstantin Haase Aug 13 '12 at 13:20
If you tell it to keep open, how can you close it? I.e., I want to keep it open until I've piped over all my content, then close. – Christian Johansen Nov 21 '12 at 09:53

score 19 · Accepted Answer · answered Sep 10 '10 at 08:26

Neither Webrick nor Thin support streaming that way. You could try Mongrel or Unicorn. If you want to use Thin or Rainbows!, you have to hook into the event loop in order to achieve streaming:

require 'sinatra'

class Stream
  include EventMachine::Deferrable
  def initialize
    @counter = 0
  end

  def each(&block)
    if @counter > 10
      succeed
    else
      EM.next_tick do
        yield counter
        each(&block)
      end
    end
  end
end

get '/' do
  Stream.new
end

I recently wrote a EventSource implementation that way:

require 'sinatra'

class EventStream
  include EventMachine::Deferrable
  def each
    count = 0
    timer = EventMachine::PeriodicTimer.new(1) do
      yield "data: #{count += 1}\n\n"
    end
    errback { timer.cancel }
  end
end

get '/' do
  EventMachine.next_tick do
    request.env['async.callback'].call [
      200, {'Content-Type' => 'text/event-stream'},
      EventStream.new ]
  end
  [-1, {}, []]
end

If you want to use Webrick for Streaming: here is a patch.

Thin has the advantage of being able to answer other requests while streaming. — Konstantin Haase, Sep 10 '10 at 08:27

score 7 · Answer 3 · answered Apr 07 '11 at 05:20

7

As Colin mentioned, Goliath can stream response data, as well as incoming (large file uploads). There is an example in the repo for streaming data to the client: https://github.com/postrank-labs/goliath/blob/master/examples/stream.rb

Instead of a timer, you can easily hook up any other data stream to push data to the client. For example, you can connect an AMQP queue, or any other message queue directly to Goliath and let it act as an HTTP frontend to that data.

answered Apr 07 '11 at 05:20

igrigorik

9,433
2
29
30

Is there already a community for goliath somewhere? The whole project seems to be pretty much bleeding edge and I am quite unsure about a few things like support for chunking and the identification of requests in callbacks ... – yawn Apr 12 '11 at 08:47
1

The mailing list is a great place to start: https://groups.google.com/forum/#!forum/goliath-io - there is a good group of people watching it. Some of the more interesting apps that are being built with Goliath are unfortunately not in the public, but that will change with time. It is bleeding edge. :-) – igrigorik Apr 25 '11 at 14:46

score 5 · Answer 4 · answered Apr 05 '11 at 14:21

5

You should definitely take a look at the rack-able Goliath web server. It supports streaming out-of-the-box. I am using it for a firehose style streaming api.

Goliath is both an app server and a lightweight framework designed to meet the following goals: fully asynchronous processing, middleware support, simple configuration, high-performance, and arguably most importantly, readable and maintainable code.

answered Apr 05 '11 at 14:21

Colin Surprenant

578
7
8

Very interesting project! Thanks hint! – yawn Apr 12 '11 at 08:45

Streaming data from Sinatra/Rack application

4 Answers4

Linked