Streaming large gzipped data to browser with minimal resources

Question

Is there a way to stream compressed data to the browser from rails?

I'm implementing an export function that dumps large amounts of text data from a table and lets the user download it. I'm looking for a way to do the following (and only as quickly as the user's browser can handle it to prevent resource spiking):

1. Retrieve item from table
2. Format it into a CSV row format
3. Gzip the row data
4. Stream to user
5. If there are more items, go to 1.

The idea is to keep resource usage down (i.e. don't pull from the database if it's not required, don't keep a whole CSV file/gzip state in memory). If the user aborts the download midway, rails shouldn't keep wasting time fetching the whole data set.

I also considered having rails just write a temporary file to disk and stream it from there but this would probably cause the user's browser to time out if the data is large enough.

Are there any ideas?

Answer 1

Here's an older blog post that shows an example of streaming: http://patshaughnessy.net/2010/10/11/activerecord-with-large-result-sets-part-2-streaming-data

You might also have luck with the new Streaming API and Batches. If I'm reading the documentation correctly, you'd need to do your queries and output formatting in a view template rather than your controller in order to take advantage of the streaming.

As for gzipping, it looks like the most common way to do that in Rails is Rack::Deflator. In older versions of Rails, the Streaming API didn't play well Rack::Deflator. That might be fixed now, but if not that SO question has a monkey patch that might help.

Update

Here's some test code that's working for me with JRuby on Torquebox:

# /app/controllers/test_controller.rb
def index
  respond_to do |format|
    format.csv do
      render stream: true, layout: false
    end
  end
end

# /app/views/test/index.csv.erb
<% 100.times do -%>
<%= (1..1000).to_a.shuffle.join(",") %>
<% end -%>

# /config/application.rb
module StreamTest
  class Application < Rails::Application
    config.middleware.use Rack::Deflater
  end
end

Using that as an example, you should be able to replace your view code with something like this to render your CSV

Name,Created At
<% Model.scope.find_each do |model| -%>
"<%= model.name %>","<%= model.created_at %>"
<% end -%>

As far as I can tell, Rails will continue to generate the response if the user hits stop half-way through. I think this is a limitation with HTTP, but I could be wrong. This should meet the rest of your requirements, though.