2

I have a rails 3.1 app running on heroku. I need to provide the user with the ability to download csv data. I'm trying to stream the data, but it is all sent in one go. Which for larger requests will timeout.

There is much talk on the heroku site about streaming and chunking but as far as I can tell thin collects all the data and sends it in one go. How do I get it to work?

Do I have to add some middleware? e.g. unicorn The code streams fine running with mongrel.

eweb
  • 739
  • 6
  • 11
  • Looks like I do need to use unicorn, though I have to increase the timeout so that the process isn't sigkilled. I don't rate this as a solutuon. – eweb Dec 14 '12 at 11:53

2 Answers2

0

I'm pretty sure you just need to add 

stream

to the top of your controller.

More info on HTTP streaming can be found on RailsCasts: http://railscasts.com/episodes/266-http-streaming

Neil Middleton
  • 22,105
  • 18
  • 80
  • 134
  • Thanks but stream is deprecated, and all it does is set up the headers and use a Rack::Chunked, which I was doing anyway. It would appear that I have to use unicorn. Though now I'm getting an error 'transfer closed with outstanding read data remaining' – eweb Dec 14 '12 at 10:37
0

This question is really old but the issue is still very common because of the 30'' limit in Heroku responses so I will add some code on how I achieved it. Works with Rails 5.2 & 6.1 on Heroku with Puma server.

I'm using #send_stream method (present only in edge rails, future rails 7) so I just copied it + set the Last-Modified header manually. Added all in a rails concern to reuse it.

module Streameable
  extend ActiveSupport::Concern
  include ActionController::Live

  def send_stream(filename:, disposition: 'attachment', type: nil)
    response.headers['Content-Type'] =
      (type.is_a?(Symbol) ? Mime[type].to_s : type) ||
      Mime::Type.lookup_by_extension(File.extname(filename).downcase.delete('.')) ||
      'application/octet-stream'

    response.headers['Content-Disposition'] =
      ActionDispatch::Http::ContentDisposition.format(disposition: disposition, filename: filename)  # for Rails 5, use content_disposition gem

    # extra: needed for streaming correctly
    response.headers['Last-Modified'] = Time.now.httpdate

    yield response.stream
  ensure
    response.stream.close
  end
end

class ExporterController < ApplicationController
  include Streameable

  def index
    respond_to do |format|
      format.html # index.html
      format.js   # index.js
      format.csv do
        send_stream(attachment_opts) do |stream|
          stream.write "email_address,updated_at\n"

          50.times.each do |i|
            line = "user_#{i}@acme.com,#{Time.zone.now}\n"
            stream.write line
            puts line
            sleep 1  # force slow response for testing respose > 30''
          end
        end
      end
    end
  end

  private

  def attachment_opts
    {
      filename: "data_#{Time.zone.now.to_i}.csv",
      disposition: 'attachment',
      type: 'text/csv'
    }
  end
end

Then, if you use something like curl you will see the output generated second by second.

$ curl -i  http://localhost:3000/exporter.csv

An important thing is to write your code to iterate the data with #each, by using the Enumerable module. Oh, a tip with ActiveRecord, use #find_each so the DB fetch is in batches.

MegaTux
  • 1,591
  • 21
  • 26