Downloading and zipping files that were uploaded to S3 with CarrierWave

Question

I have a small Rails 3.2.1 app that uses CarrierWave 0.5.8 for file uploads to S3 (using Fog)

I want users to be able to select some images that they'd like to download, then zip them up and send them a zip. Here is what I've come up with:

def generate_zip
  #A collection of Photo objects. The Photo object has a PhotoUploader mounted.
  photos = Photo.all

  tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
  zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
  zip.close

  photos.each do |photo|
    file_to_add = photo.photo.file
    zip = Zip::ZipFile.open(tmp_filename)
    zip.add("tmp/", file_to_add.path)
    zip.close
  end

  #do the rest.. like send zip or upload file and e-mail link

end

This doesn't work because photo.photo.file returns an instance of CarrierWave::Storage::Fog::File instead of a regular file.

EDIT: The error this leads to:

Errno::ENOENT: No such file or directory - uploads/photos/name.jpg

I also tried the following:

tmp_filename = "#{Rails.root}/tmp/" << Time.now.strftime('%Y-%m-%d-%H%M%S-%N').to_s << ".zip"
    zip = Zip::ZipFile.open(tmp_filename, Zip::ZipFile::CREATE)
    zip.close

    photos.each do |photo|
      processed_uri = URI.parse(URI.escape(URI.unescape(photo.photo.file.authenticated_url)).gsub("[", "%5B").gsub("]", "%5D"))
      file_to_add = CarrierWave::Uploader::Download::RemoteFile.new(processed_uri)
      zip = Zip::ZipFile.open(tmp_filename)
      zip.add("tmp/", file_to_add.path)
      zip.close
    end

But this gives me a 403. Some help would be greatly appreciated.. It probably is not that hard I'm just Doing it Wrong™

What gem you were using? Rubyzip gem? – Rafael Oliveira Aug 26 '13 at 18:20 — Rafael Oliveira, Aug 26 '13 at 18:20

score 16 · Accepted Answer · answered Apr 04 '12 at 11:09

I've managed to solve the problem with help from @ffoeg

The solution offered by @ffoeg didn't work quite so well for me since I was dealing with zip files > 500 MB which caused me problems on Heroku. I've therefor moved the zipping to a background process using resque:

app/workers/photo_zipper.rb:

require 'zip/zip'
require 'zip/zipfilesystem'
require 'open-uri'
class PhotoZipper
  @queue = :photozip_queue

  #I pass 
  def self.perform(id_of_object_with_images, id_of_user_to_be_notified)
    user_mail = User.where(:id => id_of_user_to_be_notified).pluck(:email)
    export = PhotoZipper.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)

    Notifications.zip_ready(export.archive_url, user_mail).deliver
  end

    # Zipfile generator
  def self.generate_zip(id_of_object_with_images, id_of_user_to_be_notified)
    object = ObjectWithImages.find(id_of_object_with_images)
    photos = object.images
    # base temp dir
    temp_dir = Dir.mktmpdir
    # path for zip we are about to create, I find that ruby zip needs to write to a real file
    # This assumes the ObjectWithImages object has an attribute title which is a string.
    zip_path = File.join(temp_dir, "#{object.title}_#{Date.today.to_s}.zip")

    Zip::ZipOutputStream.open(zip_path) do |zos|
      photos.each do |photo|
        path = photo.photo.path
        zos.put_next_entry(path)
        zos.write photo.photo.file.read
      end
    end

    #Find the user that made the request
    user = User.find(id_of_user_to_be_notified)

    #Create an export object associated to the user
    export = user.exports.build

    #Associate the created zip to the export
    export.archive = File.open(zip_path)

    #Upload the archive
    export.save!

    #return the export object
    export
  ensure

    # clean up the tempdir now!
    FileUtils.rm_rf temp_dir if temp_dir
  end


end

app/controllers/photos_controller.rb:

  format.zip do
    #pick the last ObjectWithImages.. ofcourse you should include your own logic here
    id_of_object_with_images = ObjectWithImages.last.id

    #enqueue the Photozipper task
    Resque.enqueue(PhotoZipper, id_of_object_with_images, current_user.id)

    #don't keep the user waiting and flash a message with information about what's happening behind the scenes
    redirect_to some_path, :notice => "Your zip is being created, you will receive an e-mail once this process is complete"
  end

Many thanks to @ffoeg for helping me out. If your zips are smaller you could try @ffoeg's solution.

Hi, is archive a column of a db table named export? What is the data type? binary? Thanks — user1883793, Oct 25 '13 at 01:24

ffoeg · Answer 2 · 2012-03-29T17:29:32.580

11

Here is my take. There could be typos but I think this is the gist of it :)

# action method, stream the zip
def download_photos_as_zip # silly name but you get the idea
  generate_zip do |zipname, zip_path|
    File.open(zip_path, 'rb') do |zf|
      # you may need to set these to get the file to stream (if you care about that)
      # self.last_modified
      # self.etag
      # self.response.headers['Content-Length']
      self.response.headers['Content-Type'] = "application/zip"
      self.response.headers['Content-Disposition'] = "attachment; filename=#{zipname}"
      self.response.body = Enumerator.new do |out| # Enumerator is ruby 1.9
        while !zf.eof? do
          out << zf.read(4096)
        end
      end
    end
  end
end


# Zipfile generator
def generate_zip(&block)
  photos = Photo.all
  # base temp dir
  temp_dir = Dir.mktempdir
  # path for zip we are about to create, I find that ruby zip needs to write to a real file
  zip_path = File.join(temp_dir, 'export.zip')
  Zip::ZipFile::open(zip_path, true) do |zipfile|
    photos.each do |photo|
      zipfile.get_output_stream(photo.photo.identifier) do |io|
        io.write photo.photo.file.read
      end
    end
  end
  # yield the zipfile to the action
  block.call 'export.zip', zip_path
ensure
  # clean up the tempdir now!
  FileUtils.rm_rf temp_dir if temp_dir
end

edited Mar 29 '12 at 17:29

answered Mar 29 '12 at 14:39

ffoeg

2,336
1
14
13

Thank you very much for your reply. I'm using Ruby 1.9.3-p125 and I get the error (after correcting some typos) that zipfile.open calls a private method. (private method `open' called for #) – Gidogeek Mar 29 '12 at 15:14
hmm, I'll check another project where I did this. – ffoeg Mar 29 '12 at 16:44
can you show me the line in your Gemfile where you bring in the rubyzip gem? – ffoeg Mar 29 '12 at 16:45
here's mine: gem 'rubyzip', '~> 0.9.4', :require => ['zip/zip', 'zip/zipfilesystem'] – ffoeg Mar 29 '12 at 16:46
It was just 'gem 'rubyzip' but even with the more explicit declaration of the gem I get the same error. – Gidogeek Mar 29 '12 at 17:12
hrm. You are requiring 'zip/zipfilesystem' right? I'll keep looking into it. – ffoeg Mar 29 '12 at 17:15
my bad. I have a fix. coming up. – ffoeg Mar 29 '12 at 17:24
Thanks a lot for your help. I awarded you a bounty of +50. – Gidogeek Apr 04 '12 at 11:36
ffoeg, what is this 'photo.photo.identifier'? It keeps returning null here, so my zipfile is always blank. I'm using ZipOutputStream that Gidogeek is using. – Rafael Oliveira Aug 26 '13 at 18:48
1

Changing `Zip::ZipFile::open` to `Zip::File.open` worked for me. – lafeber Jul 20 '17 at 11:17

Downloading and zipping files that were uploaded to S3 with CarrierWave

2 Answers2