RAILS what is the best way to upload large file > 2 GB

Question

I am using carrierwave to upload Video files and encoding that videos using Transloadit. What is the best way to do this in rails with delayed job. please suggest.

Answer 1

There are a number of options (purely the HTTP side of things, not even talking about programming languages).

The question is really depending on your environment(s), skills, support etc. What I've noticed in the 'real world' is that if you send >100MB to a server over an HTTP request, it will fail. Your clients most likely have really bad upload speeds (most soho internet connections are >10M down but <1M up) so you'll eventually hit a timeout (router/nat tables/firewall/web server/scripts).

1) Really large POST (bad practice, could potentially consume a lot of memory, failure means you have to start all over and leaves your server open to DDoS)

2) Using an 'upload module' for Apache/nginx (requires compilation and generally a lot of headache to get it set up but it works well, may not work with all hosting situations)

3) Streaming within your client and server scripts. Works well. I would also recommend chunking your uploads to <10MB and when they fail, the possibility of restarting chunks.

Answer 2

I'm not much experienced with DJ, but backgroud processing have similar approach with all tools.

First you should just upload your file somewhere(filesystem, Amazon S3, whatever). DJ will not handle this task. You should do it at your controller action.

Then, after upload you can create DJ task, that encodes your video and does other related tasks.

For example, you can run DJ after commit in your video model, like

class Video < AR::Base
  after_commit :encode_in_background

  private

  def encode_in_background
    self.delay.encode(id)
  end

  def encode
    # code that runs in background
  end
end

My example can have incorrect syntax, but the main idea is that you upload video via controller and then run background processing job.