0

I'm trying to upload a large file into GridFS using PyMongo and an API I wrote in Tornado. Since the file size is 4.1GB Tornado throws an Content-Length too long.

This scenario works with NodeJS and this module but I am unsure if PyMongo supports file streaming.

ivica
  • 1,388
  • 1
  • 22
  • 38
  • 1
    You are seemingly having a problem in your Tornado implementation at the moment. There was a Gist you could compare as a starter to get the upload part working https://gist.github.com/nephics/1134964 After you have that then your code in the post would be most helpful. – Neil Lunn Mar 12 '14 at 09:16

1 Answers1

1

PyMongo supports streaming an input file to GridFS, but Tornado does not. See issue 231. Although the original issue report is old, rumors on the Tornado mailing list suggest this is going to be fixed within the next few minor releases. (Currently at Tornado 3.2.) When Tornado sees a content-length over 100MB it raises the exception you observe.

One idea is to front your application with Nginx and use its file-upload module to stream the user's upload into a temp file. Nginx will tell your application when the file is uploaded, and you can then stream the temp file into GridFS. Avoid blocking the Tornado process while you do this, either by spawning a subprocess to copy the file into GridFS with PyMongo, or by using Motor.

A. Jesse Jiryu Davis
  • 23,641
  • 4
  • 57
  • 70
  • I also found Motor http://motor.readthedocs.org/en/latest/examples/gridfs.html which should help me. – ivica Mar 14 '14 at 07:49