3

I'm developing a web app in Python where one of the use-cases is for a user to:

  • Upload a large file over HTTP POST and

  • Simultaneously download and display a response, which is a processed version of the file of a similar size.

The client is developed by us in C++ but I would like to use HTTP. The server doesn't need the whole file to begin to generate its response, it can start processing the data once the first 250KBs or so has arrived. The latency between the upload start and the first pieces of the response should be as low as possible (for example within 100ms of what you might reach with raw sockets for example)

Presumably it would be ideal to use chunked transfer encoding rather than multiple small HTTP requests? The length of the total request/response can't be known ahead of time but I suppose it could be split into multiple requests/responses of known size, is there a web server that would happily stream (rather than buffer + deliver) those chunks as they're being uploaded?

I've heard twisted is good with chunked transfer encoding but I'd prefer to use a more conventional web framework if possible, especially for the rest of my application (which, outside of this use-case doesn't need anything fancy like this).

icktoofay
  • 126,289
  • 21
  • 250
  • 231
user1478842
  • 135
  • 4
  • 10
  • To clarify, just in case, I would want a web framework that enables the POST handler to be called before the entire request has been downloaded by the server. – user1478842 Apr 29 '13 at 00:57

3 Answers3

1

WSGI supports this, I believe. Here we'll echo whatever they send us:

def application(environ, start_response):
    content_type = environ.get('CONTENT_TYPE', 'text/plain')
    headers = [('Content-Type', content_type)]
    if 'CONTENT_LENGTH' in environ:
        headers.append(('Content-Length', environ['CONTENT_LENGTH']))
    start_response('200 OK', headers)
    input = environ.get('wsgi.input')
    if input is None:
        yield ''
        return
    while True:
        datum = input.read(4096)  # or so
        if not datum:
            return
        yield datum

Web servers may elect to use each yield as a Transfer-Encoding: chunked chunk, though they are not required to.

icktoofay
  • 126,289
  • 21
  • 250
  • 231
  • How might I integrate this with a higher-level framework like django, flask, pyramid etc? – user1478842 May 01 '13 at 12:54
  • 1
    @user1478842: Most of those frameworks have a WSGI deployment option. If you can get access to their WSGI application object, then within your custom WSGI application object, you can determine if you'd like to handle it yourself or pass it on. If you want to pass it on, Python 3 has a neat little bit of syntax you could use: `yield from`. With it, you'd pass on like `yield from other_application(environ, start_response)`. – icktoofay May 02 '13 at 04:31
  • @user1478842: For some frameworks, you might even be able to do this directly without having to wrap the WSGI application if the framework is low-level enough itself. For example, you can do that with Werkzeug, which Flask is based on (which leads me to believe that Flask might support it somehow). The other frameworks you mentioned might support it too, but I haven't ever tried to integrate them with other raw WSGI applications. – icktoofay May 02 '13 at 04:33
0

Have a look at: https://github.com/jakobadam/plupload-backends which has a Python WSGI implementation for plupload.

It works by (IIRC) by combing multiple large requests into one file which may or may not be using chunked transfer encoding.

Adam Gent
  • 47,843
  • 23
  • 153
  • 203
0

I like web.py for simplicity. It can do chunked-transfer encoding.

http://webpy.org/cookbook/streaming_large_files

But...

That will only work for feeding the response in multiple parts. If you want to try to "stream" your data up to the server from your client you'll need you client to do multiple smaller POSTs. And you can't handle multiple responses from different POSTs via a single response... Possibly obviously?

This is a harder problem to solve than it may appear to be at first, but I'd still suggest building a ReSTful interface using web.py or some similar light weight framework.

underrun
  • 6,713
  • 2
  • 41
  • 53
  • Interesting, I'm surprised that it won't accept chunked POST data. I have noticed that django seemingly allows you to pull data from an incoming upload (see https://docs.djangoproject.com/en/dev/topics/http/file-uploads/), but I presume this that whats actually happening is that it waits for the entire POST request to arrive before calling the handler and that the 'chunks' loop is simply to make local I/O run smoother. – user1478842 Apr 29 '13 at 00:50