Process part of a POST request before it finished, is that possible?

Question

curl -v -F "file=@bigfile.zip" http://server/handler

I know the "bigfile.zip" will be split to several parts and sent to server part by part, that might need a long time. So how could I read the first part before the last part sent? If that's impossible with Apache/Nginx + PHP/Python, what about build another HTTP server with node.js?

What I want to find is a "long request" (another side like "long polling"). Browser can write to server immediately by using an exists long request without create a new one.

Any suggestion?

=================

Connection: Keep-Alive ?

josh3736 · Answer 1 · 2013-02-11T16:18:39.427

If I'm reading your question correctly, you want to start processing data inside a zip file while it's being uploaded, before the upload actually completes.

You need some kind of streaming zip parser that can take the incoming stream of bytes, determine when a particular file ends, and apply appropriate decompression.

You'll notice that at the beginning of a file entry, there's a 30+ byte header (dark blue) that will tell you how long the compressed file is. You can read the incoming bytes until you hit that length, at which point you process the next file's header and repeat.

As the incoming bytes are read, you hand them off to a decompressor. Zip files support many types of compression, so you have to use the one specified by the file's header.

Finally, your decompressor will give you a stream of decompressed bytes. You can either write them to disk or do something else with them.

Node is very well-suited to dealing with streams. Apache and Nginx don't really give you the level of control over the request cycle that you need to achieve your goal.

Great solution, thanks for your help. But my goal is find a "long request" solution, which might be much simple. What I need is how to read the first parts from a big POST request (before it actually completed). Could you please provide some sample code? — Alix, Feb 07 '13 at 06:10

score 0 · Accepted Answer · answered Feb 07 '13 at 13:16

This is an example code of a long lived http post request in node.js

var http = require('http');

var server = http.createServer(function(req, res) {
    if(req.method == 'POST' && req.url == '/handler') {
        req.on('data', function(data) {
          // i'm getting chunks of data in here !
        });
        req.on('end', function() {
          res.writeHead(200, 'OK');
          res.end('Got your file\n');
        });
    }
    else {
        res.writeHead(404, 'Not Found');
        res.end();
    }
});
server.listen(80);

Of course this is the most basic example and file uploads over http are slightly more complicated. This is why using something like formidable can be useful.

With node, once you are getting data you can start sending it to other places where they can be processed, even though the rest of the data is still coming. Usually you would use streams

this is an example of how to do it http://debuggable.com/posts/streaming-file-uploads-with-node-js:4ac094b2-b6c8-4a7f-bd07-28accbdd56cb

Process part of a POST request before it finished, is that possible?

2 Answers2