Piping POST body with nginx proxy_buffering and node js formidable

Question

My setup might seem a bit peculiar, but shouldn't be too far fetched: I have an apache2 instance servering php pages and static files. I have a node js server handling file uploads (POST) using node-formidable. On top of it, I use nginx 1.0.14. In order to use the node server for upload in forms served in php files, I decided to put up nginx so I can keep the same domain, port and protocol. So basically I have /upload which I want to (directly) pipe to my node js, preferably outside of the proxy altogether. The rest is handlede by the apache2 server:

location /upload {
    client_max_body_size        10m;
    client_body_buffer_size     128k;
    proxy_connect_timeout       600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    proxy_buffer_size           4k;
    proxy_buffers               4 32k;
    proxy_busy_buffers_size     64k;
    proxy_temp_file_write_size  64k;
    send_timeout                600;
    proxy_buffering             off;
    proxy_pass https://node_backend;
}
location / {
    proxy_set_header Host $http_host;
    proxy_pass https://apache_backend;
}

Which is based on https://stackoverflow.com/a/1167559

My problem is that the nginx buffers the entire POST request before sending it from nginx to node js. I can verify this as the progress indicator in Chrome count all the way up to 100% before my node js is hit (with an instant upload from 0-100% (as it's send from localhost to localhost)).

I do not understand why proxy_buffering off isn't working the way I'm expecting (that it should not buffer, and instead pass the request to the backend as it's received).

Alternatively... Is this possible in some other way?

Any tip/help is greatly appreciated.

• Casper

• Note: I've tried using the nginx_tcp_proxy_module which is not so widespread in use. I've considered using Varnish as it is capable of doing what I want, piping directly to node-js based on url (excellent blog post by author btw., well worth a read), however I'm running an https server (two actually) so I'm not sure if Varnish can do that (https is not supported, so I'd have to wrap it, but with two domains (certs) that'd be impossible)

Answer 1

The current docs make it clear the proxy_buffering directive only affects the buffering of responses not requests.

Enables or disables buffering of responses from the proxied server.

The directive needed for buffering POST requests is the aptly named proxy_request_buffering. Again, from the current docs:

Enables or disables buffering of a client request body.

(For Casper's sake it's worth noting the proxy_request_buffering directive was added in nginx version 1.7.11, released several years after this question was asked.)