112

I get the following error in my log files every time I try to upload a large file.

a client request body is buffered to a temporary file /var/lib/nginx/body/0000000001

Although the file uploads successfully, I always get the above error.

I increased the client_body_buffer_size to 1000m which is what I expect the largest file uploaded to be. However, this is was just a guess and although I don't get that error anymore I am wondering if this is an appropriate value to set for the client_body_buffer_size?

I would appreciate it if anyone can shed some light on this directive and how it should be used.

Abs
  • 1,559
  • 5
  • 19
  • 32

3 Answers3

99

This is a warning, not an error. That's why it was prefaced with [warn] in the log.

It means that the size of the uploaded file was larger than the in-memory buffer reserved for uploads.

The directive client_body_buffer_size controls the size of that buffer.

If you can afford to have 1GB of RAM always reserved for the occasional file upload, then that's fine. It's a performance optimization to buffer the upload in RAM rather than in a temporary file on disk, though with such large uploads a couple of extra seconds probably doesn't matter much. If most of your uploads are small, then it's probably a waste.

In the end, only you can really make the decision as to what the appropriate size is.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • 7
    Your answer has helped me make a decision. I will lower the value to about 512k to 1m. It's a shame I will get a lot of these warnings. – Abs May 29 '13 at 20:53
  • 1
    Because of [virtual memory](https://en.wikipedia.org/wiki/Virtual_memory), using a large value won't lead to "1GB of RAM always reserved for the occasional file upload". (No more RAM than it's actually needed for the current upload will be used.) – Kirill Bulygin Jun 02 '18 at 07:38
  • 6
    If I set this to 50MB and had 200 people view a page at the same instant, would that take up 10GB of memory, or would the 50MB only be allocated for any users performing a file upload? – Codemonkey Sep 18 '18 at 23:02
  • 1
    @Codemonkey This buffer is only used while a request body is being uploaded. Once the upload is complete the memory is free to be used for another request. And as the other commenter pointed out, while an upload is not in progress, no memory is used. So it depends on how many simultaneous uploads you have going on at any given instant. – Michael Hampton Sep 18 '18 at 23:51
  • Often ten, probably never more than 20. 128GB box so I have plenty of memory..! Honestly though, I'd be doing it just to get rid of those [warn] lines in my error log - I should probably just ignore them! – Codemonkey Sep 19 '18 at 01:14
  • @MichaelHampton Does allocating more buffer_size make requests faster in ideal conditions? In our case, we have max body size around 200KB. – titogeo Sep 23 '20 at 16:32
  • @titogeo How fast is your disk? Or would it just end up in cache anyway and not really affect anything? Give it a try and see what the results are. – Michael Hampton Sep 23 '20 at 16:34
  • @MichaelHampton disk is aws gp2 without Provisioned IOPS. Will give it a try anyway. Thanks. – titogeo Sep 23 '20 at 17:11
36

If you don't want to NginX store the body content in a temporary file, you can set your config. like this:

    client_body_buffer_size     10M;
    client_max_body_size        10M;

If you set both of this configurations at the same max. size (in k, M or G for kB, MB or GB, respectively), you will prevent that NginX creates a temp. file.

For more info: http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size and http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size

eddy85br
  • 485
  • 4
  • 5
  • 12
    But with that config you will also prevent all uploads which are larger than 10 MiB – Josef Feb 16 '17 at 14:55
  • @Josef: Not *prevent* by any means. It is just that the request will need to be buffered to disk, and you will get the warning. Check the accepted answer. – OmarOthman Mar 07 '17 at 15:18
  • 12
    @OmarOthman, Josef is right because of the `client_max_body_size` parameter. See the info on this link: [client_max_body_size doc.](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size): `Sets the maximum allowed size of the client request body, specified in the “Content-Length” request header field. If the size in a request exceeds the configured value, the 413 (Request Entity Too Large) error is returned to the client. Please be aware that browsers cannot correctly display this error. Setting size to 0 disables checking of client request body size.` – eddy85br Mar 07 '17 at 19:07
  • @SudipBhattarai It depends on what you are calling a temporary file, is it the NginX cache? Or this temp directory: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_temp_path ? – eddy85br Feb 27 '20 at 15:27
  • `2020/02/27 08:31:46 [warn] 4539#4539: *1823970 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000671` I get this warning when uploading image to my docker registry. `/etc/nginx/nginx.conf` has `proxy_buffering off'` and then I have proxy passed to registry in `/etc/nginx/conf.d/default.conf` – Sudip Bhattarai Feb 27 '20 at 15:46
  • @SudipBhattarai, check this article about NginX caching https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/ and this other question about it: https://stackoverflow.com/questions/6236078/how-to-clear-the-cache-of-nginx – eddy85br Feb 27 '20 at 17:42
  • 5
    @eddy85br Thanks. I had to do `proxy_buffering off;` to turn off response buffering and `proxy_request_buffering off;` to turn off request buffering. The `proxy_cache off;` is about caching response and it's turned off by default. :https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache – Sudip Bhattarai Feb 27 '20 at 17:57
4

If your upstream server that nginx reverse-proxies to supports streaming uploads of large files, it makes most sense to simply disable nginx buffering these uploads at all, using:

proxy_request_buffering off;
proxy_http_version 1.1;
client_max_body_size 0;

In case you also like to copy comments into your config:

# Explanation:
#     proxy_request_buffering off; -- upstream server can hanle large streaming uploads
#     proxy_http_version 1.1;      -- required to disable request buffering also for clients that send chunked encoding
#     client_max_body_size 0;      -- otherwise nginx imposes a size restriction; only upstream should decide that
Directive details

  • Also use proxy_http_version 1.1 because as described in the proxy_request_buffering docs, request buffering will still be on in some cases:

    When HTTP/1.1 chunked transfer encoding is used [by the client], the request body will be buffered regardless of the directive value

Rationale for disabling request buffering

There is not point in nginx storing the request in such case:

  • It delays your upstream application seeing it.
    • This also means it cannot return errors early, such as "permission denied"; it's bad UX for the user to learn this only after waiting for a 30 minute upload.
  • It unnecessarily consumes additional disk space.
    • That introduces unexpected unreliability: People may expect that for a "passthrough-only" proxy server, only network and maybe RAM are important; they do not expect that requests will fail if the disk is too small.
  • Disk IO is often slow.

Buffering the entire request body is designed for upstream applications that cannot handle slow streaming uploads (e.g. PHP scripts with a maximum runtime duration). If your upstream app does not have such a restriction, you do not need request buffering to work around it.

Further links

  • See also this comment for a more detailed explanation of how nginx does proxy buffering, in particular proxy_request_buffering.

    In a link I put in there, an nginx developer explains why for (client <- nginx) connections, some caching can still be beneficial. But for (nginx <- upstream), "some caching" is not currently possible, as proxy_request_buffering always fully buffers the entire client request and does no streaming at all.

  • This post describes proxy buffering in the other direction (client <- nginx): https://www.getpagespeed.com/server-setup/nginx/tuning-proxy_buffer_size-in-nginx Check out especially the section Disable proxy buffering?, which has a similar argument as the one I have for (nginx -> proxy) above.

nh2
  • 818
  • 3
  • 11
  • 21