4

I am running ThreadPool rainbows + nginx (unix socket)

On large file uploads I am getting the following in nginx error log (nothing in the application log):

readv() failed (104: Connection reset by peer) while reading upstream

The browser receives response:

413 Request Entity Too Large

Why does this happen?

  • "client_max_body_size 80M;" is set both on http and server level (just in case) in nginx
  • nginx communicates with rainbows over a unix socket (upstream socket + location @ proxy_pass)
  • I don't see anything in the other logs. I have checked:
    • rainbows log
    • foreman log
    • application log
    • dmesg and /var/log/messages
  • This happens when uploading a file ~> 1 MB size
glebm
  • 20,282
  • 8
  • 51
  • 67

3 Answers3

1

The ECONNRESET (Connection reset by peer) error means that connection was uncleanly closed by a backend application. This usually happens if backend application dies, e.g. due to segmentation fault, or killed by the OOM killer. To find out exact reason you have to examine your backend logs (if any) and/or system logs.

Maxim Dounin
  • 6,365
  • 1
  • 26
  • 27
  • 1
    Usually `dmesg` or even `dmesg | grep oom -i` helps with that. – Reactormonk Oct 31 '12 at 16:32
  • I don't see anything in the logs. I have checked: rainbows log (no record of a request), foreman log (nothing), and application log (no record of a request). – glebm Nov 01 '12 at 19:05
  • I can reproduce this problem 100% of the time (always when uploading over 1 MB). – glebm Nov 01 '12 at 19:08
  • Yes. I have checked both dmesg and /var/log/messages. Is there anything else I could look at? – glebm Nov 02 '12 at 00:37
  • Or is there maybe a way to make nginx display more error details? – glebm Nov 02 '12 at 01:00
  • Switching on [debugging](http://nginx.org/en/docs/debugging_log.html) in nginx is possible, but isn't likely to help, as the error message already contains all info it has: nginx tried to read response from the upstream server but the read operation failed as connection was reset by the upstream server. You may try switch on debugging on your backend instead (if it's possible), or strace/ktrace it, or just instrument it with printf()'s or like. – Maxim Dounin Nov 02 '12 at 11:47
  • Updated the ticket with a new discovery (should've checked first). the browser receives 413 request entity too large – glebm Nov 02 '12 at 23:35
  • This suggests backend is ok, but it rejects the request due it's being too big. You have to look at backend to find out how to tune request body size it's willing to accept. – Maxim Dounin Nov 03 '12 at 00:29
  • Backend is rainbows, which shouldn't have any limits. Is it possible that I need to set some unix socket setting? – glebm Nov 03 '12 at 00:58
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/19000/discussion-between-glebm-and-maxim-dounin) – glebm Nov 03 '12 at 01:00
  • I got another question. How do I set an error page for such errors? – glebm Nov 08 '12 at 16:35
  • 1
    The [proxy_intercept_errors](http://nginx.org/r/proxy_intercept_errors) directive should help. – Maxim Dounin Nov 08 '12 at 17:32
1

Maybe you have client_max_body_size set into your nginx.conf that limits the size of the body to 1Mb, e.g. 

client_max_body_size 1M;

In this case you'd need to remove it to allow uploading files of more than 1M.

WispyCloud
  • 4,140
  • 1
  • 28
  • 31
  • Now you have edited your answer. Previously it was question and should be posted as comment not an answer. – Janak Nirmal Nov 02 '12 at 05:35
  • Yes, I have it set, but I am using lower-case `m` (set to 80m). Could that be an issue? Will try later today with upper-case – glebm Nov 02 '12 at 11:37
1

Turns out Rainbows had a configuration option called client_max_body_size that defaulted to 1 MB The option is documented here

If this options is on, Rainbows will 413 to large requests silently. You might not know it's breaking unless you run something in front of it.

Rainbows! do
  # let nginx handle max body size
  client_max_body_size nil 
end
glebm
  • 20,282
  • 8
  • 51
  • 67