We do have a weird production issue of nginx timeouts. The timeout says below :
"upstream timed out(110: Connection timed out)"
Here, the upstream server is jetty running on the same host where nginx is also running. Jetty is running on port 8080 where nginx is running on 443.
Having checked the above error, I verified the logs of jetty logs and nginx logs. Though jetty is returning the response in less than a second, the response sent to nginx is being delayed by close to 60 secs. And nginx is timing is firing the timedout alert after 60 secs with http response code of 504. This is happening intermittently.
Below are the logs of jetty & nginx for a timed out request.
Jetty log :
{"evt":1494519426927,"intelId":"50","intelSeq":112506,"intelVer":"1","time":"2017-05-11T16:17:06.927Z","uiCorrelationIdV1":"SUI-1494519425839-42047","threadName":"qtp754853679-357","wResource":"http://m.xxx.com/search/facet/women/womens-handbags-9780510203/shoulder//N-53f3Z7hk9Z1z0roil","wMethod":"GET","wStatus":200,"wDurationMicros":1087801,"wJlpLocation":"","wFwdFor":"NADA","wHostHdr":"m.xxxx.com","wReferer":"https://m.xxxx.com/search/facet/women/womens-handbags-9780510203//N-53f3Z7hk9?search-term=Bags&sortBy=priceLow&facet=Handbag%20Style","wHttpVer":"HTTP/1.0","wWsgClientIp":"80.192.191.2","wSrcIP":"127.0.0.1","wUserAgent":"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1","intelCropped":false,"intelLength":834}
Nginx log :
2017-05-11T17:18:09+01:00 intelId="56" intelVer="2" wMethod="GET" wResource="/search/facet/women/womens-handbags-9780510203/across-body/shoulder//N-53f3Z7hk9Z1z0roq4Z1z0roil?search-term=Bags&sortBy=priceLow&facet=Handbag%20Style" wStatus="504" wCacheStatus="MISS" wSrcIP="172.17.233.135" wSize="176" wDurationSeconds="60.001" wHostHdr="m.xxx.com" wReferer="https://m.xxxx.com/search/facet/women/womens-handbags-9780510203/shoulder//N-53f3Z7hk9Z1z0roil?search-term=Bags&sortBy=priceLow&facet=Handbag%20Style" wSSL="on" wSSLver="771" wSSLciph="TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" wWsgClientIp="80.192.191.2" wFwdFor="-" wJlpLocation="-" wProtocol="HTTP/1.1" wUpstreamAddr="127.0.0.1:8080" wPort=443 s_vi="[CS]v1|2B4D4CA80501261B-600001064000B11D[CE]" s_ppv="jl%253Asearch%2C14%2C100%2C5789%2C414%2C628%2C414%2C736%2C3%2CP" recognisedUser="true" wUiCorrelationIdV1="-" wUserAgent="Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1" deviceType="mobile"
From the above two logs, we can infer that jetty returned a 200 response at 2017-05-11T16:17:06.927Z, but it was received by nginx at 2017-05-11T17:18:09+01:00. This 60 plus secs is causing the timeouts. Its bit weird as both nginx and jetty are hosted on the same host.
If someone could help us in debugging the issue or provide suggestions, it would be great.
Many Thanks in advance.
