0

I have a Rest API built in Python that is served by gunicorn in a docker container behind an Apache proxy which handles SSL. Requests to that API are made from Python on a Windows 10 machine using the requests package:

Python 3.4 (Win 10) -(HTTPS)-> Apache (Ubuntu 16.04) -(HTTP)-> Gunicorn (Docker)

Lately I noticed that about every second POST request (JSON payload) to the API returned a 502 server error and I saw these errors in the Apache logs:

[proxy:error] [pid 14977:tid 140336549893888] (32)Broken pipe: [client xx.xx.xx.xx:51720] AH01084: pass request body failed to 127.0.0.1:13770 (127.0.0.1)

I tried adding the KeepAlive and timeout parameters to the ProxyPass directive, which is often advised when this error occurs. I ended up with the following Apache config, but the error persisted:

<VirtualHost *:443>
    ServerName api.example.org

    ProxyPass / http://127.0.0.1:13770/ timeout=600 Keepalive=On retry=1 acquire=3000
    ProxyPassReverse / http://127.0.0.1:13770/

    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
    RequestHeader set "X-Forwarded-SSL" expr=%{HTTPS}

    ErrorLog /var/log/apache2/api_error.log
    CustomLog /var/log/apache2/api_access.log combined

    SSLCertificateFile /etc/letsencrypt/live/api.example.org/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/api.example.org/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateChainFile /etc/letsencrypt/live/api.example.org/chain.pem
</VirtualHost>

In the end, I realized that only one client machine caused these errors and it had been running for about a week. I rebooted Windows on that machine and the errors disappeared.

Does the client's uptime somehow influence the POST request body made from that machine?

FWIW, requests are authenticated with a time-based HMAC signature (like the AWS API), but that signature is only verified in the Python backend, not by the Apache proxy.

Thanks for your input!

wonkypie
  • 1
  • 1
  • 1
  • I think the root cause lies on the backend and the error you see in the Apache log is just a side effect which won't do much to help understanding the problem. Sequence of events probably is: **1.** Apache receives request from client. **2.** Apache passes request to backend. **3.** Backend has an error in processing request. **4.** Backend sends 500 error code to Apache. **5.** Backend close connection. **6.** Apache receives request body from client. **7.** Apache cannot deliver request body to backend because the connection has been closed. – kasperd Mar 06 '19 at 11:27
  • If my understanding of the problem is correct, then the error message in the Apache log is harmless and to be expected. Apache will still be able to deliver the response from the backend to the client even if that response was produced before Apache received the request body from the client. In that case the real problem is why the backend produced a 502 error in the first place, and we cannot answer that question from Apache logs. – kasperd Mar 06 '19 at 11:29
  • Thank you @kasperd for your thoughts. I compared the exact timestamps in the apache logs to the gunicorn logs and I realized that 1) Apache logged the POST request and the 502 response in `access.log` 3 seconds before the corresponding error was logged in `error.log` Could this be the 3 seconds the proxy is configured to `acquire` a connection? 2) Nothing at all was logged by Gunicorn at that timestamp. I'll look into the logging configuration there again. I still can't wrap my head around the fact that rebooting the client stopped the errors though... could that have been some coincidence? – wonkypie Mar 06 '19 at 12:50

0 Answers0