I have a web-app with DigitalOcean (gunicorn/nginx) using Selenium and Django.
I'm trying to scrape data from 3 websites and save this data in a database, but I get this error if the process take more than 60 seconds
502 Bad Gateway
nginx/1.14.0 (Ubuntu)
How can I extend or disable response waiting time for nginx ?
This error message...
502 Bad Gateway
nginx/1.14.0 (Ubuntu)
...using DigitalOcean (gunicorn/nginx) can occur for a number of reasons. Determining the exact cause of 502 error varies depending on the webserver you are using as well as the application server interpreting the requests.
Bad gateway errors are typically caused by a breakdown in communication between webserver and application handler. In many cases, the underlying issue is either excessive latency or exceedingly short timeout windows.
At times, 502 Bad Gateway is also caused by a misconfigured application server, when the webserver understood the request and passed it along to the appropriate handler, but a disruption of some sort occurred between the two.
Gunicorn being one of the widely used Python WSGI server, diagnosing the cause of 502 Bad Gateway error would mostly depend on the application server you are using and some of the common issues are as follows:
Gunicorn is not running: You need to ensure that Gunicorn is running using ps. To ensure Gunicorn is running you must see a similar output:
www-data@nginx0:/var/log/nginx$ ps aux | grep gunicorn
www-data 13805 0.0 1.8 52292 18460 pts/0 S 20:32 0:00 /home/www-data/test_app/bin/python /home/www-data/test_app/bin/gunicorn --error-logfile /var/log/gunicorn/errors.log -b 0.0.0.0:8080 wsgi
www-data 13836 0.0 1.5 52432 15392 pts/0 S 20:34 0:00 /home/www-data/test_app/bin/python /home/www-data/test_app/bin/gunicorn --error-logfile /var/log/gunicorn/errors.log -b 0.0.0.0:8080 wsgi
Gunicorn won’t start: At times, Gunicorn won’t start due to a typo in the configuration file or port conflict or inaccessible log directory or any combination of these situations. In these cases to check your Gunicorn configuration execute the command:
gunicorn --check-config [APP_MODULE]
NGINX is misconfigured: Incase Gunicorn is properly configured, it may be possible that NGINX is not looking for it in the right place. In those cases, open your NGINX configuration file (/etc/nginx/nginx.conf) and look for a block starting with the upstream keyword as follows:
upstream app_servers {
server 127.0.0.1:8080;
}
This setting is to configure NGINX to redirect requests for _app_servers_ to, in this case, 127.0.0.1:8080. If Gunicorn is not bound to 127.0.0.1 or is not listening on 8080, either change Gunicorn’s configuration to match NGINX’s, or change NGINX’s configuration to match Gunicorn’s. Additionally, verify that your site configuration is redirecting your application to the appropriate upstream server. To ensure that, you need to open your site’s configuration i.e. /etc/nginx/sites-enabled/your_site and look for the block defining the URL endpoint to your application. As an example:
location /my-app {
proxy_pass http://app_servers;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
Gunicorn is timing out: Incase your application takes a long time to respond ( > 30s by default), Gunicorn may return a 502 to NGINX. This can be verified by checking the Gunicorn logs (defaults to STDOUT if no logfile set). As an example,
[2016-09-21 20:33:04 +0000] [13805] [CRITICAL] WORKER TIMEOUT (pid:13810)
The above log implies that, the application is taking too long to respond to Gunicorn, resulting in the worker thread being killed because Gunicorn thinks the worker hung. In such cases, increasing Gunicorn’s max execution time would be the best solution. However, from the application and the data set processing perspective increasing timeout windows may not be the best solution and you may require to profiling and optimizing the application in use.
Tweaking read_timeout: If you still see 502 Bad Gateway after modifying Gunicorn’s timeout threshold, you need to follow these steps mentioned below to increase the timeout window for NGINX:
/etc/nginx/nginx.conf)fastcgi_read_timeout XXX; within the http block, where XXX is the timeout window in seconds (see below for an example)An example:
http {
...
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
}
1/502 Bad Gateway caused by GUNICORN
a - sudo nano /etc/systemd/system/gunicorn.service.
b-
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=sammy
Group=www-data
WorkingDirectory=/home/sammy/myproject
ExecStart=/home/sammy/myproject/myprojectenv/bin/gunicorn --access-logfile - --timeout 300 --workers 3 --bind unix:/home/sammy/myproject/myproject.sock myproject.wsgi:application
[Install]
WantedBy=multi-user.target
c -
sudo systemctl start gunicorn
sudo systemctl enable gunicorn
sudo systemctl daemon-reload
sudo systemctl restart gunicorn
2/504 Bad Gateway caused by NGINX
a - sudo nano /etc/nginx/nginx.conf
b - add those to http
client_body_timeout 999;
client_header_timeout 999;
keepalive_timeout 999;
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 999;
fastcgi_send_timeout 999;
fastcgi_read_timeout 999;
c - change this in events
worker_connections 1024;
d - service nginx reload