Why does Error 502 show every now and then on my site even after upgrading my server?

Question

I tried to check and read articles about this error but until now I cannot fix this. I used Cloudflare to protect my site. How can I fix the error 502 Bad Gateway

My site is currently running on Digital Ocean (PHP7.2 - Codeigniter 3, MySQL and Nginx), I recently upgraded my server because I think this may due to my site traffic. I used $5 droplet before, now I use the $15 (2 vCPU and 2GB Ram).

Nginx Error log: https://pastebin.com/XjGGjNPL

Here's the configuration:

server {
        listen 80;
        root /var/www/<site_dir>;
        index index.php index.html index.htm index.nginx-debian.html;
        server_name <domain>;

        #location ~* \.(ico|css|js|gif|jpe?g|png)(\?[0-9]+)?$ {
        #        expires max;
        #        log_not_found off;
        #}

        location / {
                try_files $uri $uri/ /index.php;
        }

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
        }

        location ~ /\.ht {
                deny all;
        }

        # Deny for accessing codes
        location ~ ^/(application|system|tests)/ {
            return 403;
        }
}

server {
        listen 80 default_server;
        server_name _;
        return 404;
}

I checked my resources using htop

Any tips about this?

Answer 1

So I raised it according to this answer: https://serverfault.com/a/844462

And easily gave anyone on the Internet the ability to shut down your website in no time.

There is absolutely no situation in which the linked answer can be any good, but I will explain why it is bad with your specific server specs below.

pm.max_children = 4000

Together with pm = ondemand, this means that as the traffic comes in fast enough, PHP-FPM will continue forking more and more processes until there is a maximum of 4k children. A typical PHP-FPM worker process consumes on average 80 MB of RAM. Multiply that to 4k and the server will need 320 GB (!!!) of RAM for this setting alone.

pm.max_requests = 0

Now this one. It controls how many requests a given PHP-FPM worker will process before it will be recycled and replaced with a new one. You typically always want to recycle PHP-FPM worker processes at some point, because PHP itself along its extensions is a memory-leak prone software in itself. So if there is no recycling going on, then high chances are that those average 80 MB for each process will easily grow more and more "fat" over time, so even 320 GB of RAM would not be sufficient to handle the traffic.

How can anyone can kill a website with these settings: by using any software which is able to issue many requests consecutively (not even necessary in parallel), ranging from curl on the command line, up to GUI tools like SEO Frog Spider. As they issue many requests continuously, thus causing the nearly unlimited (up to 4k) forking of PHP-FPM workers, eventually your server, once reached out of memory condition, will start killing processes in an attempt to gain free RAM.

The typical choice of this killing is MySQL process, and once it's sacrificed by the kernel - your website stops working.

As to how to configure PHP-FPM in a sane way, you would need to actually measure how much a PHP-FPM worker process "weights" to get the max_children setting right.

However, you can get around with simple calculations and further fine-tuning as needed. Suppose we take into account the stated average to be 80 MB. Take your server RAM and subtract what MySQL "eats" as well as any other residential programs. Suppose that this is 500 MB. So you have 1500 MB left for PHP-FPM workers.

1500 / 80 = that leaves you with pm.max_children=19.

And of course, you need to:

• Adjust pm.max_requests to at most 10000 (most safe is 500 or even lower, e.g. 100). Never 0 (which would mean no recycling)
• Have some swap configured so in case even those 19 PHP worker processes will result in out of memory condition, your website stays online.

When you monitor swap usage over time (free -m is your friend), you can get idea whether it is safe to raise PHP-FPM workers a bit further, but if anything, with modern frameworks, you will likely have a bit more than 80 MB per worker and, you will likely need to lower instead.

But that's just how to get PHP-FPM right. The real solution is efficient caching both in and outside your app:

Inside: Codeigniter caching

Outside: Varnish, NGINX FastCGI cache

Further Outside: Cloudflare cache enabled for HTML output (Cache Everything policy).

Answer 2

So I checked the logs to verify what's the issue. I'm not into servers so I leave it as default when I spin my droplet.

Upon checking /var/log/php7.2-fpm.log it shows:

'WARNING: [pool www] server reached pm.max_children setting (5), consider raising it'

So I raised it according to this answer: https://serverfault.com/a/844462

pm = ondemand
pm.max_children = 4000
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_requests = 0

I also optimized the php-fpm by raising the worker processes.

worker_processes  2; #my server has 2 vCPU cores
worker_connections  1024;

(https://www.if-not-true-then-false.com/2011/nginx-and-php-fpm-configuration-and-optimizing-tips-and-tricks/)

So far this resolves my issue whenever my server encounters spikes on web request