My website got 499 & 502 + php-fpm.sock failed (11: Resource temporarily unavailable) errors

Question

My website playsongs.pk got 499, 502 and php-fpm.sock failed (11: Resource temporarily unavailable) errors when I received 300-400 active users on site.

Sample/example error logs:

2017-02-27 14:56:23 Error   162.158.78.50   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.78.170  502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.79.63   502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.78.50   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access

and After few time I got

2017-02-27 14:56:23 Error   172.68.65.4     24026#0: *2784527 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24023#0: *2784529 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.100       24028#0: *2784539 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784541 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784545 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784547 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24028#0: *2784549 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   162.158.78.50       24028#0: *2784551 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24028#0: *2784552 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error

My Nginx configuration /etc/nginx/nginx.conf:

user  nginx;
worker_processes auto; #some last versions calculate it automatically

worker_rlimit_nofile 99999;

error_log /var/log/nginx/error.log crit;

include /etc/nginx/modules.conf.d/*.conf;

events {

    worker_connections 66536;
    use epoll;
    multi_accept on;
}

http {
    include   mime.types;
    default_type  application/octet-stream;

    open_file_cache max=2048 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 5;
    open_file_cache_errors off;


    access_log off;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout  2;
    keepalive_requests 100000;
    reset_timedout_connection on;
    client_body_timeout 30;
    send_timeout 15;

client_header_timeout 12;
proxy_connect_timeout  60;
proxy_send_timeout  60;
proxy_read_timeout  60;


fastcgi_buffers 8 128k;
fastcgi_buffer_size 256k;

# reduce the data that needs to be sent over network
    gzip on;
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/v$
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";


    server_tokens off;

    include /etc/nginx/conf.d/*.conf;
}

# override global parameters e.g. worker_rlimit_nofile
include /etc/nginx/*global_params;

My php-fpm pool setting are following:

;[php-fpm-pool-settings]
pm = dynamic
pm.max_children = 5000
pm.start_servers = 50
pm.min_spare_servers = 20
pm.max_spare_servers = 70
pm.max_requests = 2000

Please note

Server have 64GB of RAM with 1 GBit dedicate INTERNET PORT and average load on CPU also does not exceed from 1.somthing % or 2.somthing %

Server OS: Centos 7.2

Control Panel: Plesk Onyx

nginx Version: nginx/1.11.4

Output of #cat /proc/$(cat /run/nginx.pid)/limits is following. May it will be helpful to solve the issue.

Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             515124               515124               processes
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       515124               515124               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                   
Max realtime priority     0                    0                   
Max realtime timeout      unlimited            unlimited            us

Output of tail /var/log/nginx/error.log

2017/03/01 15:16:15 [notice] 31867#0: getrlimit(RLIMIT_NOFILE): 1024:4096
2017/03/01 15:16:15 [notice] 31868#0: start worker processes
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31869
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31870
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31871
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31872
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31873
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31874
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31875
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31876

I suspect it's PHP playing up. Read your PHP error logs, or if you don't understand them edit your question to include them. 499 is an Nginx specific error code http://stackoverflow.com/questions/12973304/nginx-499-error-codes — Tim, Mar 01 '17 at 08:18
@Tim there is nothing in php_error logs. I have update my question. Added output of `#cat /proc/$(cat /run/nginx.pid)/limits` may it helps — Muhammad Hassaan, Mar 01 '17 at 09:55
You include nginx `access.log` in your question. That shows only a single line summary of the request. The nginx `error.log` shows the errors from PHP if there is nothing in PHP error log. Please include entries from nginx `error.log` to the question. — Tero Kilkanen, Mar 01 '17 at 10:34
OK, then it seems that your Plesk control panel is hiding the PHP error messages from you. Anyway, the problem is in your application, it cannot keep up with the requests. You need to find out your PHP error log, or how to output more stuff into it. There you will have some clues on what is not working in your application. Then you need to fix it. — Tero Kilkanen, Mar 01 '17 at 10:42
@TeroKilkanen this issue happened when I receive 400-500 active users on my site. I have test application code, no serious errors found — Muhammad Hassaan, Mar 01 '17 at 10:50
It is a scalability issue with your application. You need to figure out which part of your application is not holding up with the load. — Tero Kilkanen, Mar 01 '17 at 11:24
On my Amazon Linux / Centos system the PHP FPM log is in /var/log/php-fpm/5.6 - use "ls -ltr" to see the files most recently written to. — Tim, Mar 01 '17 at 17:55

score 2 · Answer 1 · answered Mar 01 '17 at 10:42

2

Try this:

sysctl net.core.somaxconn=65535

php/fpm/pool.d/www.conf:

listen.backlog 65535;

Or try using local IP connection instead of unix socket:

php/fpm/pool.d/www.conf:

listen 127.0.0.1:9000;

nginx/sites-enabled/sitename:

fastcgi_pass 127.0.0.1:9000;

answered Mar 01 '17 at 10:42

Selivanov Pavel

2,206
3
26
48

Thank you for response. Can you tell me were I can see current value of `listen.backlog` – Muhammad Hassaan Mar 01 '17 at 10:55
@Hassaan In fpm pool config, if not mentioned - php defaults are used – Selivanov Pavel Mar 01 '17 at 10:58
Its not mentioned in fpm pool config, how can I see php's default number for `listen.backlog`? – Muhammad Hassaan Mar 01 '17 at 11:01
@Hassaan Somewhere in PHP documentation I suppose. It may be reasonable, try tuning sysctl and see if error is gone – Selivanov Pavel Mar 01 '17 at 11:06
`net.core.somaxconn = 65535` is exits in `/etc/sysctl.conf`. Should I increase this one? – Muhammad Hassaan Mar 01 '17 at 11:14
If `sysctl net.core.somaxconn` shows 65535, then it's already ok. – Selivanov Pavel Mar 01 '17 at 11:19
Should I try something else? – Muhammad Hassaan Mar 01 '17 at 11:27
Now you can try using local IP connection instead of unix socket, second option I suggested. – Selivanov Pavel Mar 01 '17 at 11:31

My website got 499 & 502 + php-fpm.sock failed (11: Resource temporarily unavailable) errors

1 Answers1