Due to an upcoming event, we're expecting high traffic (about 2000 simultaneous users) for several weeks on our Magento2 powered e-commerce website running on Ubuntu, NGINX and PHP7.1-fpm. Since our catalog is quite large we have upgraded our Digitalocean droplet to the highest specs bein 192 GB Ram and 32 vCPUs.
The site works blazing fast, however, once we reach 800+ users it starts to get quite slow and after a while, it will end up in showing 502 Bad Gateway errors which we can't really afford during this temporary event.
Our most recent errors are now pm.start_servers, or pm.min/max_spare_servers related, which are:
[16-Mar-2020 21:50:35] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle$
[16-Mar-2020 21:50:36] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idl$
We've tried several settings, but nothing seems to be working reliably. Here are a few settings:
/etc/php/7.1/fpm/pool.d/www.conf:
pm = dynamic
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 50
;pm.process_idle_timeout = 10s;
pm.max_requests = 0
So now we're looking for a solution or a way to have these calculated exactly? Are there any other considerations or settings we should change?
Here our Magento nginx.conf:
root $MAGE_ROOT/pub;
index index.php;
autoindex off;
charset UTF-8;
error_page 404 403 = /errors/404.php;
#add_header "X-UA-Compatible" "IE=Edge";
# PHP entry point for setup application
location ~* ^/setup($|/) {
root $MAGE_ROOT;
location ~ ^/setup/index.php {
fastcgi_pass fastcgi_backend;
fastcgi_param PHP_FLAG "session.auto_start=off \n suhosin.session.cryptua=off";
fastcgi_param PHP_VALUE "memory_limit=2048M \n max_execution_time=18000";
fastcgi_read_timeout 600s;
fastcgi_connect_timeout 600s;
fastcgi_buffers 256 16k;
fastcgi_max_temp_file_size 0;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~ ^/setup/(?!pub/). {
deny all;
}
location ~ ^/setup/pub/ {
add_header X-Frame-Options "SAMEORIGIN";
}
}
# PHP entry point for update application
location ~* ^/update($|/) {
root $MAGE_ROOT;
location ~ ^/update/index.php {
fastcgi_split_path_info ^(/update/index.php)(/.+)$;
fastcgi_pass fastcgi_backend;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
include fastcgi_params;
}
# Deny everything but index.php
location ~ ^/update/(?!pub/). {
deny all;
}
location ~ ^/update/pub/ {
add_header X-Frame-Options "SAMEORIGIN";
}
}
location / {
try_files $uri $uri/ /index.php$is_args$args;
}
location /pub/ {
location ~ ^/pub/media/(downloadable|customer|import|theme_customization/.*\.xml) {
deny all;
}
alias $MAGE_ROOT/pub/;
add_header X-Frame-Options "SAMEORIGIN";
}
location /static/ {
# Uncomment the following line in production mode
# expires max;
# Remove signature of the static files that is used to overcome the browser cache
location ~ ^/static/version {
rewrite ^/static/(version[^/]+/)?(.*)$ /static/$2 last;
}
location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
# add_header Cache-Control "public";
add_header X-Frame-Options "SAMEORIGIN";
expires +1y;
if (!-f $request_filename) {
rewrite ^/static/?(.*)$ /static.php?resource=$1 last;
}
}
location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
# add_header Cache-Control "no-store";
add_header X-Frame-Options "SAMEORIGIN";
expires off;
if (!-f $request_filename) {
rewrite ^/static/?(.*)$ /static.php?resource=$1 last;
}
}
if (!-f $request_filename) {
rewrite ^/static/?(.*)$ /static.php?resource=$1 last;
}
add_header X-Frame-Options "SAMEORIGIN";
}
location /media/ {
try_files $uri $uri/ /get.php$is_args$args;
location ~ ^/media/theme_customization/.*\.xml {
deny all;
}
location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
# add_header Cache-Control "public";
add_header X-Frame-Options "SAMEORIGIN";
expires +1y;
try_files $uri $uri/ /get.php$is_args$args;
}
location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
# add_header Cache-Control "no-store";
add_header X-Frame-Options "SAMEORIGIN";
expires off;
try_files $uri $uri/ /get.php$is_args$args;
}
add_header X-Frame-Options "SAMEORIGIN";
}
location /media/customer/ {
deny all;
}
location /media/downloadable/ {
deny all;
}
location /media/import/ {
deny all;
}
location /Preread/ {
#add_header Cache-Control "no-store";
#add_header X-Frame-Options "SAMEORIGIN";
root /var/www/html/;
#try_files $uri $uri/ /Preread/index.php?$args;
#try_files $uri $uri/ /Preread/index.php?q=$uri&$args;
#index index.php index.html index.htm;
#try_files $uri $uri/ /index.php?q=$uri&$args;
try_files $uri $uri/ /index.php?q=$uri&$args;
#try_files $uri $uri/ /get.php$is_args$args;
#allow all;
# try_files $uri $uri/ /get.php$is_args$args;
}
# PHP entry point for main application
location ~ (index|get|static|report|404|503|cs|davidfile|health_check)\.php$ {
try_files $uri =404;
fastcgi_pass fastcgi_backend;
fastcgi_buffers 1024 4k;
fastcgi_param PHP_FLAG "session.auto_start=off \n suhosin.session.cryptua=off";
fastcgi_param PHP_VALUE "memory_limit=1048M \n max_execution_time=18000";
fastcgi_read_timeout 600s;
fastcgi_connect_timeout 600s;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
gzip on;
gzip_disable "msie6";
gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types
text/plain
text/css
text/js
text/xml
text/javascript
application/javascript
application/x-javascript
application/json
application/xml
application/xml+rss
image/svg+xml;
gzip_vary on;
# Banned locations (only reached if the earlier PHP entry point regexes don't match)
location ~* (\.php$|\.htaccess$|\.git) {
deny all;
}
Some expert help would be greatly appreciated, thank you
