0

We are facing a problem with extremely high data writes to disk and huge amounts of traffic over loopback interface.

Environment: Debian 10.3 (4.19.0.8) - virtualized in hyper-v nginx/1.14.2 php-fpm 7.3 mariadb Ver 15.1 Distrib 10.3.22-MariaDB

The machine is serving 5 php Laravel portals with some dynamic but mostly static content (e-books with a lot of assets - pictures, audio, video material). Having 300-1500 concurrent users alltogether, the disk writes varies all the time from 50 - 200 MB/s (measured with iotop), data read is practically 0. In one day there could be more than 1 TB of data written to disk, having inbound traffic on eth0 of just around 10 GB. Free space on hdd is being practically unchanged. Disk writes correlate to traffic that passes over loopback interface. It looks like the nginx/fpm are all the time "rewriting" the data on the disk.

The machine has dedicated 48 cores, 48 GB of ram, raid-6 SSD, swap partition never touched.

So far we've tried many possible cfgs (switching php-fpm from socket to tcpip, sendfile off/on, tweaking nginx fastcgi_cache) but with no luck. Here is the current cfg: https://pastebin.com/S6rdjbvE

Has anyone experienced such behaviour? We are quite out of ideas..

Thank you. Nejc

nb_nwl
  • 1
  • did you triee with nginx to cache and also use memcached? – djdomi Mar 21 '20 at 10:23
  • True, the problem was in session files that were being constantly overwritten. Putting them to memcached reduced the amount of data by 80%, thanks a lot! – nb_nwl Mar 23 '20 at 18:32

1 Answers1

0

Run the below command and try to set the value based on the result in www.conf

ps --no-headers -o "rss,cmd" -C php-fpm7.3 | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"Mb") }'

Calculate the values 

min_spare_servers + (max_spare_servers - min_spare_servers) / 2
Vinoth Rc
  • 172
  • 5
  • Thanks, it was not a problem in fpm spare servers but in session files. Storing them to memcached solved the issue, however i still see a lot of writes like this. I persume this is some buffer-cache for fastcgi data. Any chance to have this stored in memory instead of disk? nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)
    nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)
    nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)
    nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)
     – nb_nwl Mar 23 '20 at 18:33
  • if your not plenty on ram you may create a zram or tmpfs device – djdomi Mar 27 '20 at 10:06