4

One of my web server was DDOS attacked. All is well except there are millions of PHP session files used up 100% inodes of the partition. There is only one partition for the entire /

Tried several solutions, but only worked to some extend. https://unix.stackexchange.com/questions/37329/efficiently-delete-large-directory-containing-thousands-of-files?newreg=07f276292205457ab9975a0ea68e9273

http://www.slashroot.in/which-is-the-fastest-method-to-delete-files-in-linux

After freed up 8% of inodes, the disk become extremely slow to delete anything more.

rm -f filename* 

rsync -a --delete empty_dir/    yourdirectory/

perl -e 'for(<*>){((stat)[9]<(unlink))}'

And the disk look like this now

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/vda1      2621440 2385895 235545   92% /
tmpfs           128789       1 128788    1% /dev/shm

There are still 6million+ small files in a dir. The above methods delete at about 2 files/sec

I read about b-tree re-balancing. But how do I diagnose/solve the slow delete problem?

```

Reed
  • 287
  • 1
  • 3
  • 10
  • Consider also using a different `session.save_handler`. Some will handle session creation DoS much better than the default filesystem one. – derobert May 04 '17 at 19:52

1 Answers1

8

A quick thing to do is to move/rename your current /tmp directory and create a new one so that normal usage of /tmp isn't impacted anymore. 

mkdir /newtmp
chmod 1777 /newtmp
mv /tmp /tmp-old && mv /newtmp /tmp

and maybe you need to do the same for /var/tmp as well. That allows your to peacefully clean up /tmp-old.

As a mitigating measure:

Consider using some of your memory to create a separate tempfs partition to use as storage for your PHP session files, which will somewhat limit the impact on the rest of your system.

i.e. 

mkdir /var/cache/php
chmod 1777  /var/cache/php
mount -t tmpfs size=500M  /var/cache/php

and edit your php.ini and set 

session.save_path = "/var/cache/php"
HBruijn
  • 77,029
  • 24
  • 135
  • 201
  • After moved the large dir, I found out a new round of DDOS attack was coming and creating hundreds of files per sec. That is the cause of slow io performance. Had to stop the webserver to clean up files. – Reed Apr 26 '17 at 14:54