0

Let me start saying I'm not a sysadmin, I'm a php programmer with few knowledge of unix systems, so, excuse me if I say some nonsense.

I'm managing this VM with a few Wordpress installed and, in searching of what is causing some front-end bugs, I noticed that the disk, every two hours starts slowly fills itself with something I do not have recognized yet, and keeps filling and filling until there is no more space left (that is causing, I presume, the bugs the costumer sees on the front-end as mysql starts failing lot of queries) and then suddenly drop the space left to initial state.

Since I recently clean a shell installed on the filesystem through a hole in one of the many outdated wordpress installations, I now presume there is some malware that is causing that (or maybe some horrendous script written by an idiot).

Screenshot of the disk usage showed by Glances + Grafana web monitor

With df I can see that the disk is constantly fillings itself until there is no more space left.

With du I can only see that only the directories /var/log/ and /var/lib/php5/sessions are increasing their size, as it normally happens

I suspected there is something in the temp folder and, indeed, with inotifywait -m /tmp/ -r -e create -e moved_to -e modify and the output is an endless subsequence of /tmp/ MODIFY tmpf9dIVwz. The name of the file remains the same for the duration of the fills and is different the next time. I tried to find out what that file is and what process is writing/editing it but the /tmp/ folder is always empty and the file is non existent..

How can I determine what is causing the filling of the disk?

Joe Valeriana
  • 101
  • I think for a quick check, you can issue `ps -ef | grep tmp` to see what process is actually writing to tmp folder? – Simon MC. Cheng Aug 12 '16 at 11:03
  • Both `ps` and `lsof` says that the /tmp/ folder, and specifically that temp files, are written or readed by `cron`, `sh` and `php`. Is there any way to go back to the original script runned by cron, sh or php? About `cron` I already check all the crontab of any users (all empty). – Joe Valeriana Aug 12 '16 at 12:39

1 Answers1

0

Found.

ps aux | grep php showed me which php script was running and the pid matched the ones showed by ls -d /proc/[1-9]*/fd/* | grep tmp.

It is a bad written php script which tries to make an SQL UPDATE of non existent data (a bad parsed csv file) in an endless while loop.

I now guess that the failing UPDATE statement (which triggers a PHP FATAL that is ain't logged anywhere) is the cause of the saturation of the /tmp folder

Joe Valeriana
  • 101