2

I have a cron that generated a lot (15GB) of PHP warnings and was writing them in a log file.

I killed the process and as a temporary measure I stopped redirecting stderr to stdout so I don't fill up my storage.

After the change, I continued getting the same "high IO" warnings and the server storage was increasing temporarily and then going back to previous size.The original log file was empty.

I found one file using lsof -p <PID of cron> that was getting bigger by the second.

sh      25626 root   10u   REG  202,1 21280244045     773 /tmp/tmpfZ14vFH (deleted)

This file is marked as deleted and I cannot find it in the /tmp directory.

Please provide some insight on this.

Is the OS writing the stderr in a tempprary file while the process is running?

rwms
  • 155
  • 1
  • 2
  • 7
  • 2
    Deleting a file with an open file handle won't actually remove the data until the last open file handle releases the file. Kill/restart the process owning the file handle and then a deleted file will finally disappear. – HBruijn Nov 02 '17 at 12:03

1 Answers1

2

You must cycle (start/stop) the process that has a hold of the file handle.

Check with ps -ef |grep 25626 as a beginning point to find the process that's holding it and if possible, cycle it. It will allow the file handle to release and the disk space will be cleared.

I know you've already done this (found that sh is the process in question), but here is my go-to command in my notes (likely found on stack exchange in the past) for finding these mystery deleted files: lsof | grep deleted | numfmt --field=7 --to=iec | head

emmdee
  • 2,187
  • 12
  • 36
  • 60
  • I have edited my question. Thanks for the direction. Hopefully it's more clear now. – rwms Nov 03 '17 at 08:02
  • You mention "cron" but what exactly is cron running? A script you wrote? A 1 line command? Did you check the process itself? `ps -ef |grep 25626` that should be the process that you need to kill/stop/restart to clear the disk space. If it's supposed to just be a cron job that runs and exits, it's likely in some sort of endless loop/etc that's causing it to remain running so it cannot release it's file handle on the tmp files. Something isn't exiting properly and/or not releasing its files properly and it appears to be PID 25626 – emmdee Nov 03 '17 at 16:56
  • Cron is running a PHP script. The warnings are not the issue. What I'm asking is: while the process is running and streaming to stderr, if stderr is not redirected to a log file, where does it go ? I was under the impression that it was kept in memory. However it seems that it's saved in some temp file that's deleted immediately after the process terminates. – rwms Nov 07 '17 at 13:17
  • Standard outputs are streams and not naturally retained. They will stream to terminal by default or their parent process. They don't write to a file unless you (or a program) tell them to. Re-examine your configs to see what's logging or redirecting stderr output. – emmdee Nov 07 '17 at 17:33
  • So in this case (no terminal and no file redirect) stderr is streaming to it's parent process ? How much higher is it pushed up the process tree ? What I'm asking is where does it end up ? – rwms Nov 08 '17 at 13:31