Process continues to hold file after file deletion

Question

I created a log file by running the iostat command to a text file, and ran the command in the background using nohup.

#nohup iostat -xm 5 > /z/logfile.txt &

Later on, I created a cronjob that runs every ten minutes doing the same as above, after I realized my process was being killed by a reboot. I've also setup log-rotation as below:

/z/logfile.txt {
        size 20M
        rotate 0
        create 0644 root root
        missingok
        notifempty

}

Now I have realized that the logfile.txt gets deleted but the iostat command keeps pointing at deleted files as shown by the lsof -n | grep deleted command. There the disk space is not freed.

How can I make sure the files are rotated and thereafter iostat points to the newly created file, freeing up disk space?

Any ideas how to set it up correctly?

Do you mean that every 10 minutes you start a new iostat that runs forever putting output into the same file as all the other previously spawned iostats? — William Pursell, May 16 '17 at 01:59
So after 61 minutes you have a total of 7 iostats running, and that count increments every 10 minutes? I'm pretty sure that's not what you want. — William Pursell, May 16 '17 at 12:53

score 1 · Answer 1 · answered May 16 '17 at 02:59

1

One solution would be to write a program that will read from iostat, write to the output file, and accept a signal to reopen the file. For example, if you did: iostat -xm 5 | log-daemon /z/logfile.txt where log-daemon is a simple script like:

#!/bin/bash
echo $$ > /var/run/log-daemon
exec > $1
trap 'exec > $1' SIGHUP
read line
while test $? -le 0; do 
        echo $line
        read line
done

Then add a postrotate clause in the logrotate config to send a HUP to the log-daemon:

postrotate
               /usr/bin/kill -HUP $(cat /var/run/log-daemon)

answered May 16 '17 at 02:59

William Pursell

204,365
48
270
300

OK thanks, that's a good idea. But what is causing iostat to keep referencing to the deleted file, doesn't logrotate have any options to solve the issue? – sawe May 16 '17 at 03:13
The typical solution is for logrotate to send a signal to the daemon writing the logfile to tell it to reopen the file. But iostat doesn't have that functionality, so you need to provide it some other way. – William Pursell May 16 '17 at 03:25
Logrotate can't force another process to close an open file, other than by killing the other process. You could get logrotate to kill the iostat but that might not be what you want. – rici May 16 '17 at 03:28
if it kills iostat, the cronjob re-runs iostat every 10minutes – sawe May 16 '17 at 03:44

score 0 · Answer 2 · answered May 16 '17 at 03:33

Would pointing your cronjob iostat command at a softlink not work?

ln -s /z/logfile.txt iostat_link.txt
nohup iostat -xm 5 > /z/iostat_link.txt &

I haven't used logrotate before, but I tested this by manually changing the file in the background while I had this running:

#Make the files
touch afile1.txt
ln -s afile1.txt file.txt

#Kick off loop
for i in {1..1000};do echo "running still $i" >> file.txt;sleep 3;done &

[localhost (2017-05-15 20:30:55) IP: 26.176 ~]# cat afile1.txt
running still 7
running still 8
running still 9

#Change the file out from under the loop
mv afile1.txt afile1.txt.backup;touch afile1.txt

[localhost (2017-05-15 20:31:21) IP: 26.176 ~]# cat afile1.txt
running still 15
running still 16
running still 17

score 0 · Answer 3 · answered May 16 '17 at 04:43

0

Check if file system where log is recording is full. If you have such a case find and kill process or reboot server in worst case.

answered May 16 '17 at 04:43

user3661564

107
2
2
8

Its not that urgent for me to reboot the server. I just want to fix it as an ongoing issue. – sawe May 16 '17 at 05:12

Process continues to hold file after file deletion

3 Answers3