Linux, waitpid, WNOHANG, child process, zombie

Question

I running my program as daemon.

Father process only wait for child process, when it is dead unexpected, fork and wait again.

for (; 1;) {
  if (fork() == 0) break;
  int sig = 0;
  for (; 1; usleep(10000)) {
    pid_t wpid = waitpid(g->pid[1], &sig, WNOHANG);
    if (wpid > 0) break;
    if (wpid < 0) print("wait error: %s\n", strerror(errno));
  }
}

But when child process being killed with -9 signal, the child process goes to zombie process.

waitpid should return the pid of child process immediately!
But waitpid got the pid number after about 90 seconds,

cube     28139  0.0  0.0  70576   900 ?        Ss   04:24   0:07 ./daemon -d
cube     28140  9.3  0.0      0     0 ?        Zl   04:24 106:19 [daemon] <defunct>

Here is the strace of the father

The father does not get stuck, wait4 was called always.

strace -p 28139
Process 28139 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 0
wait4(28140, 0x7fff08a2681c, WNOHANG, NULL) = 0
nanosleep({0, 10000000}, NULL)          = 0
wait4(28140, 0x7fff08a2681c, WNOHANG, NULL) = 0

About 90 seconds later father got the SIGCHILD and wait4 returned the pid of the dead child.

--- SIGCHLD (Child exited) @ 0 (0) ---
restart_syscall(<... resuming interrupted call ...>) = 0
wait4(28140, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], WNOHANG, NULL) = 28140

Why the child process does not exit immediately? On the contrary, it turns into zombie unexpectedly.

score 3 · Accepted Answer · answered Apr 30 '14 at 15:45

3

I finally find out there were some fd leaks during deep tracing by lsof.

After fd leaks were fixed, the problem was gone.

answered Apr 30 '14 at 15:45

Cube

181
2
9

score 1 · Answer 2 · answered Mar 29 '14 at 18:20

1

You could simply use

  for (;;) {
    pid_t wpid = waitpid(-1, &sig, 0);
    if (wpid > 0) break;
    if (wpid < 0) print("wait error: %s\n", strerror(errno));
  }

instead of sleep for a while and try again.

answered Mar 29 '14 at 18:20

Lee Duhem

14,695
3
29
47

I have the same problem with http://stackoverflow.com/questions/19059615/process-stuck-in-exit-shows-as-zombie-but-cannot-be-reaped/22734421#22734421 – Cube Mar 30 '14 at 08:48
The process was actually doing useful work all this time. The process held the last reference to a large file on a slow filesystem. When the process terminates, the last reference to the file is release, forcing the OS to reclaim the space. The file was so large that this required tens of thousands of I/O operations, taking 10 minutes or more – Cube Mar 30 '14 at 08:48
@Cube Maybe the similar phenomenon that you observed caused by a similar reason, i.e. when you find out that the child process is in zombie state by `ps`, the child process actually does not finished its exit yet, the kernel probably doing some useful works, such as flush filesystem, in behalf of that child process at that time. – Lee Duhem Mar 30 '14 at 13:04
@leeduherm Do you know is there any way to handle such 'Useful works', maybe i can do the works before sending exit signal to child process, and child process should exit immediately instead of turning into zombie status for a while. – Cube Mar 31 '14 at 03:08
@Cube Did you see the question link you provided was answered? If you have find your same problem elsewhere in stack, it is not good to repeat the problem again with more detail. Your question from Mar 31 might be better as a separate question. It would be interesting for you perhaps to look into better resource allocation and process ordering specifically to avoid different cases of zombie appearances, providing an example that has been a little more abstracted. :) – Chris Marie Apr 29 '14 at 21:02
@Ma-at I finally find out there were some fd leaks during deep trace by lsof. After fd leaks were fixed, the problem was gone. – Cube Apr 30 '14 at 15:40
@Cube that is awesome. You should put write that as an answer then select it to answer you own question. The SO goal is to archive question and answers so I think you may get rep points. Plus if you don't mind, I am actually a little curious about the your leak deep trace and the changes that made it non leaky. – Chris Marie Apr 30 '14 at 15:45

score 1 · Answer 3 · answered Apr 29 '14 at 21:13

1

It looks to me like waitpid is not returning the child pid immediately simply because that process is not available.

Furthermore, it looks like you actually want your code to do this because you specify waitpid() with the NOHANG option, which, prevents blocking, essentially allowing the parent to move on if the child pid is not available.

Maybe your process using something you didn't expect? Can you trace its activity to see if you find the bottleneck?

Here is a pretty useful link that might help you: http://infohost.nmt.edu/~eweiss/222_book/222_book/0201433079/ch08lev1sec6.html

answered Apr 29 '14 at 21:13

Chris Marie

1,382
1
15
25

1

As we known, when a process exit, the OS will recycle all resources which was allocated: memory allocated by `malloc`, file name specifies allocated by `open`, domain specifies allocated by `socket`. The file name specifies recycling will case a lot of `OS IO`, if the file was huge, tons of `OS IO` would be done and the process had to be turned into zombie during the execution. The `lsof` command will display all files which opened already. I finally found out i forgot close the `fd` after open it, so many leaked `fds` case the problem. – Cube May 01 '14 at 16:54