how to kill zombie processes created by multiprocessing module?

Question

I'm very new to multiprocessing module. And I just tried to create the following: I have one process that's job is to get message from RabbitMQ and pass it to internal queue (multiprocessing.Queue). Then what I want to do is : spawn a process when new message comes in. It works, but after the job is finished it leaves a zombie process not terminated by it's parent. Here is my code:

Main Process:

 #!/usr/bin/env python

 import multiprocessing
 import logging
 import consumer
 import producer
 import worker
 import time
 import base

 conf = base.get_settings()
 logger = base.logger(identity='launcher')

 request_order_q = multiprocessing.Queue()
 result_order_q = multiprocessing.Queue()

 request_status_q = multiprocessing.Queue()
 result_status_q = multiprocessing.Queue()

 CONSUMER_KEYS = [{'queue':'product.order',
                   'routing_key':'product.order',
                   'internal_q':request_order_q}]
 #                 {'queue':'product.status',
 #                  'routing_key':'product.status',
 #                  'internal_q':request_status_q}]

 def main():
     # Launch consumers
     for key in CONSUMER_KEYS:
         cons = consumer.RabbitConsumer(rabbit_q=key['queue'],
                                        routing_key=key['routing_key'],
                                        internal_q=key['internal_q'])
         cons.start()

     # Check reques_order_q if not empty spaw a process and process message
     while True:
         time.sleep(0.5)
         if not request_order_q.empty():
             handler = worker.Worker(request_order_q.get())
             logger.info('Launching Worker')
             handler.start()

 if __name__ == "__main__":
     main()

And here is my Worker:

 import multiprocessing
 import sys 
 import time
 import base

 conf = base.get_settings()
 logger = base.logger(identity='worker')

 class Worker(multiprocessing.Process):

     def __init__(self, msg):
         super(Worker, self).__init__()
         self.msg = msg 
         self.daemon = True

     def run(self):
         logger.info('%s' % self.msg)
         time.sleep(10)
         sys.exit(1)

So after all the messages gets processed I can see processes with ps aux command. But I would really like them to be terminated once finished. Thanks.

You need the parent to `join` its children. See the links in this answer: http://stackoverflow.com/questions/18477320/python-multiprocessing-kill-processes?lq=1 — Markku K., Oct 11 '13 at 15:54
Zombie processes are already dead, and therefore cannot be killed. They have already terminated. What you are trying to do is to "reap" them and remove their entry from the process table. — William Pursell, Oct 11 '13 at 15:56
A multiprocessing.Process object has an `is_alive()` member function that you can call to see if it is still running. — Markku K., Oct 11 '13 at 16:03
Thank you @MarkkuK. that's exactly what I need. it should really be an answer — Vor, Oct 11 '13 at 16:13

0 _ · Answer 1 · 2021-09-18T11:37:18.137

Using multiprocessing.active_children is better than Process.join. The function active_children cleans any zombies created since the last call to active_children. The method join awaits the selected process. During that time, other processes can terminate and become zombies, but the parent process will not notice, until the awaited method is joined. To see this in action:

import multiprocessing as mp
import time


def main():
    n = 3
    c = list()
    for i in range(n):
        d = dict(i=i)
        p = mp.Process(target=count, kwargs=d)
        p.start()
        c.append(p)
    for p in reversed(c):
        p.join()
        print('joined')


def count(i):
    print(f'{i} going to sleep')
    time.sleep(i * 10)
    print(f'{i} woke up')


if __name__ == '__main__':
    main()

The above will create 3 processes that terminate 10 seconds apart each. As the code is, the last process is joined first, so the other two, which terminated earlier, will be zombies for 20 seconds. You can see them with:

ps aux | grep Z

There will be no zombies if the processes are awaited in the sequence that they will terminate. Remove the call to the function reversed to see this case. However, in real applications we rarely know the sequence that children will terminate, so using the method multiprocessing.Process.join will result in some zombies.

The alternative active_children does not leave any zombies. In the above example, replace the loop for p in reversed(c): with:

while True:
    time.sleep(1)
    if not mp.active_children():
        break

and see what happens.

while multiprocessing.active_children(): time.sleep(1) – Pelle Aug 26 '16 at 12:26 — Pelle, Aug 26 '16 at 12:26

score 15 · Accepted Answer · edited May 23 '17 at 12:25

15

A couple of things:

Make sure the parent joins its children, to avoid zombies. See Python Multiprocessing Kill Processes
You can check whether a child is still running with the is_alive() member function. See http://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process

edited May 23 '17 at 12:25

Community

1
1

answered Oct 11 '13 at 16:23

Markku K.

3,840
19
20

score 5 · Answer 3 · answered Apr 15 '15 at 23:15

5

Use active_children. multiprocessing.active_children

answered Apr 15 '15 at 23:15

Al Conrad

1,528
18
12

how to kill zombie processes created by multiprocessing module?

3 Answers3

Linked