1

My main goal is to open 30 child process from the parent process and then open unknown number of new processes from each of those 30 child processes. I am going to call redis for some location data from those new child processes and I am not sure how many times I have to call, it would be 100 or more than 1000. When I am calling more than 1000 times, I am crossing page limit, Error is:

OSError: [Errno 24] Too many open files

I don't want to manually increase the page limit on the production server. I want to put a throttle on the process creation, so that in no way it has more than 1000 connections open.

Here is my template code:

import multiprocessing
import time
from multiprocessing.dummy import Pool
from random import randint


class MultiProcessing():

    def second_calculation(self, index1, index2):
        random = randint(1, 10)
        time.sleep(random)
        print("Slept for: {} seconds".format(random))
        print("Call done: index: {} | index2: {}".format(index1, index2))

    def calculation(self, index):
        child_process = list()
        random = randint(1, 5)
        time.sleep(random)
        print("Slept for : {} seconds".format(random))

        counter = 0
        for i in range(0, 1500):
            counter += 1
            new_child_process = multiprocessing.Process(target=self.second_calculation, args=(index, counter))
            child_process.append(new_child_process)
            new_child_process.start()

        for process in child_process:
            process.join()

        print("Request done: {}".format(index))


if __name__ == '__main__':
    index = 0
    parent_process = list()
    m = MultiProcessing()
    for i in range(0, 30):
        index += 1
        print("Index: {}".format(index))
        new_process = multiprocessing.Process(target=m.calculation, args=(index,))
        parent_process.append(new_process)
        new_process.start()

    for process in parent_process:
        process.join()

Thank you.

O_o
  • 1,103
  • 11
  • 36
  • Hmm. I've used thread pools in related problems (not in Python though) — which I believe are certainly part of the multiprocessing module in Python. If you absolutely need to use processes as opposed to threads, though, you can certainly emulate threadpool-like behavior with a group of spawned processes with careful signal handling. The threadpool framework, however, seems really convenient for a problem like this. – chang_trenton Jun 30 '19 at 06:34
  • Is threadpool and multiprocessingPool is similar? – O_o Jun 30 '19 at 06:37
  • I'm not 100% sure how it's internally implemented in Python — I'm simply referring to the general concurrency framework: https://en.wikipedia.org/wiki/Thread_pool – chang_trenton Jun 30 '19 at 06:38
  • Okay. I will take a look into it – O_o Jun 30 '19 at 06:40
  • You **are** actually using threads since you're using the `multiproccesing.duumy` module which is a wrapper on `threading` module. have a read in ths [interesting article](https://www.codementor.io/lance/simple-parallelism-in-python-du107klle) that covers the subject – Tomerikoo Jun 30 '19 at 09:44
  • A fix to my comment above ^ - I just noticed you are not actually using the imported Pool but rather directly creating processes. Try reading in the article I attached and see if you can transform your code to use a Pool. I have got to use it in several projects and it really works like magic. It's super fun – Tomerikoo Jun 30 '19 at 09:47

0 Answers0