0

I have a multiprocessing programs in python, which spawns several sub-processes and manages them (restarting them if the children identify problems, etc). Each subprocess is unique and their setup depends on a configuration file. The general structure of the master program is:

def main():

    messageQueue = multiprocessing.Queue()
    errorQueue = multiprocessing.Queue()

    childProcesses = {}

    for required_children in configuration:
        childProcesses[required_children] = MultiprocessChild(errorQueue, messageQueue, *args, **kwargs)

    for child_process in ChildProcesses:
        ChildProcesses[child_process].start()

    while True:
        if local_uptime > configuration_check_timer:    # This is to check if configuration file for processes has changed. E.g. check every 5 minutes
            reload_configuration()
            killChildProcessIfConfigurationChanged()
            relaunchChildProcessIfConfigurationChanged()

        # We want to relaunch error processes immediately (so while statement)
        # Errors are not always crashes. Sometimes other system parameters change that require relaunch with different, ChildProcess specific configurations.
        while not errorQueue.empty():
            _error_, _childprocess_ = errorQueue.get()
            killChildProcess(_childprocess_)
            relaunchChildProcess(_childprocess)
            print(_error_)

        # Messages are allowed to lag if a configuration_timer is going to trigger or errorQueue gets something (so if statement)
        if not messageQueue.empty():
            print(messageQueue.get())

Is there a way to prevent the contents of the infinite while True loop take up 100pct CPU. If I add a sleep event at the end of the loop (e.g. sleep for 10s), then errors will take 10s to correct, ans messages will take 10s to flush.

If on the other hand, there was a way to have a time.sleep() for the duration of the configuration_check_timer, while still running code if messageQueue or errorQueue get stuff inside them, that would be nice.

asuprem
  • 554
  • 1
  • 5
  • 17
  • 1
    Use a [Pool](https://docs.python.org/3.7/library/multiprocessing.html) (first example). – Klaus D. May 16 '19 at 02:37
  • 1
    You can do using `nice` like this: https://stackoverflow.com/questions/42103367/limit-total-cpu-usage-in-python-multiprocessing you can use sleep to ensure it never consumes ALL of your processing power in this way. ``` from time import sleep def f(i): sleep(0.01) return math.sqrt(i) ``` – EchoCloud May 16 '19 at 03:14

0 Answers0