0

I'm trying to code a kind of task manager in Python. It's based on a job queue, the main thread is in charge of adding jobs to this queue. I have made this class to handle the jobs queued, able to limit the number of concurrent processes and handle the output of the finished processes.

Here comes the problem, the _check_jobs function I don't get updated the returncode value of each process, independently of its status (running, finished...) job.returncode is always None, therefore I can't run if statement and remove jobs from the processing job list.

I know it can be done with process.communicate() or process.wait() but I don't want to block the thread that launches the processes. Is there any other way to do it, maybe using a ProcessPoolExecutor? The queue can be hit by processes at any time and I need to be able to handle them.

Thank you all for your time and support :)

from queue import Queue 
import subprocess
from threading import Thread
from time import sleep


class JobQueueManager(Queue):
    def __init__(self, maxsize: int):
        super().__init__(maxsize)
        self.processing_jobs = []
        self.process = None
        self.jobs_launcher=Thread(target=self._worker_job)
        self.processing_jobs_checker=Thread(target=self._check_jobs_status)
        self.jobs_launcher.start()
        self.processing_jobs_checker.start()


    def _worker_job(self):
        while True:
            # Run at max 3 jobs concurrently
            if self.not_empty and len(self.processing_jobs) < 3:
                # Get job from queue
                job = self.get()
                # Execute a task without blocking the thread
                self.process = subprocess.Popen(job)
                self.processing_jobs.append(self.process)
                # util if  queue.join() is used to block the queue
                self.task_done()
            else:
                print("Waiting 4s for jobs") 
                sleep(4)

    def _check_jobs_status(self):
        while True:
            # Check if jobs are finished
            for job in self.processing_jobs:
                # Sucessfully completed
                if job.returncode == 0:
                    self.processing_jobs.remove(job)
            # Wait 4 seconds and repeat
            sleep(4)


def main():

    q = JobQueueManager(100)
    task = ["stress", "--cpu", "1", "--timeout", "20"]

    for i in range(10): #put 10 tasks in the queue
        q.put(task)

    q.join() #block until all tasks are done


if __name__ == "__main__":
    main()
gongar
  • 1
  • 1
  • I think variable `process` is defined inside `_worker_job` scope then will return an error, but if you want to get help with your question please add the whole code of your program and also you should check [How to create a Minimal, Reroducible Example](https://stackoverflow.com/help/minimal-reproducible-example) – user11717481 Feb 11 '22 at 16:54
  • Thanks for your support! I have read the reprex guide and edited the question, I hope it helps your understanding. – gongar Feb 14 '22 at 10:31

1 Answers1

0

I answer myself, I have come up with a working solution. The JobExecutor class handles in a custom way the Pool of processes. The watch_completed_tasks function tries to watch and handle the output of the tasks when they are done. This way everything is done with only two threads and the main thread is not blocked when submitting processes.

import subprocess
from threading import Timer
from concurrent.futures import ProcessPoolExecutor, as_completed
import logging


def launch_job(job):
    process = subprocess.Popen(job, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    print(f"launching {process.pid}")
    return [process.pid, process.stdout.read(), process.stderr.read()]


class JobExecutor(ProcessPoolExecutor):
    def __init__(self, max_workers: int):
        super().__init__(max_workers)
        self.futures = []
        self.watch_completed_tasks()

    def submit(self, command):
        future = super().submit(launch_job, command)
        self.futures.append(future)
        return future

    def watch_completed_tasks(self):
        # Manage tasks completion
        for completed_task in as_completed(self.futures):
            print(f"FINISHED  task with PID {completed_task.result()[0]}")
            self.futures.remove(completed_task)

        # call this function evevery 5 seconds
        timer_thread = Timer(5.0, self.watch_completed_tasks)
        timer_thread.setName("TasksWatcher")
        timer_thread.start()

def main():

    executor = JobExecutor(max_workers=5)
    for i in range(10):
        task = ["stress",
        "--cpu", "1",
        "--timeout", str(i+5)]
        executor.submit(task)
gongar
  • 1
  • 1