3

I am trying to develop part of system that has the following requirement:

  1. send health status to a remote server(every X seconds)
  2. receive request for executing/canceling CPU bound job(s)(for example - clone git repo, compile(using conan) it.. etc).

I am using the socketio.AsyncClient to handle these requirements.

class CompileJobHandler(socketio.AsyncClientNamespace):
    def __init__(self, namespace_val):
        super().__init__(namespace_val)
        // some init variables

    async def _clone_git_repo(self, git_repo: str):
        // clone repo and return its instance
        return repo

    async def on_availability_check(self, data):
         // the health status
         await self.emit('availability_check', " all good ")

    async def on_cancel_job(self, data):
         // cancel the current job

    def _reset_job(self):
       // reset job logics

    def _reset_to_specific_commit(self, repo: git.Repo, commit_hash: str):
        // reset to specific commit

    def _compile(self, is_debug):
        // compile logics - might be CPU intensive

    async def on_execute_job(self, data):
        // **request to execute the job(compile in our case)**

        try:
            repo = self._clone_git_repo(job_details.git_repo)
            self._reset_to_specific_commit(repo, job_details.commit_hash)
            self._compile(job_details.is_debug)

            await self.emit('execute_job_response',
                            self._prepare_response("SUCCESS", "compile successfully"))

        except Exception as e:
            await self.emit('execute_job_response',
                            self._prepare_response(e.args[0], e.args[1]))

        finally:
            await self._reset_job()

The problem with the following code is that when execute_job message arrives, there is a blocking code running that blocks the whole async-io system.

to solve this problem, I have used the ProcessPoolExecutor and the asyncio event loop, as shown here: https://stackoverflow.com/questions/49978320/asyncio-run-in-executor-using-processpoolexecutor

after using it, the clone/compile functions are executed in another process - so that almost achieves my goals.

the questions I have are:

  1. How can I design the code of the process more elegantly?(right now I have some static functions, and I don't like it...)

one approach is to keep it like that, another one is to pre-initialize an object(let's call it CompileExecuter and create instance of this type, and pre-iniailize it prior starting the process, and then let the process use it)

  1. How can I stop the process in the middle of its execution?(if I received on_cancel_job request)

  2. How can I handle the exception raised by the process correctly?

Other approaches to handle these requirements are welcomed

nonamer92
  • 1,887
  • 1
  • 13
  • 24
  • Are you sure you need `ProcessPoolExecutor` for this? Since you are spawning other processes (the git executable), it seems that a `ThreadPoolExecutor` would do just as well, and might simplify some things. Also, if you are just interacting with `git`, you could use `asyncio.subprocess` and then your code wouldn't need to be blocking in the first place, and you could handle interruptions by catching `asyncio.CancelledError` and terminating the subprocess. – user4815162342 Oct 26 '19 at 16:44
  • @user4815162342 I don't want to use it because I use gitPython and conan modules.. I'm not running it as an external program. I hope I clarified it – nonamer92 Oct 27 '19 at 11:58

0 Answers0