How to use Python multiprocessing library in a chess search algorithm?

Question

I am currently working on a chess engine, and I am looking at parallelization as a way to optimize the time it takes to search legal moves to find the best one. I found the Python module multiprocessing, however since it needs to execute in "name"=="main" how would I use it in my current class model? My code looks something like this:

class ourEngine:
    def __init__(self, board, engine_color, thinking_time=0):
        self.board = board
        self.thinking_time = thinking_time
        self.engine_color = engine_color

    def evaluate(self, board):
        # Evaluation calculation
    
    def search(self, board, depth, engine_color, alpha=float("-inf"), beta=float("inf")):
        # Minimax algorithm with alpha-beta pruning for every move in python-chess board.legal_moves

How would I implement the Pool from the multiprocessing library to process legal_moves simultaneously on all cpu's without having an init? I also realize I will have to set up a queue, but I can't find an example online anywhere

I tried implementing the module in a separate function which would then call search but it did not work.

Edit: Fixed typos

Here is an example of multiprocessing with a queue: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods — mkrieger1, Aug 13 '23 at 19:25

score 0 · Accepted Answer · answered Aug 13 '23 at 20:36

On platforms that use the spawn method of creating child processes (for example, Windows), to launch a child process Python launches a new Python interpreter into the new process's address space that re-processes the source code executing everything at global scope (import statement, global variables, function definitions, class definitions, etc.) to initialize memory and only then executes the function/method you specified to initially execute in the new process.

The problem with this scheme is that any statement at global scope that directly creates a child process or indirectly does so (e.g. by calling a function that ultimately creates a child process) would result in a recursive child process-creating loop (this doesn't actually happen since Python detects this attempt to create a new child process during this memory-initialization stage).

To get around this problem, any such direct or indirect process-creating statement must be placed in a if __name__ == '__main__': block. The recursion will be prevented since __name__ will not be '__main__' in any child process and therefore the process-creating statement will not be executed.

So imagine you modified your search method so that it creates child processes among which it distributes the searching. So any code at global scope that ultimately ends up calling this method needs to be in that special if block. For example:

class OurEngine:
    ... # Code omitted for brevity

def create_engine_and_search():
    ... # Code omitted
    engine = OurEngine(board, some_color, some_thinking_time_
    ...
    engine.search()

def main():
    ...
    create_engine_and_search()

# Invoice main:
main()

The call to main is at global scope. As a result of calling main the call to OurEngine.search will ultimately be made and we postulated that the search method creates child processes. To initialize memory for these new processes, this file will be re-interpreted in the new process and main, which is at global scope will be called again. This is a problem! But all we need to do is:

...

# Only call main if we are the main process:
if __name__ == '__main__':
    main()

What is your platform? We must tag all multiprocessing questions with our current platform since answers depend on the platform. If you were running under Linux, for example, a different mechanism is used to initialize memory for the new child processes and we do not need a special if test.

My platform MacOS, would this process work for it? Thank you, this explanation was very helpful — kiraksi, Aug 13 '23 at 22:30
I believe that by default MacOS now uses the *spawn* method to create child processes, so you would need the `if __name__ == '__main__':` block. But I also believe that MacOS supports the *fork* method to create child processes and if you use that (you need to search for how to do that) you would not need it -- but it is never wrong to have it and provides portability across all platforms. Also, see [What should I do when someone answers my question?](https://stackoverflow.com/help/someone-answers). — Booboo, Aug 13 '23 at 22:59

How to use Python multiprocessing library in a chess search algorithm?

1 Answers1