I want processes in the process pool to start with some GPU-allocated array ( allocating the array on request is too time-costly ).
Is there a way to customize initialization of the pool s.t. every process in the pool has a GPU-allocated array and only frees the memory when it dies?
if the above is possible, then how would a task access the executing thread's GPU tensor allocated at initialization?
I can see that -- from celery's github -- pefork.py defines process_initializer
function which is used as an initializer to the process pool. I'm planning on editing the source, but I am not sure how to approach this.
thanks