1

I want processes in the process pool to start with some GPU-allocated array ( allocating the array on request is too time-costly ).

  1. Is there a way to customize initialization of the pool s.t. every process in the pool has a GPU-allocated array and only frees the memory when it dies?

  2. if the above is possible, then how would a task access the executing thread's GPU tensor allocated at initialization?

I can see that -- from celery's github -- pefork.py defines process_initializer function which is used as an initializer to the process pool. I'm planning on editing the source, but I am not sure how to approach this.

thanks

Darkonaut
  • 20,186
  • 7
  • 54
  • 65
AgnosticCucumber
  • 616
  • 1
  • 7
  • 21

0 Answers0