3

Original problem: using celery task queue, I want the processes in the process pool to use shared CUDA arrays ( i.e. instead of each process having its unique array, I want one array to be accessed by all the processes. This is safe because only reads are performed ). Pytorch's torch.multiprocessing library allows this and according to the doc, it is a simple drop-in replacement for multiprocessing.

billiard and multiprocessing seem to be two viable options for creating process pools. Currently, celery, the python task queue library, uses billiard over multiprocessing due to some feature improvements. Someone asked a question here, but the answer is not really specific.

It backports changes from the Python 2.7 and 3.x. 
The current version is compatible with Py2.4 - 2.7 and falls back to multiprocessing for 3.x, 
the next version will only support 2.6, 2.7 and 3.x.

I need to replace billiard with multiprocessing in celery's source code ( in order to use pytorch's multiprocessing library torch.multiprocessing), but would this be okay? What are the differences between multiprocessing and billiard?

Darkonaut
  • 20,186
  • 7
  • 54
  • 65
AgnosticCucumber
  • 616
  • 1
  • 7
  • 21
  • 1
    Please describe the original problem instead of your attempted solution. – Klaus D. Feb 25 '19 at 02:28
  • 2
    "but would this be okay" - I can assure that it will not be a drop-in replacement. `billiard` and `multiprocessing` are differs now **a lot**. I suggest you describe your original problem as @KlausD. proposed – Alex Yu Feb 25 '19 at 02:41
  • @KlausD. Added description of the original problem. – AgnosticCucumber Feb 25 '19 at 05:22
  • so, you want to init some variable before *fork*? – georgexsh Feb 27 '19 at 02:16
  • @georgexsh yes, and I want the variable to be shared among all the processes. However, since variable is a GPU array and the language is Python, it is a little harder to use certain CUDA IPC API that would be available in C, but isn't in Python. The only way to do is rely on Pytorch's library which implements some CUDA API. – AgnosticCucumber Feb 27 '19 at 02:20

1 Answers1

5

To those who are looking to replace celery's billiard with multiprocessingit is not possible according to this reply from celery's dev

AgnosticCucumber
  • 616
  • 1
  • 7
  • 21