2

I have access to a couple dozens Dask servers without GPU but with complete control of the software (can wipe them and install something different) and want to accelerate pytorch-lightning model training. What could be a possible solution to integrate them with as little additional code possible?

I've researched this topic a bit, finding possible options, cannot determine which one to choose:

# option info  pro  con 
1. dask-pytorch-ddp Package to be used to writing models with easier integration into Dask  will likely work cannot use existing model out of the box, need rewriting the model itself
2. PL docs, on-prem cluster (intermediate) multiple copies of pytorch lighning on the network  simples way according to lightning docs fiddly to launch according to the docs
3. PL docs, SLURM cluster wipe/redeploy cluster, setup SLURM  less fiddly to launch individual jobs need to redeploy the cluster OS/software
4. Pytorch + dask officially supported and documented use of Skorch  has a package handling this - skorch will need to use pytorch, not lightning

Are there any more options or tutorials to learn about this?

Aniruddh Parihar
  • 3,072
  • 3
  • 21
  • 39
Kirill Setdekov
  • 329
  • 1
  • 11

1 Answers1

2

I'd recommend looking into Horovod for this. Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. You can use the Horovod integration with PyTorch Lightning to distribute the training of your model. This approach will require installing Horovod on your servers and minimal changes to your existing PyTorch Lightning code.

As an alternative, you can also consider using Ray for distributed training with PyTorch Lightning.

Grimlock
  • 1,033
  • 1
  • 9
  • 23
  • Thanks a lot for many suggestions. Those are the ones I have not found at first, but looks promising. Will update my comments based on my results – Kirill Setdekov Apr 25 '23 at 05:22