-1

I am newbie in dask & considering using it for parallelization for ml model tuning purposes. Should i try dask-yarn or dask-kubernetes for such requirement?

Any general ideas on where to use which of these will also be helpful for broader understanding.

Thanks

mdurant
  • 27,272
  • 5
  • 45
  • 74
Himanshu Gautam
  • 359
  • 1
  • 4
  • 17

1 Answers1

0

Most people choose the appropriate deployment solution to match the infrastructure that they already have. This is particularly true for YARN/hadoop - you would probably only make such a cluster from scratch if you know you need it.

Surveys indicate that kubernetes is the most commonly used orchestrator for dask (not counting local, bare/ssh or HPC), and it seems to be the most common way to make clusters these days, in general. Note that you can use dask-kubernetes itself, or dask-gateway (which can spawn on multiple backends).

I do not vouch for EKS being a particularly good kubernetes implementation.

mdurant
  • 27,272
  • 5
  • 45
  • 74