2

I've installed Hadoop via a helm chart on my microk8s kubernetes cluster. I would like to know how to create a dask cluster on my different machines on this hadoop cluster. I tried following the the tutorials on the Dask websites, but I keep getting errors because it is looking for the local yarn/hadoop. How do I point to the hadoop on kubernetes so I can create the cluster?

Akshat Zala
  • 710
  • 1
  • 8
  • 23
Vinh Tran
  • 169
  • 1
  • 12
  • please add links to the tutorials you've tried as well as error you've got. Have you been checking the https://docs.dask.org/en/latest/remote-data-services.html ? – Nick Jun 16 '20 at 18:44
  • If you have k8s what not launch on k8s instead of hadoop ? – quasiben Jun 17 '20 at 20:09

1 Answers1

0

If you want to launch Dask on Yarn we recommend using https://yarn.dask.org

However, if you are using Kubernetes already you might consider https://kubernetes.dask.org, which is more commonly used today.

MRocklin
  • 55,641
  • 23
  • 163
  • 235