0

Im using dask-yarn version 0.3.1. Following the basic example on https://dask-yarn.readthedocs.io/en/latest/. 

from dask_yarn import YarnCluster
from dask.distributed import Client

# Create a cluster where each worker has two cores and eight GB of memory
cluster = YarnCluster(environment='environment.tar.gz',
                  worker_vcores=2,
                  worker_memory="8GB")

The application is successfuly submitted to cluster but control does not return to console after YarnCluster constructor. The following is the final output from starting.

18/09/19 16:14:24 INFO skein.Daemon: Submitting application...
18/09/19 16:14:24 INFO impl.YarnClientImpl: Submitted application application_1534573350864_34823
18/09/19 16:14:27 INFO skein.Daemon: Notifying that application_1534573350864_34823 has started. 1 callbacks registered.
18/09/19 16:14:27 INFO skein.Daemon: Removing callbacks for application_1534573350864_34823

One thing I noticed when I was initially testing from within docker container was an exception related to grpc not parsing http_proxy environment variable. When running from dedicated cluster edge node, I don't see this exception but also don't see control returned after Constructor.

user1738628
  • 51
  • 1
  • This appears to be a bug. Would you mind filing an issue on github: https://github.com/dask/dask-yarn/issues – jiminy_crist Sep 20 '18 at 13:30
  • Thanks, will do. – user1738628 Sep 20 '18 at 16:15
  • I believe this issue was fixed in dask-yarn version 0.3.2. Some clusters had default network interfaces that were incompatible with gRPC (e.g. infiniband), which would result in this behavior. We now determine hostnames in a more robust manner. Hopefully this fixes your issue. – jiminy_crist Oct 30 '18 at 10:09

0 Answers0