dask distributed 1.19 client logging?

Question

The following code used to emit logs at some point, but no longer seems to do so. Shouldn't configuration of the logging mechanism in each worker permit logs to appear on stdout? If not, what am I overlooking?

import logging
from distributed import Client, LocalCluster
import numpy as np

def func(args):
    i, x = args
    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s %(name)s %(levelname)s %(message)s')
    logger = logging.getLogger('func %i' % i)
    logger.info('computing svd')
    return np.linalg.svd(x)

if __name__ == '__main__':
    lc = LocalCluster(10)
    c = Client(lc)
    data = [np.random.rand(50, 50) for i in range(50)]
    fut = c.map(func, zip(range(len(data)), data))
    results = c.gather(fut)
    lc.close()

As per this question, I tried putting the logger configuration code into a separate function invoked via c.run(init_logging) right after instantiation of the client, but that didn't make any difference either.

I'm using distributed 1.19.3 with Python 3.6.3 on Linux. I have

logging:
    distributed: info
    distributed.client: info

in ~/.dask/config.yaml.

score 0 · Answer 1 · answered Nov 17 '17 at 18:28

Evidently the submitted functions do not actually execute until one tries to retrieve the results from the generated futures, i.e., using the line

print(list(results))

before shutting down the local cluster. I'm not sure how to reconcile this with the section in the online docs that seems to state that direct submissions to a cluster are executed immediately.

dask distributed 1.19 client logging?

1 Answers1