Right way to set memory parameters for LocalCluster in dask

Question

I tried the code below,

from dask.distributed import Client, LocalCluster
worker_kwargs = {
    'memory_limit': '2G',
    'memory_target_fraction': 0.6,
    'memory_spill_fraction': 0.7,
    'memory_pause_fraction': 0.8,
    'memory_terminate_fraction': 0.95,
}
cluster = LocalCluster(ip='0.0.0.0', n_workers=8, **worker_kwargs)

but it doesn't work

...
TypeError: __init__() got an unexpected keyword argument 'memory_target_fraction'

What's the right way to call? LocalCluster doesn't seem to care about ~/.config/dask/distributed.yaml, either.

I've read about these posts:

score 1 · Accepted Answer · answered Apr 21 '19 at 16:20

1

Seems it's a version issue. Updating dask to 1.2.0 solves the problem.

from dask.distributed import Client, LocalCluster
worker_kwargs = {
    'memory_limit': '2G',
    'memory_target_fraction': 0.6,
    'memory_spill_fraction': 0.7,
    'memory_pause_fraction': 0.8,
    # 'memory_terminate_fraction': 0.95,
}
cluster = LocalCluster(ip='0.0.0.0', n_workers=8, **worker_kwargs)

works

answered Apr 21 '19 at 16:20

zyxue

7,904
5
48
74

But it still doesn't seem to be working, I asked here, https://github.com/dask/distributed/issues/2456#issuecomment-485265025 – zyxue Apr 21 '19 at 21:10
1

It's deprecated now, use dask.config.set( { "distributed.worker.memory.target": 0.6, "distributed.worker.memory.spill": 0.7, "distributed.worker.memory.pause": 038, "distributed.worker.memory.terminate": 0.95, } ) – Anatoly Alekseev Mar 21 '22 at 13:10

Right way to set memory parameters for LocalCluster in dask

1 Answers1