Switch off dask client warnings

Question

Dask client spams warnings in my Jupyter Notebook output. Is there a way to switch off dask warnings?

Warning text look like this: "distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 3.16 GB -- Worker memory limit: 4.20 GB"

The problem appear after these code:

import pandas as pd
from sqlalchemy import create_engine, MetaData
from sqlalchemy import select, insert, func
import dask.dataframe as dd

from dask.distributed import Client
client = Client(n_workers=4, threads_per_worker=4, processes=False)

engine = create_engine(uri)
meta_core = MetaData()
meta_core.reflect(bind=engine)

table = meta_core.tables['table']

dd_main = dd.read_sql_table(
    table=table,
    uri=uri,
    index_col='id'
)

dd_main.head()

After executing the chunk above, I get a lot of these warnings in every Jupyter cell, so I can't even find my actual output.

score 7 · Answer 1 · edited Aug 28 '21 at 00:21

7

I had to use a variation on MRocklin's answer:

import logging

client = Client(..., silence_logs=logging.ERROR)

The string "error" is not recognized by python's logging, which uses constants. An alternative is to pass the numerical value of the constant directly instead (here logging.ERROR == 50). See logging levels.

edited Aug 28 '21 at 00:21

Aule Mahal

688
5
10

answered Jan 24 '21 at 17:46

ian-whitestone

126
1
3

While this code may solve the question, [including an explanation](//meta.stackexchange.com/q/114762) of how and why this solves the problem would really help to improve the quality of your post, and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please [edit] your answer to add explanations and give an indication of what limitations and assumptions apply. Please especially explain why the change to the other answer is needed, you seem to have a special reason. – Yunnosch Jan 26 '21 at 07:20
Does this solution filter logs from the workers as well or just the client itself? – ionox0 Aug 08 '22 at 18:52

score 6 · Accepted Answer · answered Aug 12 '19 at 12:13

6

You can pass a logging level to the Client constructor like the following:

client = Client(..., silence_logs='error')

answered Aug 12 '19 at 12:13

MRocklin

55,641
23
163
235

This didn't work for me. I got an error from `logging` saying that the level was not recognized. ian's solution below did work, or passing 50 (the numerical value of logging.ERROR) directly. – Aule Mahal Aug 26 '21 at 18:41

Switch off dask client warnings

2 Answers2