Solve python ValueError: max_workers must be <= 61 when running pre-commit?

Question

I am using Django to develop an ERP and I want to use pre-commit with my project.

I have installed pre-commit, black, flake8, flake8-black. and this is my

.pre-commit-config.yaml

file configurations content

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
-   id: check-yaml
-   id: end-of-file-fixer
-   id: trailing-whitespace
-   repo: https://github.com/psf/black
rev: 19.3b0
hooks:
-   id: black
    language_version: python3.7

When I commit my code it gives me this error

Check 
Yaml...............................................................Passed
Fix End of Files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing vms/movement/test/test_model.py
Fixing vms/payment/test/test_forms.py

Trim Trailing Whitespace.................................................Passed
black....................................................................Failed
- hook id: black
- exit code: 1

Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\Scripts\black.EXE\__main__.py", line 7, in <module>
  File "c:\users\diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\lib\site-packages\black.py", line 3754, in patched_main
    main()
  File "c:\users\diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\lib\site-packages\click\core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\lib\site-packages\click\core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "c:\users\diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\lib\site-packages\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\diaa\.cache\pre-commit\repo4t4vqwkg\py_env-python3\lib\site-packages\black.py", line 435, in main
    executor = ProcessPoolExecutor(max_workers=os.cpu_count())
  File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 523, in __init__
    f"max_workers must be <= {_MAX_WINDOWS_WORKERS}")
ValueError: max_workers must be <= 61

This is my pip list

-------------------------------- --------
appdirs                          1.4.4
asgiref                          3.3.4
aspy.refactor-imports            2.2.0
atomicwrites                     1.4.0
attrs                            21.2.0
Babel                            2.9.1
black                            21.6b0
cached-property                  1.5.2
cfgv                             3.3.0
click                            8.0.1
colorama                         0.4.4
coverage                         5.5
distlib                          0.3.2
Django                           3.2
django-autocomplete-light        3.8.2
django-autoslug                  1.9.8
django-colorfield                0.4.1
django-countries                 7.2.1
django-crispy-forms              1.11.0
django-datatables-view           1.19.1
django-excel                     0.0.10
django-guardian                  2.4.0
django-ipware                    3.0.2
django-js-asset                  1.2.2
django-login-required-middleware 0.5.0
django-mptt                      0.12.0
django-nested-inline             0.4.3
django-querysetsequence          0.14
django-rest-authtoken            2.1.3
django-timezone-field            4.1.1
djangorestframework              3.12.2
docopt                           0.6.2
enum34                           1.1.10
Faker                            8.8.0
filelock                         3.0.12
flake8                           3.9.2
flake8-black                     0.2.1
fpdf2                            2.4.1
identify                         2.2.10
importlib-metadata               4.5.0
iniconfig                        1.1.1
IP2Location                      8.5.1
isort                            5.9.1
lml                              0.0.9
mccabe                           0.6.1
mixer                            7.1.2
mypy-extensions                  0.4.3
nodeenv                          1.6.0
num2words                        0.5.10
packaging                        20.9
pathspec                         0.8.1
Pillow                           8.1.0
pip                              21.1.1
pluggy                           0.13.1
pre-commit                       2.13.0
psycopg2                         2.8.6
py                               1.10.0
pycodestyle                      2.7.0
pycountry                        20.7.3
pydocstyle                       6.1.1
pyexcel                          0.6.2
pyexcel-io                       0.6.4
pyexcel-webio                    0.1.2
pyflakes                         2.3.1
pynput                           1.7.3
pyparsing                        2.4.7
pytest                           6.2.4
pytest-django                    4.4.0
python-dateutil                  2.8.1
pytz                             2021.1
PyYAML                           5.4.1
regex                            2021.4.4
reportbro-fpdf                   1.7.10
reportbro-lib                    1.6.0
reportlab                        3.5.67
seed-isort-config                2.2.0
setuptools                       57.0.0
setuptools-scm                   3.4.3
simpleeval                       0.9.10
six                              1.15.0
snowballstemmer                  2.1.0
sqlparse                         0.4.1
text-unidecode                   1.3
texttable                        0.8.2
toml                             0.10.2
tox                              3.23.1
typed-ast                        1.4.3
typing                           3.7.4.3
typing-extensions                3.10.0.0
virtualenv                       20.4.7
XlsxWriter                       1.4.3
xlwt                             1.3.0
zipp                             3.4.1

When I run os.cpu_count() it gives me 64 processors. I using windows server 2016 Datacenter.

Please help me solving this issue.

Note: I am using python 3.7

Simon Bowly · Accepted Answer · 2021-06-27T07:04:34.337

2

This is a known issue with cpython on windows. The error occurs when black tries to run multiple workers on >60 core machines because the default number of process workers given by os.cpu_count() breaks some other windows limit (number of waiting processes? I'm not quite sure). Black >=19.10b0 has a fix for this, so try updating the version of black in your pre-commit config if you can?

Python bug report: https://bugs.python.org/issue26903
Fix applied by this PR in black: https://github.com/psf/black/pull/838

edited Jun 27 '21 at 07:04

answered Jun 27 '21 at 06:55

Simon Bowly

1,003
5
10

2

you can bump to the latest by running `pre-commit autoupdate` – anthony sottile Jun 28 '21 at 02:39
That works for me I have updated black to 21.6b0 and every thing just works as well. – Dhia Shalabi Jun 28 '21 at 11:30

score 0 · Answer 2 · answered Jun 27 '21 at 07:12

You may try to patch this line site-packages\black.py, line 435

    executor = ProcessPoolExecutor(max_workers=os.cpu_count())

to

   executor = ProcessPoolExecutor(max_workers=min(32, os.cpu_count() + 4))

which is what ProcessPoolExecutor in Python 3.8 and later essentially do I believe.

Solve python ValueError: max_workers must be <= 61 when running pre-commit?

2 Answers2