2

Polars 'POLARS_MAX_THREADS' doesn't actually work, although it does create POLARS_MAX_THREADS number of threads at beginning, but after i do some calculation, the number of threads it uses boost a lot.

import os
os.environ['POLARS_MAX_THREADS'] = '4'

from time import sleep
import psutil
import polars as pl

print(pl.threadpool_size())

df = pl.DataFrame({'a': [1,2,3],
                   'b': [4,5,6]})
df = df + 1

print(pl.threadpool_size())
print('Num threads before: {}'.format(psutil.Process().num_threads()))
sleep(1000)

The code get

4
4
Num threads before: 149

The threads it use actually equals to the number of cores on the machine.

吴慈霆
  • 523
  • 2
  • 15
Rolnan
  • 23
  • 5
  • If I run a processor intensive polars operation with htop open, the number of cores that go to 100% is limited by whatever I set with `os.environ['POLARS_MAX_THREADS']` so it's definitely working. I don't know enough about how the inner workings of python, psutil, and polars to say why `psutil.Process().num_threads()` reporting a different number than `pl.threadpool_size()` doesn't support the idea that the setting doesn't work though. – Dean MacGregor Sep 01 '23 at 14:20
  • 1
    This has just been fixed for anyone else reading: https://github.com/pola-rs/polars/issues/10847 – jqurious Sep 01 '23 at 16:45

0 Answers0