3

Is there a way to limit the number of threads used by polars?

I am doing this because I am doing a second layer of parallelization around some polars code, and would like to limit the inner parallelism. This should still be better than Pandas due to the SIMD right.

bumpbump
  • 542
  • 4
  • 17

1 Answers1

6

You can set the env var POLARS_MAX_THREADS. This value initializes the thead pool size. This must be set before polars is imported.

More configuration env vars are documented here.

ritchie46
  • 10,405
  • 1
  • 24
  • 43
  • Is it possible to do it programatically? – Guillem Jan 03 '23 at 15:33
  • No, the threadpool is created on startup and cannot be modified later. – ritchie46 Jan 03 '23 at 16:38
  • Of course, it is possible. ``` import os os.environ["POLARS_MAX_THREADS"] = "1" # it is important that the import happens AFTER the line where we set POLARS_MAX_THREADS import polars as pl ``` – roskoN Jan 16 '23 at 14:37