0

I am developing a multithreads app using Python3. I want to run over 100k threads at a time for backend server, but threads are created 10k in max. I think my server pc has enough resources for lots of threads.(cpu, ram). How can I increase the number of threads in Python3 multithreading?

for i in range(0, 1000000):
    t = i.Thread(target=func, args=(i))
    t_list.append(t)

for t in t_list:
    t.start()
py10522
  • 1
  • is there a setting on your server pc that has a thread limit? – rhavelka Jun 17 '21 at 14:50
  • @rhavelka, of course. echo 1000000 > /proc/sys/kernel/threads-max, ulimit -s 256, ulimit -i 1000000. – py10522 Jun 17 '21 at 14:54
  • 2
    Out of curiosity, what does each of those threads represent? Often, it's wise to de-couple the idea of a thread (i.e., a thing that does work) from a task (a unit of work that needs to be done.) You might end up writing a little more code, but depending on the size of your task state objects, you might end up saving a _huge_ amount of memory. – Solomon Slow Jun 17 '21 at 14:59

1 Answers1

1

I'm going to argue that "you shouldn't do that". It would be rare that you need that many threads, or that the system can handle it well. Threads are not a magic solution to deal with lots of incoming requests to process.

Instead, I suggest you create a limited number of threads in a "pool" and use a queue or similar mechanism to send tasks to the pool. See this answer for details on how to implement something like this.

Wes Hardaker
  • 21,735
  • 2
  • 38
  • 69