So I encountered this problem in my code:
worker_threads = 950
threads = []
for thr in range(worker_threads):
thread = threading.Thread(target=worker_function, args=(in_queue, out_queue, status_control))
thread.daemon = True
threads.append(thread)
thread.start()
for thread in threads:
thread.join()
Threads are using requests module and do simple web crawling. For some reason, when threads count above 650 I've got message:
Traceback (most recent call last):
File "C:/Users/bb/Desktop/wp/Новая папка/upload.py", line 132, in <module>
thread.start()
File "C:\Users\bb\AppData\Local\Programs\Python\Python35-32\lib\threading.py", line 844, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
I'am using VDS windows 2008 x64 server with 4GB of RAM. How can I solve that issue? I didn't find any usefull solutions for that kind of problem. Thank you.
UPD. Here's the example of worker function:
def worker_function(in_queue, out_queue, status_control):
while not in_queue.empty():
try:
url = in_queue.get_nowait()
except queue.Empty:
break
try:
print('Checking '+url)
except Exception as e:
print(str(e))
Now python just crashes.