0

So I encountered this problem in my code:

worker_threads = 950
threads = []
for thr in range(worker_threads):
    thread = threading.Thread(target=worker_function, args=(in_queue, out_queue, status_control))
    thread.daemon = True
    threads.append(thread)
    thread.start()

for thread in threads:
        thread.join()

Threads are using requests module and do simple web crawling. For some reason, when threads count above 650 I've got message:

Traceback (most recent call last):
  File "C:/Users/bb/Desktop/wp/Новая папка/upload.py", line 132, in <module>
    thread.start()
  File "C:\Users\bb\AppData\Local\Programs\Python\Python35-32\lib\threading.py", line 844, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

I'am using VDS windows 2008 x64 server with 4GB of RAM. How can I solve that issue? I didn't find any usefull solutions for that kind of problem. Thank you.

UPD. Here's the example of worker function:

def worker_function(in_queue, out_queue, status_control):
    while not in_queue.empty():
        try:
            url = in_queue.get_nowait()
        except queue.Empty:
            break
        try:
            print('Checking '+url)

        except Exception as e:
            print(str(e))

Now python just crashes.

Alex Petrov
  • 75
  • 10

0 Answers0