0

I need to call two different methods simultaneously in order to save time and using results obtained from those methods i need to call another methods, So i came across deferToThread method in twisted.internet.threads which return a deferred object so that i can attach callback methods.

Here is my code

for key,val in result.items():
    d=threads.deferToThread(myMethod1,key,val)
    d1=threads.deferToThread(myMethod2,key)
    d.addCallback(myMethod3)
    d1.addCallback(myMethod4)

reactor.run()

In the given code, result is a dictionary. myMethod1 and myMethod2, these are the two methods which i need to run asynchronously.

As per my understanding, deferToThread will create a separate thread to execute given method for each item in a dictionary, So every time the loop iterates, It will create two different threads. That's fine. But i need to run this code after fix interval continuously. I tried to run it after fix interval but after executing for 16 to 17 times, every thing stops working. does it mean that lots of threads are running in background?? and if lots of thread are running then how to stop or how to kill those threads?

Yudi
  • 831
  • 4
  • 10
  • 19

1 Answers1

0

deferToThread does not create a new thread. It takes a thread from the reactor's thread pool. There is an upper bound on the size of that thread pool. Once all of the threads are occupied with one of your methods the next deferToThread call won't be able to start the method you pass it until one of the threads from the pool finishes and becomes available.

The way to free up a thread from the pool is to have the method it is running complete (either raise an exception or return). There is no other way to free up the thread (they are normal Python threads; Python threads do not support unilateral termination from the outside).

You may want to reconsider you use of threads at all. There are a lot of APIs in Twisted to help you perform tasks asynchronously without using threads. You haven't explained what you're doing in myMethod1 and myMethod2 so I can't suggest any particular APIs, though.

Jean-Paul Calderone
  • 47,755
  • 6
  • 94
  • 122
  • I am using pySphere library. result dictionary contains list of ESXi host where key=host name and val=host IP myMethod1 : this method is for getting more information about host myMethod2 : this method is for getting list of VMs running on particular host myMethod3 and myMethod4 are used to print or to insert into Database myMethod1 and myMethod2 are time consuming if executed sequentially – Yudi Mar 06 '14 at 06:07
  • pysphere appears to be a library for accessing an HTTP API. Twisted Web includes a pretty good natively asynchronous HTTP client. It's not likely you can use this together with pysphere, though, so most likely you would have to deal with a lower level interface to the vSphere Web Services SDK (at least until you built a pysphere-alike using Twisted Web's client). – Jean-Paul Calderone Mar 06 '14 at 12:13