Running long blocking calculations in parallel in twisted

Question

I am trying to learn twisted framework. But, I am not able to get a handle of it.

Say, I have this function.

def long_blocking_call(arg1, arg2):
     # do something
     time.sleep(5) # simulate blocking call
     return result

results = []
for k, v in args.iteritems():
      r = long_blocking_call(k,v)
      results.append(r)

But, I was wondering how can I leverage deferToThread (or something else in twisted world) to run the long_blocking_call in "parallel"

I found this example: Periodically call deferToThread But, I am not exactly sure if that is running things in parallel?

use from multiprocessing import Process p1 = Process(target=long_blocking_call, args, name='k') p1.start() p2 = Process(target=long_blocking_call, args, name='k') p2.start() p1.join() p2.join() this is not a correct code, just google it further, and don't use args as it is a keyword — Madhur Yadav, Jun 21 '19 at 04:02
The multiprocessing module has nothing to do with Twisted. If the question is about how to uses multiple processes to achieve parallel computation, it is a possible answer. If the question is about how to do parallel computation with Twisted, it is not. — Jean-Paul Calderone, Jun 21 '19 at 10:35

Jean-Paul Calderone · Accepted Answer · 2019-06-23T13:35:30.820

2

deferToThread uses Python's built-in threading support to run the function passed to it in a separate thread (from a thread pool).

So deferToThread has all of the same properties as the built-in threading module when it comes to parallelism. On CPython, threads can run in parallel as long as only one of them is holding the Global Interpreter Lock.

Since there is no universal cause of "blocking" there is also no universal solution to "blocking" - so there's no way to say whether deferToThread will result in parallel execution or not in general. However, a general rule of thumb is that if the blocking comes from I/O it probably will and if it comes from computation it probably won't.

Of course, if it comes from I/O, you might be better off using some other feature from Twisted instead of multithreading.

edited Jun 23 '19 at 13:35

answered Jun 21 '19 at 10:38

Jean-Paul Calderone

47,755
6
94
122

Thanks for the answer.. So.. what's a good way to parallelize this `long_blocking_call` function above in twisted world? – frazman Jun 21 '19 at 18:28
_This_ one? It parallelizes fine because time.sleep releases the GIL. Or you could use reactor.callLater instead of time.sleep. But are you really interested in parallelizing mostly-sleeping-based-on-system-clock functions? – Jean-Paul Calderone Jun 23 '19 at 00:56
Ah.. no.. basically.. think of sleep as another function call... that does something which takes couple of seconds to process.. What I am trying to acheive is that there is a for loop.. and then.. each member of for loop calls another function which can be processed independently other members in for loop.. Essentially.. it would have been like `pool.map(long_blocking_call, for_loop_list)` But, I understand that twisted and multiprocessing library dont play well.. I am guessing the deferToThread is able to run things independently.. but I am not too sure.. Can you please guide/advise? – frazman Jun 23 '19 at 03:27
Please re-read the part of my answer about "no universal cause of blocking". And I'll edit some more words in to make that part more explicit, too. – Jean-Paul Calderone Jun 23 '19 at 13:34
I guess.. what I am trying to ask is.. is there an equivalent of asyncio `create_task` in twisted? https://asyncio.readthedocs.io/en/latest/hello_world.html – frazman Jun 23 '19 at 19:48
Considering what you wrote in the original question and your last question here, I suspect there may be a larger misunderstanding that still needs to be corrected. `create_task` is a _less_ powerful solution to removing blocking than `deferToThread`. As for whether or not there's an equivalent ... it's messy. Maybe? `create_task` puts a coroutine into the event loop. The closest parallel in Twisted is probably inlineCallbacks. But it's not an exact parallel. – Jean-Paul Calderone Jun 23 '19 at 22:44
I see. I do think that i have some void in understanding twisted.. and that is what I am trying to understand. Thanks for all the help though. So, basically, I have a function.. and a for loop... Now, the function is CPU bound (in terms of blocking).. But this function can be parallelized over the for loop. How do i do this in twisted? Would very much appreciate like a dumb example.. – frazman Jun 23 '19 at 22:54

Running long blocking calculations in parallel in twisted

1 Answers1