0

I would like to use a Dask delayed task to call an external program, which outputs it's progress to STDOUT. In the delayed, I plan to monitor the STDOUT and would like to update the Client process that is waiting for the delayed task with progress information extracted from the STDOUT. Is there a recommended way for a delayed task to communicate with its Client processes, or do I need to roll my own?

ipetrik
  • 1,749
  • 18
  • 28

1 Answers1

1

You could achieve this kind of flow with any of the coordination primitives or actors provided by dask. From your description, the Queue or pubsub mechanisms seem like they might be favourite. You should note that all of these are generally means for low-frequency and low-volume communications.

mdurant
  • 27,272
  • 5
  • 45
  • 74
  • Duh... I can't believe I missed that. :-P BTW, I'm wondering why you say "_all of these are generally means for low-frequency and low-volume communications_"? Accoring to the docs: "_Pubs and Subs use the central scheduler to find each other, but not to mediate the communication. This means that there is very little additional latency or overhead, and they are appropriate for very frequent data transfers._" *Edit* NVM, I read on, "_When using a Pub or Sub from a Client all communications will be routed through the scheduler._" – ipetrik Nov 20 '19 at 19:47