0

I have 2 functions:

  1. A method that maintains a streaming RPC connection (with a third party service) that constantly receives data in a loop.
  2. A method that periodically checks for the data received and performs some computation.

e.g.,

def streaming_loop(global_shared_data, data_lock: asyncio.Lock):
  # A generator for input.
  input_generator = input.generator()
  # stream_data is a third party API that takes in a generator and maintains a 
  # RPC connection 
  responses_generator = client.stream_data(input_generator)
  for response in responses_generator:
     # Write data in a thread safe manner
     await write_data(response, global_shared_data, data_lock)

async def process_loop(global_shared_data, data_lock: asyncio.Lock):
   while True:
      # To schedule other tasks in between.
      asyncio.sleep(0)
      async with self._lock:
         # process data in a thread safe manner
         await process_data(global_shared_data, data_lock)

I want to run streaming_loop in a ThreadPoolExecutor since I cannot make it a co-routine due to the third party API client.stream_data consumes input as a generator. I also want to run process_loop as an asyncio task. However, when I tried

executor = ThreadPoolExecutor(max_workers=2)
loop = asyncio.get_running_loop()
loop.run_in_executor(executor, streaming_loop(global_shared_data, lock))

I'm getting the error:

TypeError: 'coroutine' object is not callable

Is there no way to run a co-routine in a threadpoolexecutor? If that's not possible, how would I make this thread safe when the 2 methods are modifying the same global object?

Discombobulous
  • 1,112
  • 2
  • 14
  • 25
  • Look here how to send the data from the thread based code to the async based code: https://stackoverflow.com/q/32889527/5378816 – VPfB Sep 15 '22 at 05:53

0 Answers0