I have 2 functions:
- A method that maintains a streaming RPC connection (with a third party service) that constantly receives data in a loop.
- A method that periodically checks for the data received and performs some computation.
e.g.,
def streaming_loop(global_shared_data, data_lock: asyncio.Lock):
# A generator for input.
input_generator = input.generator()
# stream_data is a third party API that takes in a generator and maintains a
# RPC connection
responses_generator = client.stream_data(input_generator)
for response in responses_generator:
# Write data in a thread safe manner
await write_data(response, global_shared_data, data_lock)
async def process_loop(global_shared_data, data_lock: asyncio.Lock):
while True:
# To schedule other tasks in between.
asyncio.sleep(0)
async with self._lock:
# process data in a thread safe manner
await process_data(global_shared_data, data_lock)
I want to run streaming_loop in a ThreadPoolExecutor since I cannot make it a co-routine due to the third party API client.stream_data
consumes input as a generator.
I also want to run process_loop
as an asyncio task. However, when I tried
executor = ThreadPoolExecutor(max_workers=2)
loop = asyncio.get_running_loop()
loop.run_in_executor(executor, streaming_loop(global_shared_data, lock))
I'm getting the error:
TypeError: 'coroutine' object is not callable
Is there no way to run a co-routine in a threadpoolexecutor? If that's not possible, how would I make this thread safe when the 2 methods are modifying the same global object?