Why do I get a value error when using multiprocessing on websocket streams validated with pydantic?

Question

Why am I getting a missing value error when I try to run 2 websockets simultaneously using ProcessPoolExecutor. It's an issue I've been banging my head against for a week. I figured someone smarter than I may have an answer or insight readily available.

Courtesy of Alpaca's API I am trying to stream market data and trade updates simultaneously using the alpaca-py sdk. Both streams run on asyncio event loops via a websocket. The code to initialize each stream is similar. Here is what it looks like for live market data.

from alpaca.data.live.stock import StockDataStream

ticker = "SPY"

stock_data_stream = StockDataStream('API_KEY', 'API_SECRET')

# Data handler where each update will arrive
async def stock_data_handler(data_stream):
    # Do something with data_stream here
    pass

stock_data_stream.subscribe_bars(stock_data_handler, "SPY") # Subscribe to data type

stock_data_stream.run() # Starts asyncio event loop

The above stream runs in an infinite loop. I could not figure out a way to run another stream in parallel to receive live trade updates. I found that the market stream blocks the trade updates stream and vice versa.

I experimented a lot trying to run each stream on its own new event loop. I also tried to run/schedule each streaming data coroutine in its own thread using to_thread() and run_coroutine_threadsafe() to no success. This is when I decided to turn to a more parallel approach.

So, I wrapped each stream implementation in its own function and then created a concurrent future process for each function using the ProcessPoolExecutor. A trading update should fire every time an order is canceled, so I cooked up place_trade_loop() to provide an infinite loop and ran that as a process as well.

def place_trade_loop(order_request):
    while True:
        order_response = trading_client.submit_order(order_request)
        time.sleep(2)
        cancel_orders_response = trading_client.cancel_order_by_id(order_id = order_response.id)

def crypto_stream_wrapper():
    async def stock_data_handler(data_stream):
        print(data_stream)
    stock_data_stream.subscribe_quotes(stock_data_handler, ticker)
    stock_data_stream.run()

def trading_stream_wrapper():
    async def trading_stream_handler(trade_stream):
        print(trade_stream)
    trading_stream.subscribe_trade_updates(trading_stream_handler)
    trading_stream.run()

if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor() as executor:
        f1 = executor.submit(stock_stream_wrapper)
        f2 = executor.submit(trading_stream_wrapper)
        f3 = executor.submit(place_trade_loop)

The place trade loop and the market data stream play perfectly well together. However, the following error results when an order is canceled. Again, a canceled order should result in the trading_stream_handler receiving a trade_stream.

error during websocket communication: 1 validation error for TradeUpdate
execution_id
  field required (type=value_error.missing)
Traceback (most recent call last):
  File "C:\Users\zachm\AppData\Local\Programs\Python\Python310\lib\site-packages\alpaca\trading\stream.py", line 172, in _run_forever
    await self._consume()
  File "C:\Users\zachm\AppData\Local\Programs\Python\Python310\lib\site-packages\alpaca\trading\stream.py", line 145, in _consume
    await self._dispatch(msg)
  File "C:\Users\zachm\AppData\Local\Programs\Python\Python310\lib\site-packages\alpaca\trading\stream.py", line 89, in _dispatch
    await self._trade_updates_handler(self._cast(msg))
  File "C:\Users\zachm\AppData\Local\Programs\Python\Python310\lib\site-packages\alpaca\trading\stream.py", line 103, in _cast
    result = TradeUpdate(**msg.get("data"))
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for TradeUpdate
execution_id
  field required (type=value_error.missing)

For reference:

execution_id is part of the TradeUpdate model as shown in link #5. I believe the error is arising when **msg.get("data") is passed to TradeUpdate (link #4). There could also be something going on with the Alpaca platform because I noticed many of my canceled orders are listed as 'pending_cancel,' which seemed to be an issue others are dealing with. Alpaca gave a response near the end of that thread.

Finally, the following is from the ProcessPoolExecutor documentation, which may have something to do with the error?

The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.

I am sorry for such a long post. Thank you in advance for any help or encouragement you can provide!

Why do I get a value error when using multiprocessing on websocket streams validated with pydantic?

0 Answers0