I am using the following libraries:
from binance.websockets import BinanceSocketManager
from twisted.internet import reactor
To create a websocket to fetch data from an API (Binance API) and print the price of bitcoin in 1s intervals:
conn_key = bsm.start_symbol_ticker_socket('BTCUSDT', asset_price_stream)
Every time there is a new update, the function asset_price_stream is executed, with the new data as parameter. This works (I can, for example, simply print the data to the console in asset_price_stream).
Now, I want to share this data with an asyncio function. At the moment, I am using a janus queue for this.
In asset_price_stream:
price_update_queue_object.queue.sync_q.put_nowait({msg['s']: msg['a']})
and in the asynchronous thread:
async def price_updater(update_queue):
while True:
priceupdate = await update_queue.async_q.get()
print(priceupdate)
update_queue.async_q.task_done()
I am using the sync_q interface in asset_price_stream and the async_q interface in the asyncio function. If I use the async_q interface also in asset_price_stream, I get an error:
Unhandled Error
Traceback (most recent call last):
File "/home/elias/.local/lib/python3.7/site-packages/twisted/python/log.py", line 101, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/home/elias/.local/lib/python3.7/site-packages/twisted/python/log.py", line 85, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/home/elias/.local/lib/python3.7/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/home/elias/.local/lib/python3.7/site-packages/twisted/python/context.py", line 83, in callWithContext
return func(*args, **kw)
--- <exception caught here> ---
File "/home/elias/.local/lib/python3.7/site-packages/twisted/internet/posixbase.py", line 687, in _doReadOrWrite
why = selectable.doRead()
File "/home/elias/.local/lib/python3.7/site-packages/twisted/internet/tcp.py", line 246, in doRead
return self._dataReceived(data)
File "/home/elias/.local/lib/python3.7/site-packages/twisted/internet/tcp.py", line 251, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/home/elias/.local/lib/python3.7/site-packages/twisted/protocols/tls.py", line 324, in dataReceived
self._flushReceiveBIO()
File "/home/elias/.local/lib/python3.7/site-packages/twisted/protocols/tls.py", line 290, in _flushReceiveBIO
ProtocolWrapper.dataReceived(self, bytes)
File "/home/elias/.local/lib/python3.7/site-packages/twisted/protocols/policies.py", line 107, in dataReceived
self.wrappedProtocol.dataReceived(data)
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/twisted/websocket.py", line 290, in dataReceived
self._dataReceived(data)
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/websocket/protocol.py", line 1207, in _dataReceived
self.consumeData()
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/websocket/protocol.py", line 1219, in consumeData
while self.processData() and self.state != WebSocketProtocol.STATE_CLOSED:
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/websocket/protocol.py", line 1579, in processData
fr = self.onFrameEnd()
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/websocket/protocol.py", line 1704, in onFrameEnd
self._onMessageEnd()
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/twisted/websocket.py", line 318, in _onMessageEnd
self.onMessageEnd()
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/websocket/protocol.py", line 628, in onMessageEnd
self._onMessage(payload, self.message_is_binary)
File "/home/elias/.local/lib/python3.7/site-packages/autobahn/twisted/websocket.py", line 321, in _onMessage
self.onMessage(payload, isBinary)
File "/home/elias/.local/lib/python3.7/site-packages/binance/websockets.py", line 31, in onMessage
self.factory.callback(payload_obj)
File "dollarbot.py", line 425, in asset_price_stream
price_update_queue_object.queue.async_q.put_nowait({msg['s']: msg['a']})
File "/home/elias/.local/lib/python3.7/site-packages/janus/__init__.py", line 438, in put_nowait
self._parent._notify_async_not_empty(threadsafe=False)
File "/home/elias/.local/lib/python3.7/site-packages/janus/__init__.py", line 158, in _notify_async_not_empty
self._call_soon(task_maker)
File "/home/elias/.local/lib/python3.7/site-packages/janus/__init__.py", line 60, in checked_call_soon
self._loop.call_soon(callback, *args)
File "/usr/lib/python3.7/asyncio/base_events.py", line 690, in call_soon
self._check_thread()
File "/usr/lib/python3.7/asyncio/base_events.py", line 728, in _check_thread
"Non-thread-safe operation invoked on an event loop other "
builtins.RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one
When using sync_q, it works so far. The async thread can print the price updates. But sometimes, it simply gets stuck and I don't know why. The API is still delivering data (as I have double checked) but it stops to arrive at the async thread through the queue. I have no idea why this happens, and it is not always reproducible (I can run the same code without modifications 5 times in a row, four times it works, and once it doesn't). The interesting thing is: as soon as i press CTRL-C to abort the program, the data immediately arrives at the queue when it didn't before! (In the few seconds while the program is waiting to shut all asyncio threads down) So I have the feeling that something with the sync-async-queue communication is wrong/concurrent with something else that is aborted when I press CTRL-C.
So my question is: How can I improve that procedure? Is there a better way to send data to an asyncio thread from a twisted.internet websocket than janus.Queue() ? I tried some different things (accessing a global object for example, but I cannot access an asyncio.lock() from the asset_price_stream as it is not an async function... so it would not be thread safe).
