1

I got orderbook history data of BTCUSDT from the binance api. According to binance api on 'how to manage local orderbook', first get and buffer data using websocket and second get orderbook data using the api and use the lastOrderId to get rid of outdated data that i buffered. however the data i got was 2 csv files. depth_snap and depth_update. so I tried to do what the api told me to do. the first part was already done because it said to get buffer data using websocket which was stored in depth_update. and by using the lastOrderId from the depth_snap I tried to do the second part, remove the outdated data, only to realize that the lastOrderId cannot be used.

I checked the lastOrderId and found that lastOrderId does not overlap in depth_snap and depth_update. so i thought i should use depth_snap data instead of depth_update. however the time gap between data was about 40 minutes which was too long.

I did check timestamp to make sure that the data is in the same date

how can i use this depth_snap and depth_update to create orderbook data? i checked the lastOrderId and pu(lastOrderId of the previous data) between timestamp and csv files(depth_snap.csv on different date) and found that they were in order. will it be okay to just use depth_snap and make orderbook data because the data is continuous?

KyuSang Jang
  • 21
  • 7

1 Answers1

0

You should add code to make clear what you have done.

From my own experience, here is how you should use binance websocket:

client = Client('PUBLICKEY')

data = {}

def spread(msg):
    latence = msg['data']['T']-time.time()*1000
    msg['lat'] = latence

    if np.abs(latence) < 500:
        msg['latence'] = False
    else:
        msg['latence'] = True

    with open('data/w'+msg['data']['s']+".dat", 'wb') as out:
        pickle.dump(msg, out)

    os.rename('data/w'+msg['data']['s']+".dat",'data/'+msg['data']['s']+".dat")
    print ("WEBSOCKET %s" % (round(latence)))

bm = BinanceSocketManager(client)
conn_key = bm.start_all_ticker_futures_socket(spread)
bm.start()

In words, you should not be doing any processing in your callback function.

You should write data anywhere you like and process somewhere else from it.

I know those guidelines may seem counter-intuitive, but if not following this I personally get a latency that builds up to dozens of seconds and it stays stuck in the past with no recovery.

You should also compute the latency to flag when to suspend operations.

If you monitor Binance API latency over time, you will observe huge latency peaks some times, especially when you have big moves in the market.

Synthase
  • 5,849
  • 2
  • 12
  • 34
  • but i'm not using binance websocket. i downloaded the depth_snap and depth_update using the binance api. what i was asking was how to convert the data that i downloaded into orderbook data – KyuSang Jang Jul 10 '21 at 08:46
  • You are using the websocket. depthUpdate = Partial Book Depth Streams. This is websocket. – Synthase Jul 10 '21 at 08:49
  • no i am not. i am using 'POST /sapi/v1/futuresHistDataId' to get historical orderbook data which is different from websockets because it comes in compressed data. it is open to only those who requested binance for permission – KyuSang Jang Jul 10 '21 at 08:58
  • Indeed, that is a recent feature, so you're right. In this situation, I would simply recommend to switch and use the websocket. According to binance docs, this API endpoint is intended to "backtesting and optimization of strategies, research and analysis". I am unsure this is intended for live trading. – Synthase Jul 10 '21 at 09:35
  • yeah for research purposes. i really need historical data. will it be okay to ignore the 2nd part of managing data and continue on? – KyuSang Jang Jul 10 '21 at 10:03
  • If you are doing research, you should not mind about running 40 minutes behind. Just set your reference accordingly and that should be fine. Otherwise, use the websocket. In both case you get historical data, we're just talking about latency here. – Synthase Jul 10 '21 at 13:36