PyQ - q.upsert and q.set append to splayed kdb database

Question

When I run the following code in pyq...

from bittrex.bittrex import Bittrex, API_V2_0
import time, json
from pyq import q, K

get_bittrex = Bittrex(None, None)
starttime = time.time()

market_result = get_bittrex.get_market_summaries()['result']

while True:
    for res in market_result:
        market_name = res['MarketName']
        ask = str(res['Ask'])
        bid = str(res['Bid'])
        last = str(res['Last'])
        timeStamp = str(res['TimeStamp'])
        if market_name in ['USDT-BTC', 'USDT-ETH', 'USDT-LTC']:
            mkt = {"Name": market_name,"Bid": bid,"Ask": ask,"Time":timeStamp}
            q.set(':alpha', [mkt])
            q.upsert(':alpha', mkt)
            q.get(':alpha').show()

    time.sleep(10.0)

it returns every 10 seconds to the console...

Name     Bid            Ask            Time
--------------------------------------------------------------
USDT-BTC 15475.00000001 15530.99999999 2017-12-09T02:46:52.547
USDT-BTC 15475.00000001 15530.99999999 2017-12-09T02:46:52.547
Name     Bid          Ask      Time
------------------------------------------------------
USDT-ETH 454.00000001 454.9999 2017-12-09T02:46:52.017
USDT-ETH 454.00000001 454.9999 2017-12-09T02:46:52.017
Name     Bid          Ask          Time
----------------------------------------------------------
USDT-LTC 133.76999998 133.98999999 2017-12-09T02:46:49.703
USDT-LTC 133.76999998 133.98999999 2017-12-09T02:46:49.703
Name     Bid            Ask            Time
--------------------------------------------------------------
USDT-BTC 15475.00000001 15530.99999999 2017-12-09T02:46:52.547
USDT-BTC 15475.00000001 15530.99999999 2017-12-09T02:46:52.547
Name     Bid          Ask      Time
------------------------------------------------------
USDT-ETH 454.00000001 454.9999 2017-12-09T02:46:52.017
USDT-ETH 454.00000001 454.9999 2017-12-09T02:46:52.017
Name     Bid          Ask          Time
----------------------------------------------------------
USDT-LTC 133.76999998 133.98999999 2017-12-09T02:46:49.703
USDT-LTC 133.76999998 133.98999999 2017-12-09T02:46:49.703

Why does it print each line 2x? Also, how can I modify this so each market_name is added to the same table, i.e., it's currently printing each market_name to it's own table instead of appending to the existing table.

Ideal output would like like the following...

Name     Bid            Ask            Time
--------------------------------------------------------------
USDT-BTC 15475.00000001 15530.99999999 2017-12-09T02:46:52.547
USDT-ETH 454.00000001 454.9999 2017-12-09T02:46:52.017
USDT-LTC 133.76999998 133.98999999 2017-12-09T02:46:49.703

score 1 · Accepted Answer · answered Dec 09 '17 at 03:59

What you do in the loop is equivalent to the following q code:

q)`:alpha set enlist`a`b!1 2
q)`:alpha upsert`a`b!1 2
q)show get`:alpha
a b
---
1 2
1 2

There is nothing surprising here. The set command saves a 1-row table (recall that in q a table is a list of dictionaries) in the file alpha, then the upsert command appends a row which is the same as the one saved and finally get reads the resulting 2-row table back.

What you probably want is to initialize the table outside of the loop, run a series of upserts in the loop and show the result after the loop.

Note that you can initialize the table like this:

>>> q.set(':alpha', q('!', ["Name","Bid","Ask","Time"], ()).flip)
k('`:alpha')
>>> q.get(':alpha').show()
Name Bid Ask Time
-----------------

Finally, if your goal is to save the feed, it is rarely a good idea to write each message to the disk as soon as it arrives. Most feed handlers collect a day worth of market data in memory and save it in HDB at the end of the day.

PyQ - q.upsert and q.set append to splayed kdb database

1 Answers1