I want to stream my data(which I get from the third party APIs) into the KDB+ database. For this, I installed the following versions of:
- KDB+ : KDB+ 4.0 2021.07.12 Copyright (C) 1993-2021 Kx Systems
- qpython3 1.0.1 : https://pypi.org/project/qPython3/
I tried running the script given in the qpython documentation: https://qpython.readthedocs.io/en/latest/usage-examples.html#data-publisher
But this gives me: qpython.qreader.QReaderException: Attempt to read data out of buffer bounds
I have also tried running more basic codes given in the documentation like these:
>>> with qconnection.QConnection(host = 'localhost', port = 5001, pandas = True) as q:
ds = q('(1i;0Ni;3i)', pandas=True)
print(ds)
But this also results in the same issue. This is the stack trace of the above code
File "/usr/lib/python3.8/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 2, in <module>
File "/home/ubox122/Practice/lib/python3.8/site-packages/qpython/qconnection.py", line 386, in __call__
return self.sendSync(parameters[0], *parameters[1:], **options)
File "/home/ubox122/lib/python3.8/site-packages/qpython/qconnection.py", line 304, in sendSync
response = self.receive(data_only = False, **options)
File "/home/ubox122/Practice/lib/python3.8/site-packages/qpython/qconnection.py", line 381, in receive
result = self._reader.read(**self._options.union_dict(**options))
File "/home/ubox122/lib/python3.8/site-packages/qpython/qreader.py", line 136, in read
message = self.read_header(source)
File "/home/ubox122/Practice/lib/python3.8/site-packages/qpython/qreader.py", line 167, in read_header
message_size = self._buffer.get_uint()
File "/home/ubox122/Practice/lib/python3.8/site-packages/qpython/qreader.py", line 510, in get_uint
return self.get('I')
File "/home/ubox122/Practice/lib/python3.8/site-packages/qpython/qreader.py", line 482, in get
return struct.unpack(fmt, self.raw(offset))[0]
File "/home/ubox122/Practice/lib/python3.8/site-packages/qpython/qreader.py", line 463, in raw
raise QReaderException('Attempt to read data out of buffer bounds')
qpython.qreader.QReaderException: Attempt to read data out of buffer bounds```
I also searched for duplicates for this issue and found one:https://github.com/exxeleron/qPython/issues/40
This somewhere looks like my issue, but the changes suggested seems to be incorporated already.
I basically want to keep inserting the data into the KDB+ tables with the help of qpython.
I have also explored PyQ library for the same purpose, but I could just see the examples on the documentation as well as on stack overflow in which all the operations are performed in pyq shell, an approach which I would not be proceeding.
Here is the link of the PyQ library: https://pythonhosted.org/pyq/whatsnew/4.0.html
So, I have majorly two questions:
1. Can please anyone suggest what might be the issue with the qpython code?
2. Are there any code examples to store the data into KDB+ with the help of pyq by writing python script (instead of opening the pyq terminal and manually performing the operation) just like we do in qpython?
Any help would be appreciated.