1

I'm using a kalman filter in pykalman, then I pickle the filter and save it into Sybase as a binary array that is sufficiently long. I'm using pyodbc as the connection.

I run the script on a Linux server, then get the same filter from my Windows desktop and unpickle it, it works fine. However, if I get the same filter in Linux and try to unpickle it, it says data is truncated. In ipython I can see that it's only getting the first 255 bytes. 

In [16]: x = cur.execute('select top 1 filter from kalman_filters where qdate = "20180115"').fetchone()

In [17]: x[0]
Out[17]: b'\x80\x03cpykalman.standard\nKalmanFilter\nq\x00)\x81q\x01}q\x02(X\x13\x00\x00\x00transition_matricesq\x03cnumpy.core.multiarray\n_reconstruct\nq\x04cnumpy\nndarray\nq\x05K\x00\x85q\x06C\x01bq\x07\x87q\x08Rq\t(K\x01K\x01K\x01\x86q\ncnumpy\ndtype\nq\x0bX\x02\x00\x00\x00f8q\x0cK\x00K\x01\x87q\rRq\x0e(K\x03X\x01\x00\x00\x00<q\x0fNNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00tq\x10b\x89C\x08\x00\x00\x00\x00\x00\x00\xf0?q\x11tq\x12bX\x14\x00\x00\x00observation_matric'

In [18]: len(x[0])
Out[18]: 255

In [19]: type(x[0])
Out[19]: bytes

If I do the same in Windows, it gets it correctly.

In [7]: len(x[1])
Out[7]: 827

In [8]: x[1]
Out[8]: b'\x80\x03cpykalman.standard\nKalmanFilter\nq\x00)\x81q\x01}q\x02(X\x13\x00\x00\x00transition_matricesq\x03cnumpy.core.multiarray\n_reconstruct\nq\x04cnumpy\nndarray\nq\x05K\x00\x85q\x06C\x01bq\x07\x87q\x08Rq\t(K\x01K\x01K\x01\x86q\ncnumpy\ndtype\nq\x0bX\x02\x00\x00\x00f8q\x0cK\x00K\x01\x87q\rRq\x0e(K\x03X\x01\x00\x00\x00<q\x0fNNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00tq\x10b\x89C\x08\x00\x00\x00\x00\x00\x00\xf0?q\x11tq\x12bX\x14\x00\x00\x00observation_matricesq\x13h\x04h\x05K\x00\x85q\x14h\x07\x87q\x15Rq\x16(K\x01K\x01K\x01\x86q\x17h\x0e\x89C\x08\x00\x00\x00\x00\x00\x00\xf0?q\x18tq\x19bX\x15\x00\x00\x00transition_covarianceq\x1ah\x04h\x05K\x00\x85q\x1bh\x07\x87q\x1cRq\x1d(K\x01K\x01K\x01\x86q\x1eh\x0e\x89C\x08\x99\xa2#\x03Y\xa0%?q\x1ftq bX\x16\x00\x00\x00observation_covarianceq!h\x04h\x05K\x00\x85q"h\x07\x87q#Rq$(K\x01K\x01K\x01\x86q%h\x0e\x89C\x08\xa4\xd4\x1fF\xd09D?q&tq\'bX\x12\x00\x00\x00transition_offsetsq(h\x04h\x05K\x00\x85q)h\x07\x87q*Rq+(K\x01K\x01\x85q,h\x0e\x89C\x08\x00\x00\x00\x00\x00\x00\x00\x00q-tq.bX\x13\x00\x00\x00observation_offsetsq/h\x04h\x05K\x00\x85q0h\x07\x87q1Rq2(K\x01K\x01\x85q3h\x0e\x89C\x08\x00\x00\x00\x00\x00\x00\x00\x00q4tq5bX\x12\x00\x00\x00initial_state_meanq6h\x04h\x05K\x00\x85q7h\x07\x87q8Rq9(K\x01K\x01\x85q:h\x0e\x89C\x08\x18WQ\x07\x1bK\'\xbfq;tq<bX\x18\x00\x00\x00initial_state_covarianceq=h\x04h\x05K\x00\x85q>h\x07\x87q?Rq@(K\x01K\x01K\x01\x86qAh\x0e\x89C\x08\x1e"\x12\xd5\xa6\xbc\x00?qBtqCbX\x0c\x00\x00\x00random_stateqDNX\x07\x00\x00\x00em_varsqE]qF(h\x1ah!h6h=eX\x0b\x00\x00\x00n_dim_stateqGK\x01X\t\x00\x00\x00n_dim_obsqHK\x01ub.'

In [9]: type(x[1])
Out[9]: bytes

So I'm pretty sure it's a pyodbc/odbc settings issue. Any idea what this is?

Thanks

swmfg
  • 1,279
  • 1
  • 10
  • 18
  • 1
    I'm guessing you're using FreeTDS on Linux, can you include your FreeTDS and unixODBC settings? – FlipperPA Jan 17 '18 at 00:16
  • Also, your examples don't seem to show any sign of unpickling, so it it really just a matter of the byte stream being truncated, or does unpickling really have something to do with it? – Gord Thompson Jan 17 '18 at 00:22
  • Possibly related question [here](https://stackoverflow.com/q/13381854/2144390). TDS protocol version 4.2 may be the culprit. – Gord Thompson Jan 17 '18 at 00:45
  • Thanks for the comments guys. Yes I'm using FreeTDS, version 4.2. I've also found a similar comment to yours [here](https://lists.ibiblio.org/pipermail/freetds/2006q3/020418.html). I tried changing TDS version to 7.0 in both /etc/odbc.ini and /etc/freetds.conf but neither worked. I'm going give that a go again on the next server restart and will report back. – swmfg Jan 17 '18 at 05:06
  • Also, I skipped the unpickling bit because seems like the byte array is truncated. So I'd imagine once I fix the truncation issue it should unpickle fine. – swmfg Jan 17 '18 at 05:07

0 Answers0