Kdb+ data fomat when writing to a file

Question

I'm trying to understand what happen when someone try to write to a file the next way:

q)h:hopen `:out
q)h (1 2;3)
3i
q)hclose h
q)read1 `:out
0x07000200000001000000000000000200000000000000f90300000000000000

this is not the same as a binary representation:

q)-8!(1 2;3)
0x010000002d00000000000200000007000200000001000000000000000200000000000000f90300000000000000

in which data format (1 2;3) was written into :out?
is there a way to read data from it back? - like -9!-8!(1 2;3)
how set/get related to these binary data formats? - do they use another 3-rd different binary data format?

rianoc · Answer 1 · 2022-06-03T13:53:30.013

-8! returns the IPC byte representation.

On disk the binary format is different. To read this data use get.

To stream data to a log file you need to create the empty file first https://code.kx.com/q/kb/replay-log/#replaying-log-files

q)`:out set () /This important part adds a header to the file to set the type
`:out
q)h:hopen `:out
q)h (1 2;3)
600i
q)hclose h
q)get `:out
1 2
3

Note if you wish to write your item as a single element to the list then use enlist

q)`:out set ()
`:out
q)h:hopen `:out
q)h enlist (1 2;3)
616i
q)hclose h
q)get `:out
1 2 3

Binary and text data can also be written to a file which is what you are doing.

https://code.kx.com/q/ref/hopen/#files

The intention is to write specific pieces of data

q)h 0x2324 /Write some binary
q)h "some text\n" /Write some text

In your code the binary representation of the raw KX object does get written but no header is added (it is neither IPC nor disk format). So when you read back the data it cannot be interpreted correctly by either -9! or get.

Valid file binary created with `:out set () has a file header: (Can be read by get)

q)read1 `:out
0xff0100000000000000000200000007000200000001000000000000000200000000000000f90300000000000000

Valid IPC binary with IPC header: (Can be read by -9!)

q)-8!(1 2;3) 
0x010000002d00000000000200000007000200000001000000000000000200000000000000f90300000000000000

Your raw object in binary - no header present to enable interpretation

q)read1 `:out
0x07000200000001000000000000000200000000000000f90300000000000000

score 2 · Accepted Answer · answered Jun 03 '22 at 15:20

Technically you can read the object if you have a little prior knowledge about the object, such that you can fabricate a header:

q)read1`:out
0x07000200000001000000000000000200000000000000f90300000000000000
q)-9!read1`:out
'badmsg
  [0]  -9!read1`:out
         ^
q)-9!{0x01,0x000000,(reverse 0x0 vs `int$count[x]+1+3+4+1+1+4),0x00,0x00,(reverse 0x0 vs 2i),x}read1`:out
1 2
3

The header here is comprised of:

0x01 - little endian
0x000000 - filler
message length (count of raw `x` plus the header additions)
0x00 - type (generic list = type 0) ... you have to know this in advance
0x00 - attributes .... none here, you would have to know this
length of list .... here we knew it was a 2-item list
x  - the raw bytecode of the object without header

As rianoc pointed out, there are better ways to write such objects such that they can be more easily read without requiring advanced knowledge

Can we create the same raw object representation without saving it into to a file? — egor7, Jun 03 '22 at 21:02
You can drop the header bytes from the front of -8, aka `14_-8!(1 2;3)` however the amount you chop off might vary.....you would need to study this carefully: https://code.kx.com/q/kb/serialization/ — terrylynch, Jun 06 '22 at 13:19

Kdb+ data fomat when writing to a file

2 Answers2