1

Streaming execute, -11!, does not work on named pipes, so an obvious solution of redirecting the gzip -cd output to a named pipe and passing it to -11! does not work.

Thomas Smyth - Treliant
  • 4,993
  • 6
  • 25
  • 36
Alexander Belopolsky
  • 2,228
  • 10
  • 26

2 Answers2

3

-11! accepts a compressed file and streams it so long as it was compressed with -19! (using 2 as the compression algorithm parameter, which is gzip).

The only difference between a normal gzipped file and a kdb compressed one is a few bytes at the beginning of the file.

EDIT (see comment) Thanks, this isn't true - the bytes are different at the end of the file

So a possible solution is to prepend your gzipped files (if they weren't produced by -19!) with the appropriate byte array first.

Manish Patel
  • 4,411
  • 4
  • 25
  • 48
  • The wiki article you referenced contains the following advise: "Do not use streaming compression with log files, as in the event of a crash, the log file will be unusable as it will be missing meta information from the end of the file." Does this refer to gzip meta data or kx meta data? If the latter, this seems to invalidate your answer. – Alexander Belopolsky Oct 03 '15 at 15:09
  • Presumably the advice is against *writing* to a compressed file, which q will do if the file is already compressed. The question was about reading, so I should think the advice doesn't apply. – Manish Patel Oct 04 '15 at 16:44
  • I mostly mentioned the wiki admonition to point out that the metadata is written at the end rather than the beginning of the compressed log. I was not sure whether the article referred to the regular gzip metadata or something kx specific. Regular gzip format contains a CRC and the uncompressed length at the end of the file, but kx compressed files have something entirely different there. This means that it is not true that _"The only difference between a normal gzipped file and a kdb compressed one is a few bytes at the beginning of the file."_ – Alexander Belopolsky Oct 04 '15 at 18:13
  • 2
    BTW, the "magic" header of kx compressed files is 8 bytes containing ASCII characters "kxzipped", but the difference in format with gzip is more than that. – Alexander Belopolsky Oct 04 '15 at 18:14
0

For anyone using kdb v3.4+ streaming execution for named pipes was introduced as the function .Q.fps.

Here is a simple example of .Q.fps in action, first create the pipe via command line:

echo "aa" > test.pipe

Next in a q session:

q).Q.fps[0N!]`:test.pipe
,"aa"

Where 0N! is a function used to display the contents of the file.

Thomas Smyth - Treliant
  • 4,993
  • 6
  • 25
  • 36