4

I hope I'm doing something wrong, but it seems like kdb can't read data from named pipes (at least on Solaris). It blocks until they're written to but then returns none of the data that was written.

I can create a text file:

$ echo Mary had a little lamb > lamb.txt

and kdb will happily read it:

    q) read0 `:/tmp/lamb.txt
enlist "Mary had a little lamb"

I can create a named pipe:

$ mkfifo lamb.pipe

and trying to read from it:

    q) read0 `:/tmp/lamb.pipe

will cause kdb to block. Writing to the pipe:

$ cat lamb.txt > lamb.pipe

will cause kdb to return the empty list:

()

Can kdb read from named pipes? Should I just give up? I don't think it's a permissions thing (I tried setting -m 777 on my mkfifo command but that made no difference).

Martin McNulty
  • 2,601
  • 3
  • 22
  • 26

4 Answers4

3

With release kdb+ v3.4 Q has support for named pipes: Depending on whether you want to implement a streaming algorithm or just read from the pipe use either .Q.fps or read1 on a fifo pipe:

To implement streaming you can do something like:

q).Q.fps[0N!]`:lamb.pipe

Then $ cat lamb.txt > lamb.pipe

will print

,"Mary had a little lamb"

in your q session. More meaningful algorithms can be implemented by replacing 0N! with an appropriate function.

To read the context of your file into a variable do:

q)h:hopen`:fifo://lamb.pipe
q)myText: `char$read1(h)
q)myText

"Mary had a little lamb\n"

See more about named pipes here.

Thomas Smyth - Treliant
  • 4,993
  • 6
  • 25
  • 36
derfugu
  • 31
  • 5
1

when read0 fails, you can frequently fake it with system"cat ...". (i found this originally when trying to read stuff from /proc that also doesn't cooperate with read0.)

q)system"cat /tmp/lamb.pipe"
<blocks until you cat into the pipe in the other window>
"Mary had a little lamb"
q)

just be aware there's a reasonably high overhead (as such things go in q) for invoking system—it spawns a whole shell process just to run whatever your command is

you might also be able to do it directly with a custom C extension, probably calling read(2) directly…

Aaron Davies
  • 1,190
  • 1
  • 11
  • 17
0

Streaming from pipe is supported from v3.4

Details steps:

  1. Check duplicated pipe file

    rm -f /path/dataPipeFileName

  2. Create named pipe

    mkfifo /path/dataPipeFileName

  3. Feed data

    q).util.system[$1]; $1=command to fetch data > /path/dataPipeFileName &

  4. Connect pipe using kdb .Q.fps

    q).Q.fps[0N!]`$":/path/",dataPipeFileName;

Reference: .Q.fps (streaming algorithm) Syntax: .Q.fps[x;y] Where x is a unary function and y is a filepath .Q.fs for pipes. (Since V3.4) Reads conveniently sized lumps of complete "\n" delimited records from a pipe and applies a function to each record. This enables you to implement a streaming algorithm to convert a large CSV file into an on-disk kdb+ database without holding the data in memory all at once.

Kevin
  • 1,335
  • 14
  • 14
0

The algorithm for read0 is not available to see what it is doing under the hood but, as far as I can tell, it expects a finite stream and not a continuous one; so it will will block until it receives an EOF signal.

algolicious
  • 1,182
  • 2
  • 10
  • 14
  • I'm happy for it to block - that seems like the correct behaviour for a continuous stream - but it then returns the empty list, rather than the data that was written. It's almost like it's waiting for EOF and then throwing away whatever it read before that point :s – Martin McNulty Feb 10 '12 at 10:45
  • Yeah it might well be timing out. The underlying algorithm would provide the answer so it might be worth emailing tech@kx.com to get clarity on the situation. – algolicious Feb 10 '12 at 11:59