Difficulty reading input pipe in SBCL

Question

I am slowly getting closer to be able to read and write to/from named pipes of a background process through SBCL. What I do is kick off the program I am trying to read/write from/to:

todd@ubuntu:~/CoreNLP$ cat ./spin | /usr/bin/java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -outputFormat text  > ./spout &

[1] 24616

So that all works out fine, so I kick off SBCL and do this:

(defparameter from-corenlp (open "./spout"))

Which also works out fine, but declaring the stream causes SBCL to spill the stream onto the screen (which is all the startup information from the background process). It does not wait until I read from the stream. Is that how things are supposed to work?

*"Is that how things are supposed to work?"* No. A simple test with "cat /dev/random > fifo" (fifo made with mkfifo) and opening the fifo from sbcl does not allow me to replicate the issue. Is there anything else you could tell us about this? — coredump, Sep 17 '16 at 16:51
Okay, I figured that it was not supposed to work that way. I did a test doing cat from a file and the output ended up dependably at the lisp pipe. Declaring an output stream from lisp into the pipe though: (defparameter to-corenlp (open "./spout" :direction :output :if-exists :append)) did not work. No data arrived at the receiving end in lisp, though I could continue to cat an input file to the spout pipe and data would arrive in the lisp stream. — Todd Pierce, Sep 19 '16 at 13:20
Interestingly, if I run the java program sending the output to the pipe, the program still does not send its data to the pipe, however ls > spout does send the data to the pipe and can be read dependably in lisp. For the record, I have tried using sb-ext:run-program and that does not work. Any java program with a quoted argument (like the asterisk above) crashes as soon as it is run. This does not happen with inferior shell but I have no idea how to get a functioning stream from lisp to the spawned program at the same time as reading from the spawned program. — Todd Pierce, Sep 19 '16 at 13:21
Just in case you were wondering, I did try redirecting the output of the java program to a file and it still sent all of its output to the screen. The file ended up being zero bytes. Heh... this is such a weird one. — Todd Pierce, Sep 19 '16 at 13:31
Ah... the java program sends its startup always to the screen, however, the output after that can be redirected to a file. Unfortunately, that is not the case with the pipe. Actual output disappears entirely when read by lisp, except (just my luck) the program shut down messages. Very weird. — Todd Pierce, Sep 19 '16 at 15:04
Here is an example of what I am doing, which works fine: http://pastebin.com/raw/6mn1VrhN. Could you provide more details in the step you are doing? It is quite difficult to understand and debug something without having the precise steps you are doing (maybe the problem does not come from Lisp only, maybe there are other issues). — coredump, Sep 19 '16 at 15:18

score 3 · Answer 1 · edited Sep 20 '16 at 08:28

The solution, as I posted it to the stanford parser mailing list (stack overflow reformatted a lot of it to something weird, but you get the idea):

It took quite a while, but I finally figured out embedding (for the most part) the CoreNLP program (while in interactive mode) in SBCL Lisp.

First of all, forget using (sb-ext:run-program ...). This combination of spawning Java with a quoted argument (like the asterisk) no matter how well escaped, simply makes the spawned program crash.

Inferior shell seems to kick off the parser but it is only good for a one-off parse, even in the interactive mode. Perhaps I could have done better, but inferior shell needs to be installed and it is poorly documented.

The initial attempted solution of using Unix named pipes ends up being the final one, but it took a bit of work, first with buffering, then with the order of operations, and finally understanding some nuances about the parser program.

First, turning off buffering completely when running the program is important, so running it looks like this:

stdbuf --i=0 --o=0 --e=0 cat ./spin | /usr/bin/java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -outputFormat text  > ./spout &

That is supposed to be running the parser in the background accepting input from spin and sending its output to spout. But if you look at the process table in Linux, you will not see it running. It is still waiting for something to pull from the output pipe before it can even run.

So, we run SBCL and start a stream pulling from the parser´s pipe:

(defparameter *from-corenlp* (open "./spout"))

NOW the parser starts running. Here, oddly, it also starts dumping output to the screen, not to the pipe! That is because all of this banner stuff when the parser starts and stops (and apparently even the NLP> prompt) is sent to stderr, not stdout. This is actually a good thing.

So then we declare the stream from Lisp to the parser:

(defparameter *to-corenlp* (open "./spin" :direction :output :if-exists :append))

Then we send some text for the parser to parse:

(write-line  "This is the first test." *to-corenlp*)

I ran into a problem here a few times, even. Remember that Lisp has its own buffer so you have to clear out the stream every time:

(finish-output *to-corenlp*)

You then can run this line below a whole bunch of times to verify you obtain the exact same behavior you would have gotten from an interactive session of the parser:

(format t "~a~%" (read-line *from-corenlp*))

Which, if you are a good boy scout, should not only be true, but you can carry on with your interactive slave parser session for as long as you like:

(write-line  "This is the second test." *to-corenlp*)
(finish-output *to-corenlp*)

Isn´t that great? And notice I pulled all of that off being terrible at Unix, terrible at Lisp and being a terrible boy scout!

Now so can you!

Difficulty reading input pipe in SBCL

1 Answers1