Node.js how to synchronously read lines from stream.Readable

Question

I'm interacting with a child process through stdio, and I need to wait for a line from childProcess.stdout each time I write some command to childProcess.stdin.
It's easy to wrap an asynchronous method for writing like below:

async function write(data){
    return new Promise(resolve=>{
        childProcess.stdin.write(data,()=>resolve());
    })
}

However, it turns out quite difficult when it comes to reading, since data from stdout must be processed using listeners. I've tried below:

const LineReader = require("readline")
const reader = LineReader.createInterface(childProcess.stdout);
async function read(){
    return new Promise(resolve=>{
        reader.once("line",line=>resolve(line));
    })
}

But it always returns the first line.
I know I may achieve this using setInterval, And I've already implemented the functionality this way. But it obviously has an impact on the performance, so now I'm trying to optimize it by wrapping it into an asynchronous method.
Any suggestions and solutions will be appreciated!

Can you back up a few steps and describe the overall problem you're trying to solve in much greater detail? You can't read data synchronously from a stream. They don't work that way. So, your two attempts at doing that are just going down the wrong path. So, you need to back up a few steps and outline the overall problem (without assuming some solution) and then we can hopefully point you down a useful path (that's different than what you're trying now). — jfriend00, Mar 13 '21 at 07:30
@jfriend00 So you mean that it's impossible to wrap line reading into an asynchronous method in principle? — 0x269, Mar 13 '21 at 07:35
That depends upon what you mean by that? The line reader object you're using is event driven. It fires events whenever it has a new line. It doesn't stop and wait for you. It's NOT demand driven (in the sense of you ask it for the next line and it holds that next line until you ask again). So, you have to use that interface in an event driven way. As I said above, this would be a whole lot more useful if you described the TOP level problem so we can point you down a good solution path rather than debating the details of the solution path you're on that doesn't seem correct to me. — jfriend00, Mar 13 '21 at 07:38
@jfriend00 Well thanks for your reply first. The reason I didn't mention the top problem is that it doesn't seem to matter. Since you insisted, I'll describe it briefly. I'm interacting with a python script that receives commands from stdin and output results to stdout. And my js script needs to wait for the results after it writes the command to stdin of the child python process. Hopefully, I'm describing the problem clear. — 0x269, Mar 13 '21 at 07:47
Wait for the result from stdout to do what next? And, are you wanting to wait for just the next line of output? Or all output? This sounds like a problem that could be solved with a little state machine, but I'd need to understand more about the sequence of operations. — jfriend00, Mar 13 '21 at 07:49
Well `.once()` for the `line` event has no chance of working because `line` events will still be fired, even if there aren't any listeners so you can easily miss data. — jfriend00, Mar 13 '21 at 07:51
@jfriend00 Just the next line. The later operation depends on the data from the python script. — 0x269, Mar 13 '21 at 07:52
So, you're trying to create a function that sends a command on stdin and returns the next line from stdout? — jfriend00, Mar 13 '21 at 07:54
OK, let me chew on that for a bit and think about how that could be done. — jfriend00, Mar 13 '21 at 07:55

jfriend00 · Accepted Answer · 2021-03-13T09:21:03.530

Well, I ended up with something pretty similar to what you were trying. It makes some assumptions that are mentioned in the code and needs more complete error handling:

const cp = require('child_process');
const readline = require('readline');

const child = cp.spawn("node", ["./echo.js"]);
child.on('error', err => {
    console.log(err);
}).on('exit', () => {
    console.log("child exited");
});

const reader = readline.createInterface({ input: child.stdout });

// this will miss line events that occurred before this is called
// so this only really works if you know the output comes one line at a time
function nextLine() {
    return new Promise(resolve => {
        reader.once('line', resolve);
    });
}

// this does not check for stdin that is full and wants us to wait
// for a drain event
function write(str) {
    return new Promise(resolve => {
        let ready = child.stdin.write(str, resolve);
        if (!ready) {
            console.log("stream isn't ready yet");
        }
    });
}

async function sendCmd(cmd) {
    // get line reader event handler installed so there's no race condition
    // on missing the return event
    let p = nextLine();
    // send the command
    await write(cmd);
    return p;
}

// send a sequence of commands and get their results
async function run() {
    let result1 = await sendCmd("hi\n");
    console.log(`Got '${result1}'`);
    let result2 = await sendCmd("goodbye\n");
    console.log(`Got '${result2}'`);
    let result3 = await sendCmd("exit\n");
    console.log(`Got '${result3}'`);
}

run().then(() => {
    console.log("done");
}).catch(err => {
    console.log(err);
});

And, for testing purposes, I ran it with this echo app:

process.stdin.on("data", data => {
    let str = data.toString();
    let ready = process.stdout.write("return: " + str, () => {
        if (str.startsWith("exit")) {
            process.exit();
        }
    });
    if (!ready) {
        console.log("echo wasn't ready");
    }
});

Yeah, exactly. Your solution was similar to my trial in the description, which turned out effective. The problem wasn't with such an approach. Since now I cannot reproduce the problem, I'm not able to figure out where was wrong initially. Anyway, thanks for your generous help and effective solution! — 0x269, Mar 24 '21 at 17:24

Node.js how to synchronously read lines from stream.Readable

1 Answers1