How do I return a reference to a Readable stream before creating it?

Question

I have a function that needs to return a Readable stream synchronously. I have a function that creates a Readable stream with the data I require, but it's an asynchronous method. How can I return a reference to the synchronous stream?

class SomeStreamCreator {
    _requestStream() {
        fetchStream()
            .then(stream => /* stream is a Readable stream with data */)

        return /* somehow need to return the Readable stream here */
    }
}

Well, you can't synchronously return something that you obtained asynchronously. Can't be done in Javascript. So, perhaps you should back up a few steps and describe the overall problem you're trying to solve and then maybe someone can help you write proper asynchronous code to use the asynchronously created stream. — jfriend00, Aug 07 '19 at 18:09
This is in relation to implemening a RandomAccessReader within the YAUZL library - https://www.npmjs.com/package/yauzl#class-randomaccessreader. It requires the ability to create a read stream (`_readStreamForRange`) synchronously. I'm using `node-fetch` in order to get that stream from a remote location. `fs.createReadStream` is synchronous, but the data it receives is asynchronous from the file system, so I assume this must be possible? I'm just not sure how to do it outside of the stdlib. — Alexander Craggs, Aug 08 '19 at 10:09
Streams don't have to provide data synchronously, so I think it's still possible. I create a 'reference stream' which I return immediately from the function, and then pipe to that reference stream from within the `node-fetch` body. The issue is I have no idea how to do this and Googling hasn't been much help. — Alexander Craggs, Aug 08 '19 at 10:10
Yeah, it's likely possible. `fs.createReadStream()` on a file launches an asynchronous file open and then immediately returns and sets its internal state that it's waiting for the file to be open. Sometime later when the file open finishes, it checks to see if there's a `data` event handler assigned and, if so, starts flowing the stream. I presume that is possible from a remote source that like `fs.open()` is async, but I honestly don't know exactly how to do it. Perhaps look into how one builds their own readStream with a custom data source since that's essentially what you have. — jfriend00, Aug 08 '19 at 10:14

score 0 · Answer 1 · answered Aug 08 '19 at 12:14

Well it all depends on if you actually need the same exact stream object or just the same data. In the first case as mentioned in comments it's more or less not possible.

If it's only the data you're after the situation is quite simple if you use a simple utility stream called PassThrough available in the built-in stream module:

import {PassThrough} from "stream";

class SomeStreamCreator {
    _requestStream() {
        // first prepare an empty stream
        const out = new PassThrough();

        fetchStream() 
            // when the stream is ready pipe it to the empty one
            .then(stream => stream.pipe(out))
            // it's wise to add error handling
            .catch(error => stream.emit("error", e))

        // but immediately simply return the output
        return out;
    }
}

The output will receive all the data from the original stream and there's very little overhead so you don't need to worry about performance that much.

How do I return a reference to a Readable stream before creating it?

1 Answers1