1

I have a Readable stream in object mode that I'm pushing data in, like this

const getReadStream = () => {
    const stream = new Readable({ objectMode: true, read: () => {} });

    const get = async (page = 1) => {
       const { data } = await client
           .request(...)
           .catch((error) => {
               stream.emit('error', error);
               return { data: undefined };
            });
    
       const results = parseFn(data);
    
       if (results.length > 0) {
           results.forEach((row) => stream.push(row));
           get(page + 1);
       } else {
           stream.push(null);
       }
    };
    
    get();

    return stream;
};

I would want to consume it into some Writable stream, like this

const consumeStream = async () => {

    const readableStream = getReadStream();

    const pipeline1 = pipeline(
        stream,
        transform1,
        transform2,
        someWritableStream,
    );

    if (!certainCondition) {
        return pipeline1;
    }

    const writeStream2 = new Writable({
        objectMode: true,
        write: (rows: any[], _, callback) => {
            somePromise(rows).then(() => callback());
        },
    });

    const pipeline2 = pipeline(
        stream,
        transform3,
        transform4,
        writeStream2,
    );

    return Promise.all([pipeline1, pipeline2]);
};

My question is, in the case where consumeStream proceed with pipeline2 , is it the correct way to implement multiple writable stream with only 1 readable stream (as the number of objects processed by pipeline1 = the number of objects processed by pipeline2. And if it is not, what is the correct way to implement this?

hieumdd
  • 13
  • 3

0 Answers0