How to map over async generators?

Question

Let's say we have an async generator:

exports.asyncGen = async function* (items) {
  for (const item of items) {
    const result = await someAsyncFunc(item)
    yield result;
  }
}

is it possible to map over this generator? Essentially I want to do this:

const { asyncGen } = require('./asyncGen.js')

exports.process = async function (items) {
  return asyncGen(items).map(item => {
    //... do something
  })
}

As of now .map fails to recognize async iterator.

The alternative is to use for await ... of but that's nowhere near elegant as with .map

`.map()` exists only on arrays, not on generators - async or not. — VLAZ, Jun 01 '21 at 19:17

score 8 · Accepted Answer · edited Jun 01 '21 at 19:33

The iterator methods proposal that would provide this method is still at stage 2 only. You can use some polyfill, or write your own map helper function though:

async function* map(asyncIterable, callback) {
    let i = 0;
    for await (const val of asyncIterable)
        yield callback(val, i++);
}

exports.process = function(items) {
    return map(asyncGen(items), item => {
       //... do something
    });
};

vitaly-t · Answer 2 · 2021-12-16T04:23:19.620

The alternative is to use for await ... of, but that's nowhere near elegant as with .map

For an elegant and efficient solution, here's one using iter-ops library:

import {pipe, map} from 'iter-ops';

const i = pipe(
    asyncGen(), // your async generator result
    map(value => /*map logic*/)
); //=> AsyncIterable

It is elegant, because the syntax is clean, simple, and applicable to any iterable or iterators, not just asynchronous generators.
It is more flexible and reusable, as you can add lots of other operators to the same pipeline.

Since it produces a standard JavaScript AsyncIterable, you can do:

for await(const a of i) {
    console.log(a); //=> print values
}

P.S. I'm the author of iter-ops.

Nitsan BenHanoch · Answer 3 · 2022-04-10T13:22:27.763

TL;DR - If the mapping function is async:

To make asyncIter not wait for each mapping before producing the next value, do

async function asyncIterMap(asyncIter, asyncFunc) {
    const promises = [];
    for await (const value of asyncIter) {
        promises.push(asyncFunc(value))
    }
    return await Promise.all(promises)
}

// example - how to use:
const results = await asyncIterMap(myAsyncIter(), async (str) => {
    await sleep(3000)
    return str.toUpperCase()
});

More Demoing:

// dummy asyncIter for demonstration

const sleep = (ms) => new Promise(res => setTimeout(res, ms))

async function* myAsyncIter() {
    await sleep(1000)
    yield 'first thing'
    await sleep(1000)
    yield 'second thing'
    await sleep(1000)
    yield 'third thing'
}

Then

// THIS IS BAD! our asyncIter waits for each mapping.

for await (const thing of myAsyncIter()) {
    console.log('starting with', thing)
    await sleep(3000)
    console.log('finished with', thing)
}

// total run time: ~12 seconds

Better version:

// this is better.

const promises = [];

for await (const thing of myAsyncIter()) {
    const task = async () => {
        console.log('starting with', thing)
        await sleep(3000)
        console.log('finished with', thing)
    };
    promises.push(task())
}

await Promise.all(promises)

// total run time: ~6 seconds

How to map over async generators?

3 Answers3

More Demoing: