Turning a callback which gets called multiple times into a generator function

Question

I have a function that fetches rows of a database asynchronously, and calls a callback function for each row. I am trying to write a wrapper which is a generator function which yields each time a row is returned, but I am not seeing how to properly yield.

The original code looks something like this:

db.each(query, (err, row) => {
  // do something here with row
}, () => {
  // called after the last row is returned
})

I'm familiar with how a generator works, but the yield seems to belong in the generator function itself, not in an anonymous function. So I think something like this wouldn't work:

function* dbEach(db, query) {
    db.each(query, (err, row) => {
      yield row
    })
}

When I actually try this I get an error "Unexpected identifier".

I looked a bit further and it appears that ES2018 now has asynchronous iterators which are supposed to make this possible. However, I'm having trouble wrapping my head around how exactly I can use async iterators in the case where I already have a callback which is getting called multiple times.

Not sure if this is what you need here, `function* dbEach(db, query) { yield db.each(query, (err, row) => row); }` — Rikin, Mar 10 '19 at 21:40
How can you tell when all `.each` callbacks have finished? Is there a `.finished` property, or something like that you can check? (Maybe there's a better function than `each` from which you can get all rows as an *array*, that would make things far easier) — CertainPerformance, Mar 10 '19 at 21:43
@CertainPerformance There is another optional callback for that. — Michael, Mar 10 '19 at 21:44
`yield*` can only be used on a generator function. Your `db.each` seems like regular function. — Rikin, Mar 10 '19 at 21:45
Not sure whether this is a duplicate of [How to convert Node.js async streaming callback into an async generator?](https://stackoverflow.com/q/50862698/1048572) — Bergi, Mar 10 '19 at 22:37

CertainPerformance · Accepted Answer · 2019-03-10T21:59:49.847

You can make the generator async, then await a Promise that resolves with all rows (so that you have reference to a rows variable on the top level of dbEach), and then you can yield each row in that rows array:

async function* dbEach(db, query) {
  const rows = await new Promise((resolve, reject) => {
    const rows = [];
    db.each(query, (err, row) => {
      if (err) reject(err);
      else rows.push(row);
    }, () => resolve(rows));
  });
  for (const row of rows) {
    yield row;
  }
}

Use with:

for await (const row of dbEach(...)) {
  // do something
}

It looks like .each is designed for running a callback on each row, which isn't exactly optimal for what you're trying to accomplish here with a generator - if possible, it would be great if there was a method in your database that allows you to get an array of rows instead, for example:

async function* dbEach(db, query) {
  const rows = await new Promise((resolve, reject) => {
    db.getAllRows(query, (err, rows) => {
      if (err) reject(err);
      else resolve(rows);
    });
  });
  for (const row of rows) {
    yield row;
  }
}

Though, I don't think that a generator helps a lot here - you may as well just await a Promise that resolves to the rows, and iterate over the rows synchronously:

function getRows(db, query) {
  return new Promise((resolve, reject) => {
    db.getAllRows(query, (err, rows) => {
      if (err) reject(err);
      else resolve(rows);
    });
  });
}

const rows = await getRows(...);
for (const row of rows) {
  // ...
}

The problem with getting an array of rows is that there could be (for instance) 50 million rows buffered up in memory... — Michael, Mar 10 '19 at 21:59
Given that the callback is called by the db's *internals*, and that said call isn't controllable / throttleable from the outside, I'm somewhat doubtful that a generator would be able to interface with that *without* having many rows possibly "pile up". Eg, what if the callback is called 1000 times before the generator consumer has had the spare resources to do its operation? — CertainPerformance, Mar 10 '19 at 22:31
How does `await` work without being inside the body of `async`? — zer00ne, Mar 23 '19 at 02:22
@zer00ne OP is looking to use the new asynchronous iteration syntax, which of course requires its `await` to be inside an `async` function, but the problem OP was running into was figuring out how to set up the generator, not with how to use `await`. The containing block would have to be an `async` function, of course — CertainPerformance, Mar 23 '19 at 09:26

Turning a callback which gets called multiple times into a generator function

1 Answers1