How can I use a event emitter as an async generator

Question

I'm trying to use the neat syntax of async generator with babel (I'm stuck with node 8) and I'm wondering how you would convert an event emitter to an async generator cleanly

What I got so far look like this

    const { EventEmitter } = require('events')

    // defer fonction for resolving promises out of scope
    const Defer = () => {
      let resolve
      let reject
      let promise = new Promise((a, b) => {
        resolve = a
        reject = b
      })
      return {
        promise,
        reject,
        resolve
      }
    }


    // my iterator function
    function readEvents(emitter, channel) {
      const buffer = [Defer()]
      let subId = 0

      emitter.on(channel, x => {
        const promise = buffer[subId]
        subId++
        buffer.push(Defer())
        promise.resolve(x)
      })

      const gen = async function*() {
        while (true) {
          const val = await buffer[0].promise
          buffer.shift()
          subId--
          yield val
        }
      }

      return gen()
    }

    async function main () {
      const emitter = new EventEmitter()
      const iterator = readEvents(emitter, 'data')

      // this part generates events
      let i = 0
      setInterval(() => {
        emitter.emit('data', i++)
      }, 1000)

      // this part reads events
      for await (let val of iterator) {
        console.log(val)
      }
    }

    main()

This is unweildy - can it be simplified?

mpen · Accepted Answer · 2020-04-03T02:23:34.670

12

I came up with this:

async *stream<TRecord extends object=Record<string,any>>(query: SqlFrag): AsyncGenerator<TRecord> {
    const sql = query.toSqlString();

    let results: TRecord[] = [];
    let resolve: () => void;
    let promise = new Promise(r => resolve = r);
    let done = false;

    this.pool.query(sql)
        .on('error', err => {
            throw err;
        })
        .on('result', row => {
            results.push(row);
            resolve();
            promise = new Promise(r => resolve = r);
        })
        .on('end', () => {
            done = true;
        })

    while(!done) {
        await promise;
        yield* results;
        results = [];
    }
}

Seems to be working so far.

i.e. you create a dummy promise like in Khanh's solution so that you can wait for the first result, but then because many results might come in all at once, you push them into an array and reset the promise to wait for the result (or batch of results). It doesn't matter if this promise gets overwritten dozens of times before its ever awaited.

Then we can yield all the results at once with yield* and flush the array for the next batch.

edited Apr 03 '20 at 02:23

answered Dec 15 '19 at 20:03

mpen

272,448
266
850
1,236

It could probably be simplified by feeding result row as an argument to the `resolve()` instead of storing in an intermediate array. Like, `resolve: (r: TRecord) => void`, `new Promise(r => resolve = r)`, `yield await promise`. – Klesun Jun 23 '21 at 23:37
@Klesun You sure that would work? What if `.on('result'` is called twice before awaiting the promise? Not positive if that can happen or not. – mpen Jun 24 '21 at 01:46
Hm, now that I think of it, I'm not positive either. Yeah, thanks, synchronously writing to the array is probably a much safer option. – Klesun Jun 24 '21 at 07:22
`.on('error', err => { throw err; })` will throw the error nowhere to be caught. – jcayzac Jul 01 '22 at 05:23
@jcayzac Won't be caught in here, but can be caught where you call this, no? – mpen Jul 02 '22 at 18:11
@mpen it can only be caught by the code calling your listener, i.e. most likely not your own code. Throwing an exception in an event listener just doesn't do what a naive programmer might think it does. – jcayzac Sep 14 '22 at 10:16

Khanh Hua · Answer 2 · 2018-12-10T06:07:57.610

Let's say we use redux-saga (as it uses generator at its core) and socket.io as an example of EventEmitter

import { call, put } from 'redux-saga/effects';

function* listen() {
  yield (function* () {
    let resolve;
    let promise = new Promise(r => resolve = r); // The defer

    socket.on('messages created', message => {
      console.log('Someone created a message', message);
      resolve(message); // Resolving the defer

      promise = new Promise(r => resolve = r); // Recreate the defer for the next cycle
    });

    while (true) {
      const message = yield promise; // Once the defer is resolved, message has some value
      yield put({ type: 'SOCKET_MESSAGE', payload: [message] });
    }
  })();
}

export default function* root() {
    yield call(listen);
}

The above setup should give you a generator that's blocked by the next event to be emitted by an event emitter (socket.io instance).

Cheers!

with this approach, would it be possible if i spam the socket with messages that some of them get lost because the promise is overwritten before it is yielded/awaited ? — miThom, Sep 05 '19 at 11:58

noseratio · Answer 3 · 2021-02-21T20:38:16.407

Here is another take at this, handling timer tick events with for await loop, custom Symbol.asyncIterator and a simple queue for any potential event buffering. Works in both Node and browser environments (RunKit, Gist).

async function main() {
  const emitter = createEmitter();
  const start = Date.now();

  setInterval(() => emitter.emit(Date.now() - start), 1000);

  for await (const item of emitter) {
    console.log(`tick: ${item}`);
  }
}

main().catch(e => console.warn(`caught on main: ${e.message}`));

function createEmitter() {
  const queue = [];
  let resolve;

  const push = p => {
    queue.push(p);
    if (resolve) {
      resolve();
      resolve = null;
    }
  };

  const emitError = e => push(Promise.reject(e));

  return {
    emit: v => push(Promise.resolve(v)),
    throw: emitError,

    [Symbol.asyncIterator]: () => ({
      next: async () => {
        while(!queue.length) {
          await new Promise((...a) => [resolve] = a);
        }
        return { value: await queue.pop(), done: false };
      },
      throw: emitError
    })
  };
}

score 0 · Answer 4 · answered Jul 22 '22 at 21:04

So, I worked on making this simple and generic. Here's what I got. It should be able to convert any callback based event listener to an async generator. I ensures all events get emitted exactly once and in the correct order.

  async function* eventListenerToAsyncGenerator(listenForEvents) {
    const eventResolvers = []
    const eventPromises = [
      new Promise((resolve) => {
        eventResolvers.push(resolve)
      })
    ]

    listenForEvents((event) => {
      eventPromises.push(
        new Promise((resolve) => {
          eventResolvers.push(resolve)
          eventResolvers.shift()!(event)
        })
      )
    })

    while (true) {
      yield await eventPromises.shift()
    }
  }

How can I use a event emitter as an async generator

4 Answers4