Async iterators with fp-ts and mongo db

Question

With mongodb in node we can use async iterators. Basic example:

const documents: Record<string, any>[] = [];
let cursor = db.collection('randomcollection').find();

for await (let doc of cursor) {
  documents.push(document);
}

How does async iterators translate to functional programming, using fp-ts? Is it possible to express the for loop above in fp-ts? I have searched, but found no documentation regarding async iterators.

  const whereIamRightNow = pipe(
    TE.bindTo('db')(createMongoClientWithEncryption),
    TE.bind('cursor', ({ db }) => TE.of(getQueryCursor(dateRange)(db.client))),

    // how to async iterate over query cursor and call a processing function for each record?
  );

Hey, you should have a look at [fp-ts-ixjs](https://github.com/werk85/fp-ts-ixjs) (referenced in the [fp-ts docs](https://gcanti.github.io/fp-ts/ecosystem/#bindings)) which seems to expose bindings for the [ReactiveX/IxJS](https://github.com/ReactiveX/IxJS) library in the fp-ts ecosystem. I haven't tried it yet so I can't propose an answer to your question though. — Benoit, Aug 11 '21 at 16:04

score 0 · Answer 1 · answered Oct 09 '21 at 09:33

My suggestion for how to write this with fp-ts change a little bit depending on the exact desired flow, but if you want to collect up all the awaited values of a collection of Promises I think you could do something like:

import { pipe } from 'fp-ts/lib/function';
// Task is the `fp-ts` way of tracking asynchronous work
import * as T from 'fp-ts/lib/Task';

const documents: T.Task<readonly any[]> = pipe(
  // For simplicity let's assume the db's find method returns a Task[]
  db.collection('randomcollection').find(),
  T.sequenceArray
);

sequenceArray is similar to awaiting all items in the array with Promise.all before adding them to documents.

I don't believe there's any better way in vanilla fp-ts. A comment indicated additional libraries which likely have better support specifically for iterators, but I wanted to introduce Task which is the fp-ts monad for asynchronous work. Once you have defined an entire Task you call it like a function and it transforms the value into a Promise.

const docs = await documents();

Hi @Souperman Thank you for taking the time to answer. Your answer does work well for promises (maybe better to use TE.tryCatch here since the promise might fail). But for high volume collections await all promises to fulfill will probably lead to memory issues. Therefore I'm still interested in how to to work with async iterators in fp-ts. — florian norbert bepunkt, Oct 10 '21 at 10:29
Yeah I’ll see if I can find some more information about about iterables specifically. You’re right about `tryCatch`. I was trying to keep the example simple to introduce sequence but if you’re curious `TaskEither` is likely to be the type you actually want to use. (You may already know) — Souperman, Oct 10 '21 at 12:11

Async iterators with fp-ts and mongo db

1 Answers1