Is it possible to combine two pooledMap arrays in deno?

Question

const pool = pooledMap(20, feeds, (url) => {
  return getWithXidel(url, '//item/link');
});
const pool2 = pooledMap(20, feedEntries, (url) => {
  return getWithXidel(url, "//entry/link[@rel='alternate']/@href");
});

for await (const data of pool) {
  console.log('data: ', data); 
  console.log('found links: ', data?.links?.length);

  for (const link of data?.links) {
    const url = link.trim();

    results.push(url);
  }
}

Is it possible to combine these two pools into one? I tried Array.concat but it breaks it. says its not iterable.

Answer 1

Deno standard library's pooledMap returns an AsyncIterableIterator that, like an AsyncGenerator, can be iterated through using for await...of and is not synchronously iterable.

Deno standard library also provides a MuxAsyncIterator that "multiplexes multiple async iterators into a single stream" which you can use to "combine" multiple pools:

import {
  pooledMap,
  MuxAsyncIterator,
} from "https://deno.land/std@0.116.0/async/mod.ts";

const pool = pooledMap(20, feeds, (url) => {
  return getWithXidel(url, '//item/link');
});
const pool2 = pooledMap(20, feedEntries, (url) => {
  return getWithXidel(url, "//entry/link[@rel='alternate']/@href");
});

const mux = new MuxAsyncIterator();
mux.add(pool);
mux.add(pool2);

for await (const data of mux) {
  console.log('data: ', data); 
  console.log('found links: ', data?.links?.length);

  for (const link of data?.links) {
    const url = link.trim();

    results.push(url);
  }
}