0

I want to implement the producer consumer pattern with a pool of bacon.js event streams. Here's the specific problem I'm trying to solve:

I have a list of 'n' urls. I want to create event streams to make http requests for those urls, but I want to limit it to 'x' streams ('x' network requests) at a time. In the event handler for the above streams, I create a new event stream that writes the http response to a file. But I want to limit the number of streams writing to file to 'y' at a time.

In Gevent/Java, I'd create thread pools of appropriate size and use threads from the appropriate thread pool. How do I do something similar for eventstreams?

tldr
  • 11,924
  • 15
  • 75
  • 120
  • 1
    Check out the newly added `flatMapWithConcurrencyLimit` method: https://github.com/baconjs/bacon.js/tree/master#observable-flatmapwithconcurrencylimit – raimohanska May 25 '14 at 16:56
  • perfect! Feel free to add it as an answer and I'll accept it. – tldr May 25 '14 at 17:55

1 Answers1

1

Using flatMapWithConcurrencyLimit you'll be able to control the number of spawned streams:

function fetchUsingHttp(url) { .. } // <- returns EventStream of http result
function writeToFile(data) { .. } // <- returns EventStream of file write result
var urls; // <- EventStream of urls
var maxRequests, maxWrites; // <- maximum concurrency limits
var httpResults = urls.flatMapWithConcurrencyLimit(maxRequests, fetchUsingHttp)
var fileWriteResults = httpResults.flatMapWithConcurrencyLimit(maxWrites, writeToFile)
fileWriteResults.log()
raimohanska
  • 3,265
  • 17
  • 28
  • Perfect! One last thing - the list of urls comes from a single event (http response). How do I create the stream of individual urls from a single list? – tldr May 26 '14 at 20:03
  • 1
    Bacon.fromArray(things) will give you a stream of things. – raimohanska May 27 '14 at 16:04