How to process RxJS stream n items at a time and once an item is done, autofill back to n again?

Question

I have a stream of events and I would like to call a function that returns a promise for each of those events, the problem is that this function is very expensive, so I would like to process at most n events at a time.

This pebble diagram is probably wrong but this is what I would like:

---x--x--xxxxxxx-------------x------------->  //Events
---p--p--pppp------p-p-p-----p------------->  //In Progress
-------d--d--------d-d-dd------dddd-------->  //Promise Done

---1--21-2-34-----------3----4-3210--------   //QUEUE SIZE CHANGES

This is the code that I have so far:

var n = 4;
var inProgressCount = 0;

var events$ = Rx.Observable.fromEvent(produceEvent, 'click')
  .map((ev) => new Date().getTime());

var inProgress$ = events$.controlled();

var done$ = inProgress$      
  .tap(() => inProgressCount++)
  .flatMap((timestamp) => Rx.Observable.fromPromise(expensiveComputation(getRandomInt(1, 5) * 1000, timestamp)));

done$.subscribeOnNext((timestamp) => {
  inProgressCount--;
  inProgress$.request(Math.max(1, n - inProgressCount));
});

inProgress$.request(n);

There are two issues with this code:

1. It's using the inProgressCount var which is updated with side effect functions.
2. The done$ subscription is only called once when I request more than 1 item from the controlled stream. This is making the inProgressCount var to update incorrectly, this eventually limits the queue to one at a time.

You can see it working in here: http://jsbin.com/wivehonifi/1/edit?js,console,output

Questions:

1. Is there a better approach?
2. How can I get rid of the inProgressCount variable?
3. Why is the done$ subscription only getting called once when requesting multiple items?

Update:
Answer to question #3: switchMap is the same as flatMapLatest, so that's why I was only getting the last one. Updated the code to flatMap instead of switchMap.

Answer 1

You actually don't need to use backpressure at all. There is an operator called flatMapWithMaxConcurrent that does this for you. It is essentially an alias for calling .map().merge(concurrency) and it only allows a maximum number of streams to be in flight at a time.

I updated your jsbin here: http://jsbin.com/weheyuceke/1/edit?js,output

But I annotated the important bit below:

const concurrency = 4;

var done$ = events$
  //Only allows a maximum number of items to be subscribed to at a time
  .flatMapWithMaxConcurrent(concurrency, 
    ({timestamp}) =>   
      //This overload of `fromPromise` defers the execution of the lambda
      //until subscription                    
      Rx.Observable.fromPromise(() => { 
        //Notify the ui that this task is in progress                                 
        updatePanelAppend(inProgress, timestamp);
        removeFromPanel(pending, timestamp);
        //return the task
        return expensiveComputation(getRandomInt(1, 5) * 1000, timestamp)
     }));