correct way to process data from an rxjs oberservable subscription with back-pressure

Question

I have an rxjs.observable (rxjs version 6.2.1) that returns urls I need to make a GET request to.

var subscription = urlObservable$.subscribe(
  function (url) {
    console.log('new URL: ' + url);
    processURL(url)
  },
  function (err) { console.log('Error: ' + err); },
  function () { console.log('Completed'); }
);

For every url I need to make the request via the function processURL(url). What is the correct way, accordingly to the react philosophy, to process all this incoming urls and make the requests one by one instead of firing all of them as soon as the subscribe emits data? Please note that in this case, the observable urlObservable$ will return data much faster than the request that needs to be made with the returned url.

processURL can return a promise.

Thanks.

Is there a limit to the number of urls your subscriber will get? Can you not modify the event emitter to only emit after they've all been collected? — bugs, Jun 13 '18 at 13:38
the observable will emit all url at once, then I use flatMap to do some async checks. this includes some delay. Since there are many urls, I don't want to wait until all have been checked and then proceed to the next step. I thought that was the whole point of rxjs, that I can better control the flow of data from 1 step to the next, isn't it? — otmezger, Jun 13 '18 at 13:41
Question and comment are ambiguous. In the question you say that you don't want to process them at once, in the comment you say that you don't want to wait. Could you please clarify your intention? — a better oliver, Jun 13 '18 at 13:44
I'm assuming there is a best practice that will allow me to start processing the URLS as soon as the first comes in inside the subscriber without at some point ending up with many requests in parallel. So @abetteroliver it's both at the same time. can this be made? probably my assumptions are wrong. — otmezger, Jun 13 '18 at 13:48
I still don't understand everything, but it seems like you want two steps in the pipeline to be processed in parallel, is that correct? — a better oliver, Jun 14 '18 at 07:12
@abetteroliver what I'm interesting in is learning what would be the best approach, according to best practices. My goal is to be able to process the data in the faster way but not all calls in parallel. If I need to do one by one, it's also good. — otmezger, Jun 14 '18 at 12:38

score 2 · Accepted Answer · answered Jun 13 '18 at 13:51

If urlObservable$ is emitting just strings you can simply use concatMap that always waits until the previous Observable completes:

urlObservable$
  .pipe(
    concatMap(url => processURL(url)),
  )
  .subscribe(...);

This will work even if processURL returns a Promise.

correct way to process data from an rxjs oberservable subscription with back-pressure

1 Answers1