Can I generate a sequence of Rx Observables from chained AJAX calls?

Question

Technical Context

I am using jQuery in a web browser to call an API that returns a set of entries from a log.

API requests take two parameters:

• offset_timestamp: an integer specifying the earliest possible entry I want
• limit: an integer specifying the number of records to return

Example request and response

Request with parameters:

- offset_timestamp = 100
- limit = 50


curl "https://example.com/log?offset_timestamp=100&limit=5"


Resulting JSON Response:

{
    next_timestamp: 222,
    end_of_log: false,
    entries: [
        {
            timestamp: 111,
            data: { ... }
        },
        {
            timestamp: 112,
            data: { ... }
        },

        ...

        {
            timestamp: 160,
            data: { ... }
        }
    ]
}

If I were using plain jQuery + callbacks, I think I'd have to chain the AJAX calls recursively. Something along the lines of:

// NOTE: I have NOT tested this code.
//       I just wrote it for illustration purposes

function getBatch(offset, limit, callback) {
    var url = "https://example.com/logs?offset_timestamp=" + offset + "&limit=" + limit;
    var ajaxParams = {
        method: "GET",
        url: url
    };

    jQuery.ajax(ajaxParams))
        .done(function(data) {
            if (data.end_of_log) {
                return callback(data.entries);
            }
            return getBatch(data.next_timestamp, limit, function(entries) { 
                return callback(entries.concat(data.entires));
            });
        });
}

function processEntries(entries) {
    for (var i = 0; i < entries.length; i++) {
        console.log(i.data);
    }
}

getBatch(0, 50, processEntries);

Naturally, I would rather have a sequence of Observables (each holding one batch of entries) so I could use flatMap() to get a sequence of all entries.

Question

If create an Observable from a jQuery call, e.g. Rx.Observable.fromPromise(jQuery.ajax({...})), is it possible to use RxJS to chain together an arbitrary number of these Observables, using the value of response.next_timestamp from the previous call in the parameters of the subsequent call?

Answer 1

I would try to use the expand operator. You can also see an example of use here : RxJs: How to loop based on state of the observable?, and here : RxJS, how to poll an API to continuously check for updated records using a dynamic timestamp

In short it allows you to do a recursion with observables, signalling the end of recursion by returning Rx.Observable.empty().

Answer 2

Rx.Subject allows us to write to (push data) as well as read from (subscribe) a stream. Rx.Subject behaves just like Rx.Observable, however, instances have an onNext method that allows us to push data.

We can model our requests as an Rx.Subject, and have our stream of responses depend on it. We can then subscribe to the responses and push the next request to the request stream:

// Represents our stream of requests
const request$ = new Rx.Subject();

// Represents our AJAX responses
const response$ = request$
    .startWith(initialRequest)
    .flatMap(makeAjaxCall)

    // Convert to a hot stream so we don't end up making an AJAX
    // request for every subscriber of this stream.
    .share();

// And we can map over response$ to get the updated timestamp
const timestamp$ = response$.map(getTimeStamp);

// And we can subscribe to the timestamp$ stream to push new requests
timestamp$.subscribe(timestamp => {
    const request = makeRequestFromTimestamp(timestamp);

    // Call onNext to push a new item onto our request$ stream
    request$.onNext(request);
});

// And we can subscribe to our responses. Note that our response$
// stream will continue indefinitely.
response$.subscribe(x => console.log(x));