Blockchain backtracking logic inside RxJs Observable pipe

Question

Background

I am using NestJS and the Observable<AxiosResponse> pattern with the HttpModule to "observe" and eventually forward values returned by a JSON-RPC server, in this case a Blockchain node.

In case Blockchains are unfamiliar, they can be thought of as a linked list, where each new element in the list points to the previous element, something like this:

let blockchain = [
  [0,{value: 'something', previous: None}], 
  [1,{value: 'somethingelse', previous: 0}], ...
  [N,{value: 'somethingsomething', previous: N-1}]
]

Sometimes a "fork" can happen in a blockchain system. It would look something like the below tree:

[A] <-- [B]<--- [C]
         |
         --- [C'] <--- [D] <--- [E] <--- ... and so on

Problem

If my NestJS application gets Block [A] at time 0, block [B] at time 1 and block [C] at time 2, but suddenly, at time 3 I get [E], I would not get block [D] and [C']. This means that I would be unable to inspect the values in these two missing blocks.

Logic

Because all blocks have a pointer to a previous block, I do have the ability to retrieve block [D] by simply passing the pointer to it from block [E]. Similarly, from block [D] I could subsequently get block [C']. Because [C'] has a pointer to [B] I would have successfully retrieved all missing blocks.

I am quite new to Observables so I am not entirely sure of how I can backtrack recursively when I use the NestJS HttpModule in this way:

export class BlockchainService {
    private url: string;
    private top?: number;

    constructor(private httpService: HttpService,
                private config: ConfigService) {
        this.url = this.config.get<string>('my_blockchain_url')
    }

    getBestBlockHash(): Observable<AxiosResponse<any>> {
        return this.httpService.post(this.url, { 
            "method" : "getbestblockhash" 
        })
    }

    getBestBlock(): Observable<AxiosResponse<any>> {
        return this.getBestBlockHash().pipe(
            mergeMap((hash) => this.getBlock(hash.data.result))
        )
    }

    getBlock(hash: string): Observable<AxiosResponse<any>> {
        return this.httpService.post(this.url, {
            "method" : "getblock",
            "params" : {
                "blockhash" : hash
            }
        })
}

Attempt 1

Because the Observable holds the block data, I cannot evaluate whether to backtrack, or not without subscribe to it, or pipe it.

Using Mrk Sef's proposal below and the iif() operator, seems to take me further since I can pass the getBestBlock() Observable as a parameter to a checkBackTrack function leveraging iif() as follows:

    checkBackTrack(obs: Observable<AxiosResponse<any>>): Observable<AxiosResponse<any>> {
        let diff: number
        let previoushash: string
        console.log(diff, previoushash)

        obs.pipe(tap(block => {
            diff = this.top - block.data.result.height
            previoushash = block.data.result.previoushash
        }))
        console.log(diff, previoushash)

        const backTrackNeeded = iif( 
            () => diff > 0, 
            this.backTrack(diff, previoushash),
            obs 
        )
        return backTrackNeeded;
    }

where the backTrakc function looks like:

backTrack(n: number, previoushash: string): Observable<AxiosResponse<any>> {
        return (n < 0) ? EMPTY : this.getBlock(previoushash).pipe(
            switchMap(previousBlock => this.backTrack(n-1, previousBlock.data.result.previousblockhash)),
        )
    }

allows me to do the following: this.checkBackTrack(this.getBestBlock()).

However, I am unable to define diff and previoushash in the checkBackTrack function... Also, this introduces side effects, which I do not want.

Answer 1

Toward a solution

"How should I make sure each M empty line is backfilled?"

What this means isn't very clear, but we might be in a position where you're not sure how to clarify further. I'm not sure this will answer your question, but perhaps it will help you clarify further.

Some code

Here is how I might ensure "missing" values are merged into a stream. This is meant to be a paired down implementation (for example, it assumes index is monotonically increasing and will fail otherwise) for the sake of simplicity.

interface Message {
  index: number,
  value: string
}

// simulate an http response that takes 1 second to return
function httpBackupMessage(i: number): Observable<Message> {
  return timer(1000).pipe(
    map(() => ({ index: i, value: "backup something" }))
  );
}

// An observable of messages, but it's missing messages with
// index 2 & 3
const sparseMessage$: Observable<Message> = of({
  index: 0,
  value: "something"
}, {
  index: 1,
  value: "something"
}, {
  index: 4,
  value: "something"
}, {
  index: 5,
  value: "something"
});

// An observable of messages with missing messages retrieved from
// httpBackupMessage
const denseMessage$ = sparseMessage$.pipe(
  // Dummy value is ignored, but tells out logic below that index 
  // 0 (-1 + 1) is the first index.
  startWith({ index: -1, value: "dummy value" } as Message),
  pairwise(),
  concatMap(([prev, curr]) => 
    range(prev.index + 1, curr.index - (prev.index + 1)).pipe(
      concatMap(httpBackupMessage),
      endWith(curr)
    )
  )
);

// Start the observable and print the messages to the console
denseMessage$.subscribe(console.log);

As you can see, sparseMessage$ is missing a message with index 2 & 3. Below is what is emitted by denseMessage$:

{ index: 0, value: 'something' }
{ index: 1, value: 'something' }
{ index: 2, value: 'backup something' }
{ index: 3, value: 'backup something' }
{ index: 4, value: 'something' }
{ index: 5, value: 'something' }

Some Notes

1. This creates a new observable that transforms the original observable with the missing messages. It doesn't create a separate observable or array (both are possible, and not too dissimilar to what's done here).

2. This is emulating a sort of middle-ware with some business logic for missing values. And yet ...

Now, say my httpService.post() request gets delayed for some reason and the JSON-RPC response I get back has returned index: N + M

this makes it sound like you're worried about the source timing out or emitting out of order. Of course it's possible to band-aid fix this in a middle-ware layer (and in an old legacy system this might be the only way), but its best to avoid this sort of design wherever possible.

3. This solution is the shape of a solution without much extra (nowhere near production level code). It does no error handling. It doesn't ensure indices are monotonically increasing, etc. It is not meant to be a robust solution. Fortunately, many of those issues are addressed in the documentation/answered in other questions here on stackoverflow :)

4. Feel free to modify/clarify your question. If you have separate questions or you want to re-frame this question entirely, it's best to just open a new question.

Update: Some more code

I see on revision that perhaps you're asking about how to navigate an asynchronous linked list. This is more interesting because each next value depends on the previous one (You can't index into a linked list, you must follow the links).

Here's an example of how you might follow an async linked list back 4 nodes and then emit them in the correct order.

// This an be any sort of resource, we'll use numbers
type Link = number

// Entries can be thought of as nodes in a linked list
interface DataEntry {
  prev: Link;
  value: string;
}

// Simulate Data stored on a server, access only allowed via getData
const serverData = new Map<Link, DataEntry>([
  [ 4153, {
    prev: 4121,
    value: 'Something 5',
  }], [ 4273, {
    prev: 4153,
    value: 'Something 6',
  }], [ 4291, {
    prev: 4273,
    value: 'Something 7',
  }], [ 4300, {
    prev: 4291,
    value: 'Something 8',
  }]
]);

// Simulate a call to the server. Each call takes 50ms
function getData(pointer: Link): Observable<DataEntry> {
  return timer(50).pipe(map(() => serverData.get(pointer)));
}

// Navigate backwards from start, n times. Buffer all values and emit
// in reverse order (The value n steps back is emitted first)
function backtrackLinks(start: DataEntry, n: number): Observable<DataEntry> {
  return n < 0 ? EMPTY : getData(start.prev).pipe(
    switchMap(previous => backtrackLinks(previous, n - 1)),
    endWith(start)
  );
}

// Pretend we've aquired a node through some earlier process
const nodeNine = {prev: 4300, value: 'Something 9'}
// An example of running the `backtrackLinks` observable and 
// printing the nodes to the console
backtrackLinks(nodeNine , 4).subscribe(console.log);

The output:

{prev: 4121, value: "Something 5"}
{prev: 4153, value: "Something 6"}
{prev: 4273, value: "Something 7"}
{prev: 4291, value: "Something 8"}
{prev: 4300, value: "Something 9"}

You can the use all of the usual RxJS suspects to create/merge this with other streams (That's where I imagine you'll acquire a start node from which to backtrack).

In the previous section, I used range, concatMap, & endWith but here you could replace that with backtrackLinks This embeds all that logic in one function since backtrackLinks(value, 0) equals of(value).