Amazon Dynamo: Anti-Entropy Replica Synchronisation using Merkel Trees

Question

Context: I am trying to implement Amazon Dynamo's Replica Synchronisation which uses Merkel Trees to detect divergence amongst replicas.

As mentioned in the web version of the paper here,

Dynamo uses Merkle trees for anti-entropy as follows: Each node maintains a separate Merkle tree for each key range (the set of keys covered by a virtual node) it hosts. This allows nodes to compare whether the keys within a key range are up-to-date. In this scheme, two nodes exchange the root of the Merkle tree corresponding to the key ranges that they host in common. Subsequently, using the tree traversal scheme described above the nodes determine if they have any differences and perform the appropriate synchronization action.

I do not understand what is meant by "the appropriate synchronisation action". By using the appropriate traversal, assume we found a discrepancy between two Merkel tree's root node - how would we know which one is the correct one? Do we always take the more updated one according to its logical clock time stamp?

Answer 1

The Dynamo paper which you linked (note that this is not the same as today's DynamoDB product, so I removed the dynamodb tag which you added on your question), explains how the reconciliation between two versions is done, using vector clocks (which aren't as simple as a "time stamp"):

Dynamo uses vector clocks in order to capture causality between different versions of the same object. A vector clock is effectively a list of (node, counter) pairs. One vector clock is associated with every version of every object. One can determine whether two versions of an object are on parallel branches or have a causal ordering, by examine their vector clocks. If the counters on the first object’s clock are less-than-or-equal to all of the nodes in the second clock, then the first is an ancestor of the second and can be forgotten. Otherwise, the two changes are considered to be in conflict and require reconciliation.

In other words, usually Dynamo can tell which value is the newer one based on the vector clocks (the paper explain what this means in detail), but it is also possible that there is a conflict - imagine that the same item was read and modified concurrently, and differently, by two different and uncommunicating replicas. In this case, both versions of the data will be saved in the database and returned to the client, and the client will need to reconcile them using some client-specific logic. For example, in Amazon's shopping cart example, if one modification was to add product A to the shopping cart, and a second modification was to add product B to the shopping cart, the reconciliation would be to add both products - A and B - to the shopping cart.