How do I calculate how long eventual consistency takes to become consistent?

Question

I am beginning to investigate nosql and document-oriented databases to store assets for HTML5 apps we are going to be serving on our website. This is intended to be a replacement for just storing files on the filesystem. They will be small, web-optimized files, including text files like html, js, css, and xml, and also binary, such as images, sounds, and fonts.

Since I'm interested in fault tolerance, the solutions I'm looking at (riak, Cassandra) use eventual consistency. While I understand the concept at an abstract level, when I'm talking to managers and decision makers, I am unable to explain in practical terms how long eventual consistency takes to become consistent. Miliseconds? Seconds? Minutes? Since I don't have any experience in this realm, I am looking for real world experience as far as what this means.

I understand that different variables will determine exactly how long any configuration takes, but I need to be able to start to understand what kinds of infrastructures we would be needed to build to support our requirements. So what I am looking for is if we need to optimize network latency, number of nodes, etc. to support our particular requirements.

We want to get to a point where we select platforms to test, and before we sink time into any particular solution, we'd like to be able to say "No, this isn't going to work for us."

We have systems now that use strict consistency (such as the filesystem on our webservers and our mysql databases), so our management is used to concepts like load and timeouts, and things being 'down'. But I am not able to communicate to them "Yes, the data is not available right now, but it's not down; it will be available eventually". They want to know "Well, how long is 'eventually'"?

How can I tell if an eventually-consistent system is going to work in practical terms for our websites?

Answer 1

As I am significantly more familiar with Riak than Cassandra, I will limit my discussion to how eventual consistency applies to Riak.

During normal operation, Riak supports tuneable consistency which allows you to tailor the consistency behaviour to your applications requirements. The default settings are however very sensible and works for most scenarios, in that they require a majority of replicas to respond to a read or write before it can be considered successful.

Although all replicas may not at every point in time be in exactly the same state these consistency settings will ensure that you read what you write. Inconsistency are traditionally corrected on reads through a process called read-repair, but can also be corrected periodically if active anti-entropy (new feature in version 1.3 of Riak) is enabled.

Eventual consistency is otherwise primarily to be considered during various failure scenarios. If e.g. a node gets separated from the rest of the cluster, it will (with default settings) continue to be able to accept writes and reads, which it will serve to the best of its ability depending on which data/replicas it holds. As it can not communicate with the rest of the cluster during this time, inconsistencies may arise. These will however be resolved once the cluster gets back to a state of normal operation. Exactly how long this can take depends on a number of external factors and can range from perhaps fractions of a second for temporary network failures to minutes or hours if manual intervention is required to correct the issue.