1

We have less than 50GB of data for a table and we are trying to come up with a reasonable design for our Cassandra database. With so little data we are thinking of having all data on each node (2 node cluster with replication factor of 2 to start with).

We want to use Cassandra for easy replication - safeguarding against failover, having copies of data in different parts of the world and Cassandra is brilliant for that.

Moreover, best model that we currently came up with would imply that a single query (consistency level 1-2) would involve getting data from multiple partitions (avg=2, 90th %=20). Most of the queries would ask for data from <= 2 partitions but some might go up to 5k.

So my question here is whether it is really a problem? Is Cassandra slow to retrieve data from multiple partitions if we ensure that all the partitions are on the single node?

eddyP23
  • 6,420
  • 7
  • 49
  • 87

2 Answers2

4

EDIT:

Misread question my apologies for other folks coming here later. Please look at the code for TokenAwarePolicy as a basis to determine replica owners, once you have that you can combine your query with the IN query to get multiple partitions from a single node. Be mindful of total query size still.

Original for reference:

Don't get data from multiple partitions in a single query, the detail of the why is here

The TLDR you're better off querying asynchronously from multiple different partitions that requiring the coordinator to do that work.

  1. You require more of a retry if you fail (which is particularly ugly when you have a very large partition or two in that query)
  2. You're waiting on the slowest query for any response to come back, when you could be returning part of the answer as it comes in (or even include a progress meter based on the parts being done).
Ryan Svihla
  • 136
  • 4
  • Thanks @RyanSvihla, I have read the post you link to. But the big difference is that any node in my cluster has all the data on it, so there is no coordinator forwarding partial queries to some other nodes. The same coordinator node, works out the response and sends it back. That is why I am hoping that in my case it might be as fast as putting everything into one big partition – eddyP23 Jul 20 '17 at 14:28
  • Yeah needed my morning coffee, in that case you'll need to make your queries token aware and then you can pull it off with the IN query. Use the TokenAwarePolicy as your code sample to build out the client side code to get a list of replicas for each partition https://github.com/datastax/java-driver/blob/97bf15811343b4e4b38cc0f002ced28f475a1d63/driver-core/src/main/java/com/datastax/driver/core/policies/TokenAwarePolicy.java#L129 – Ryan Svihla Jul 20 '17 at 14:33
  • Also this should be faster that one big partition because of the issues you see with CASSANDRA-9754 but I'd be interested in your benchmarks – Ryan Svihla Jul 20 '17 at 14:36
  • I will try to test it on my machine. – eddyP23 Jul 20 '17 at 14:42
  • please take a look at my answer below. – eddyP23 Jul 25 '17 at 10:22
1

I did some testing on my machine and results are contradicting what Ryan Svihla proposed in another answer.

TL;DR storing same data in multiple partitions and retrieving via IN operator is much slower than storing the data in a single partition and retrieving it in one go. PLEASE NOTE, that all of the action is on a single Cassandra node (as the conclusion should be more than obvious for a distributed Cassandra cluster)

Case A Insert X rows into a single partition of the table defined below. Retrieve all of them via SELECT specifying the partition key in WHERE.

Case B Insert X rows each into a separate partition of the table defined below. Retrieve all of them via SELECT specifying multiple partition keys using WHERE pKey IN (...).

Table definition

pKey: Text PARTITION KEY
cColumn: Int CLUSTERING KEY
sParam: DateTime STATIC
param: Text (size of each was 500 B in tests)

Results

Using Phantom Driver

  1. X = 100 A - 10ms B - 150ms r = 15
  2. X = 1000 A - 20ms B - 1400ms r = 70
  3. X = 10000 A - 100ms B - 14000ms r = 140

Using DevCenter (it has a limit of 1000 rows retrieved in one go)

  1. X = 100 A - 20ms B - 900ms r = 45
  2. X = 1000 A - 30ms B - 1300ms r = 43

Technical details:

  • Phantom driver v 2.13.0
  • Cassandra 3.0.9
  • Windows 10
  • DevCenter 1.6
eddyP23
  • 6,420
  • 7
  • 49
  • 87
  • The Spark Cassandra connector retrieves several partitions at once from the same machine at a great performance boost (and has this logic baked into the driver), So I suspect based on the # of rows you're retrieving is resulting in a total request size is just too large to get a benefit from combining requests. I'd be curious what nodetool cfhistograms outputs. – Ryan Svihla Jul 25 '17 at 13:57
  • Also I suggest looking at CASSANDRA-9754 for the rationale why cell count is important when determining if a query is efficient or not. If you are retrieving several larger partitions at once then you'd definitely be better off doing it asynchronously, though it may strain the server more in the process. – Ryan Svihla Jul 25 '17 at 14:04