1

If I have a single partition with 100'000 deleted rows in one cluster followed by a second cluster in the same partition with no deleted rows, will the performance of doing a SELECT * FROM example_table WHERE partition=that_partition AND cluster=the_second_cluster be affected by the tombstones present in the_first_cluster?

I'm expecting that if the retrieval of row sets with a where clause is constant then Cassandra will just jump past all of the tombstones to the second cluster, but I don't understand how the where clause finds the correct row so I don't know if this is the case and I didn't manage to find anything online that could enlighten me.

// Example table
CREATE TABLE example_table (
  partition TEXT,
  cluster TEXT,
  value BLOB,

  PRIMARY KEY (partition, cluster);

// Example layout of rows in a table
partition      |cluster            |value
that_partition |the_first_cluster  |some_value1 // Deleted, a tombstone
that_partition |the_first_cluster  |some_value2 // Deleted, a tombstone
... 99'997 more similar tombstone rows
that_partition |the_first_cluster  |some_value  // Deleted, a tombstone
that_partition |the_second_cluster |some_valueA // Not a tombstone
that_partition |the_second_cluster |some_valueB // Not a tombstone
... no tombstones in the_second_cluster
AMVaddictionist
  • 116
  • 1
  • 13

1 Answers1

4

A lot of tombstones on a partition will impact performance significantly IF its included in the result. A good write-up https://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets talks about it. Depending on the query, it may end up reading all 100,000 tombstones, and also possibly the original data if on a different sstable to satisfy the query. That generates a lot of garbage on the heap and will effect the JVMs GC along with a significant amount of CPU and IO for a single query.

However if the tombstones are point deletes and not range tombstones, and your query goes directly to the partition + clustering of a key not deleted you will be Ok. It's a fine line though and I would recommend not attempting it (what if someone tries reading it out of app as a ops/test task? It could cause long GCs and negatively impact cluster). Range tombstones kept in the partition index are deserialized as part of reading where to jump to to get within the column index size of the row, so even if not directly reading them it can still significantly impact allocation rate depending on how your tombstone was inserted.

There is a tombstone warn/failure threshold in cassandra.yaml that is set so it will let you know if your query is hitting them but it can be hard to tell until you hit failure point and queries die since just reported in logs.

I would recommend you time box your partitions to limit number of tombstones in each one.

Eugen Constantin Dinca
  • 8,994
  • 2
  • 34
  • 51
Chris Lohfink
  • 16,150
  • 1
  • 29
  • 38
  • Will this query create a range tombstone? `DELETE FROM example_table WHERE partition=that_partition AND cluster=the_first_cluster` – AMVaddictionist Apr 24 '18 at 14:58
  • Here they give an example of a tombstone generated by specifying both the partition and clustering key when doing a delete: http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html And they say that a row tombstone is made, not a range tombstone. – AMVaddictionist Apr 25 '18 at 08:24
  • 1
    in 3.0 deleting a row (partition + clustering) is a point delete, before 3.0 it was a range tombstone unless specifying the column too – Chris Lohfink Apr 27 '18 at 04:19