2

I have set up a fully-replicated, 3-node Cassandra cluster on a single machine using Docker containers, with the following status:

Datacenter: dc_n1
=================
Status Address Load Tokens Owns Host_ID Rack
UN 172.18.0.3 83.98 MiB 256 100.0% 5bf rack_n1

Datacenter: dc_n2
=================
Status Address Load Tokens Owns Host_ID Rack
UN 172.18.0.6 83.52 MiB 256 100.0% 0518 rack_n2

Datacenter: dc_n3
=================
Status Address Load Tokens Owns Host_ID Rack
UN 172.18.0.2 83.52 MiB 256 100.0% ca95 rack_n3

Now consider the following keyspace:

create KEYSPACE stackoverflow WITH replication = {'class': 'NetworkTopologyStrategy', 'dc_n1':1,'dc_n2':1,'dc_n3':1};

and a table defined as (let's assume T_notID is unique):

create TABLE stackoverflow.TABLE (T_ID int PRIMARY KEY, T_notID int, T_Data text);

When I dispatch a number (say, a hundred) of concurrent Java threads submitting the following two JDBC queries to Cassandra nodes (repeatedly, for a minute), I witness a 100X performance drop for (B) queries:

(A) SELECT T_Data FROM TABLE WHERE T_ID = ?

(B) SELECT T_Data FROM TABLE WHERE T_notID = ? ALLOW FILTERING

(B) queries also raise many Cassandra errors that: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (timeout while waiting for repair of inconsistent replica):

I understand that, in general, usage of 'ALLOW FILTERING' in queries is an antipattern and should be used with extreme care, but in the simplified example above, since the data is fully replicated and one copy of each item resides at every node, I do not understand why PK-queries and nonPK-queries behave differently.

In other words, considering the fact that read consistency in this scenario is ONE and each node can respond to the queries without communicating with other nodes in the cluster (regardless of the primary key definition), I would have expected similar behavior from Cassandra to a centralized SQL database.

Can someone please explain to me why this is happening and/or how I can fix it?

Community
  • 1
  • 1
Kiarahmani
  • 47
  • 1
  • 8

1 Answers1

5

When you have condition on the partition key - Cassandra knows where data is located on disk, and can jump to begin of partition and read the data sequentially. But if you have condition on non-partition then your query will need to go through all data and filter out only necessary pieces - you don't have any indexes that will allow Cassandra to figure out where data is located.

If you need to run the query on the T_notID very often, then you can create materialized view or secondary index (but you need to understand what limitations they have).

DataStax has a very good blog post about ALLOW FILTERING and where it could be used.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132