0

I have configured locally 3 nodes in on 'Test Cluster' of Cassandra. When I run them and create some keyspace or table also on all three nodes the keyspace or the table appears.

The problem I'm dealing with is, when I'm importing from CSV millions of rows in the table I already built the whole data suddenly appears on all three nodes. I have the same data replicated over the three nodes.

As I'm familiar with, the data I'm importing should be replicated/distributed over the nodes but partially. One partition on the first node, second on third, third on second node, fourth again on first node and ... Am I right or I'm missing something big?

Also, my write speed locally is about 10k rows / second for the multi-node cluster. Isn't that a little bit too low?

I want to create discussion so I can maybe learn something more from your experience and see where I'm messing things.

Thank you!

iMajna
  • 489
  • 4
  • 28

1 Answers1

0

The number of nodes that data is written to in your cluster is determined by the Replication Factor for that keyspace. If you have 3 nodes and the data is being written to all the nodes, then this setting must be set to 3. If you only want the data the be replicated to two nodes, you'd set this value to two.

Your write speed will be affected by the consistency level you are specifying on the write. If you have it set to ALL then you have to wait until all the nodes that are going to write the data have written the data (in your case all 3 nodes based on your replication factor). Dropping your consistency level on the write will probably net you faster write times. There is a balance between your replication factor, write consistency level, and read consistency level that you can research further.

gsteiner
  • 776
  • 5
  • 20
  • So what do you think, why all of my data is replicated to all nodes not by the partitions? And what do you propose for qurom set, is it better to set it to check only one node? Also, as I see you have better knowledge than me, I set `endpoint_snitch: GossipingPropertyFileSnitch` is that ok, or is it better to configure with `SimpleSnitch`? Thank you :) – iMajna Aug 10 '16 at 07:58
  • GossipingPropertyFileSnitch is fine as long as you configured your cassandra-rackdc.properties. See this page for more details: https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchGossipPF_c.html I'm not sure I understand your first question. Your data is replicated by a replication factor. If this replication factor is set to the number of nodes you have, then it replicate to all of them. If it's lower then it will replicate based on partition and where it falls in the token range. – gsteiner Aug 10 '16 at 12:28
  • As for a quorom config, it's up to you whether you want to guarantee consistency on write or read with the consistency level you set. If you are more write heavy then you would want to guarantee it on reads (since it will take less time to write). You can guarantee consistency as long as your CL at read + your CL at write is greater than your replication factor (CLr+CLw>RF). – gsteiner Aug 10 '16 at 12:31
  • Hmm, I'm trying just to prove that is possible with three nodes reach over etc. 30k rows /s while inserting the data. So I'm trying to trade consistency for higher throughput. I assumed that by the default configuration of three node cluster that lvl of throughput should be easy to reach, but about 18k rows/s is my avg limit. Also, I set keyspace to `replication_factor:1` and have consistency lvl on **ONE** Also I read that the way to speed up write ops is to set for keyspace `durability_write:false` where I would disable commit_log. Is that good thing to da or not? – iMajna Aug 10 '16 at 14:17
  • The way you have it configured it doesn't provide for any partition tolerance in your cluster. If you lose a node, all the data that resides on that node will be unavailable. With durability off and only writing to one node, I believe any time you lose a node you are guaranteed to lose data that is in memory and not flushed to disk. If you're okay with all that, yes, you will get faster writes. However, you could get equally faster writes, with the same consistency trade-off, and have some partition tolerance if you had a RF of 2 and CL at 1 for both read and write. – gsteiner Aug 10 '16 at 15:13
  • So if consistency level is by default on ONE it means that for both (read&write) is on ONE when I'm trying to read or write on node? – iMajna Aug 11 '16 at 12:14
  • Yes. On a write it will wait for one node to write the data before returning a success, on a read it will wait for one node to return the data before returning a success. – gsteiner Aug 11 '16 at 20:34
  • So I can change consistency level for single write or read in queries, with `USING` option or there is a way using command in CLI like `CONSISTENCY ONE` but just for single write or read? – iMajna Aug 12 '16 at 08:38