0

I've been trying to setup a redundant stateful set in kubernetes with the google cassandra image, as depicted in kubernetes 1.7 documentation.

According to the image used It's a stateful set with a consistency level of ONE. In my testing example I'm using a SimpleStrategy replication with a replication factor of 3, as I have setup 3 replicas in the stateful set in one datacenter only. I've defined cassandra-0,cassandra-1,cassandra-2 as seeds, so all are seeds.

I've created a keyspace and a table:

"create keyspace if not exists testing with replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }"

"create table testing.test (id uuid primary key, name text, age int, properties map<text,text>, nickames set<text>, goals_year map<int,int>, current_wages float, clubs_season tuple<text,int>);"

I am testing with inserting data from another unrelated pod, using the cqlsh binary, and I can see that data ends up in every container, so replication is successfull. nodetool status on all pods comes up with:

Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.16.0.161  71.04 KiB  32           100.0%            4ad4e1d3-f984-4f0c-a349-2008a40b7f0a  Rack1-K8Demo
UN  10.16.0.162  71.05 KiB  32           100.0%            fffca143-7ee8-4749-925d-7619f5ca0e79  Rack1-K8Demo
UN  10.16.2.24   71.03 KiB  32           100.0%            975a5394-45e4-4234-9a97-89c3b39baf3d  Rack1-K8Demo

...and all cassandra pods have the same data in the table created before:

 id                                   | age | clubs_season | current_wages | goals_year | name     | nickames | properties
--------------------------------------+-----+--------------+---------------+------------+----------+----------+--------------------------------------------------
 b6d6f230-c0f5-11e7-98e0-e9450c2870ca |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}
 5fd02b70-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}
 5da86970-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}

But then I delete one of those db replica pods(cassandra-0), a new pod springs up again as expected, a new cassandra-0 (thanks kubernetes!), and I see now that all the pods have lost one row of those 3:

 id                                   | age | clubs_season | current_wages | goals_year | name     | nickames | properties
--------------------------------------+-----+--------------+---------------+------------+----------+----------+--------------------------------------------------
 5fd02b70-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}
 5da86970-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}

...and nodetool status now comes up with:

 Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.16.0.161  71.04 KiB  32           81.7%             4ad4e1d3-f984-4f0c-a349-2008a40b7f0a  Rack1-K8Demo
UN  10.16.0.162  71.05 KiB  32           78.4%             fffca143-7ee8-4749-925d-7619f5ca0e79  Rack1-K8Demo
DN  10.16.2.24   71.03 KiB  32           70.0%             975a5394-45e4-4234-9a97-89c3b39baf3d  Rack1-K8Demo
UN  10.16.2.28   85.49 KiB  32           69.9%             3fbed771-b539-4a44-99ec-d27c3d590f18  Rack1-K8Demo

... shouldn't the cassandra ring replicate all the data into the newly created pod, and still have the 3 rows there in all cassandra pods?

... this experience is documented in github.

...has someone tried this experience, what might be wrong in this testing context?

super thanks in advance

joão tiago viegas
  • 67
  • 1
  • 9

1 Answers1

2

I think that after bringing down the node, you need to inform the other peers from the cluster that the node is dead and needs replacing.

I would recommend some reading in order to have a correct test case.

Horia
  • 2,942
  • 7
  • 14
  • In addition to this. Best way to do this in Kubernetes is probably to add PreStop hook where you remove the node. More info here: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/ – Simon Fontana Oscarsson Nov 06 '17 at 09:03
  • Thank you for the answers so far, but I have to clarify that the circumstance I wanted to simulate was an accidental failure of a specific node. I got the idea that cassandra caters for this specific situation. – joão tiago viegas Nov 07 '17 at 17:15
  • @Horia, thank you, did the reading there, `nodetool removenode` does re-establish the token assignments and replicates the data onto the new nodes replacing the dead ones. Still working in a clearer test scenario, but this has helped a lot. – joão tiago viegas Nov 12 '17 at 21:54
  • ...now I wonder, is there an automated way of removing the node as in `nodetool removenode` ? Has this step to be manual ? – joão tiago viegas Nov 12 '17 at 22:16