0

I was planning to upgrade my datastax instance in AWS from t2.large to t2.2xlarge. Our current cluster contain 6 SearchGraph nodes

Datacenter: SearchGraph
=======================
UN  192.168.8.1  469 MiB     1            ?       936a1ac0-6d5e-4a94-8953-d5b5a2016b92  rack1
UN  192.168.8.2  427.71 MiB  1            ?       3f41dc2a-2672-47a1-90b5-a7c2bf17fb50  rack1
UN  192.168.8.3  431.27 MiB  1            ?       29f8fe44-3431-465e-b682-5d24e37d41d7  rack2
UN  192.168.8.4  480.73 MiB  1            ?       1f7de531-ff51-4581-bdb8-d9a686f1099e  rack2
UN  192.168.8.5  498.9 MiB   1            ?       27d37833-56c8-44bd-bac0-7511b8bd74e8  rack2
UN  192.168.8.6  882.4 MiB   1            ?       0822145f-4225-4ad3-b2be-c995cc230830  rack1

Since our Replication factor is 3, we can survive even if our 2 instance got down for instance upgradation purpose. I need clarity on below upgradtion step which i was planning to proceed with is correct or not ?

Step 1) 

nodetool flush 
sudo service dse stop

Step 2) Take AMI of an instance

Step 3) Launch a new t2.2xlarge instance from taken AMI.
(Note : New instance ip should be same as of previous one )

Step 4) sudo service dse start

Sreeraju V
  • 535
  • 2
  • 5
  • 19
  • it's better to use `nodetool drain`: https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsDrain.html – Alex Ott Feb 08 '18 at 08:22
  • I can't say much about it - I never tried to use it such way – Alex Ott Feb 08 '18 at 12:32
  • @AlexOtt Thanks. I will follow your recommendation instead of Step 1. Did i need to do any other changes due to old instance details from ami get replicate in new instance ? Will i have to remove any saved caches of previous instance. – Sreeraju V Feb 08 '18 at 12:36

1 Answers1

0

Right so this is not an upgrade, rather you are moving the data onto a large machine. As long as you preserve the data directories then the node will persist the same token ranges and node id (all held in the cassandra system tables).

However note that it sounds like you're re-mounting an AMI snapshot, the node will be a bit "behind" compare to the other nodes so if your read consistency is not Quorum then your reads could hit the old node and get out of date data. Its probably also a good idea to run repair once you're done.

markc
  • 2,129
  • 16
  • 27
  • If the downtime is small, then it should catch up with changes via hints... But I agree about running repair - just to be sure that everything in sync – Alex Ott Feb 09 '18 at 10:03
  • 1
    Yes good point, I'd forgotten about hints! By default they are 3 hours as I remember. Looking at the operations here they probably would be done within the 3 hour window but not guaranteed. However worth noting if you see dropped messages in `nodetool tpstats` then you shouldn't rely fully on hints iirc – markc Feb 12 '18 at 15:49