2

We have two nodes. Node1 was down for a long time. During this time Node2 raised 1 TB capacity when Node1 has 100 GB.

We tried to repair Node1 with nodetool repair but nothing changed. After that we started nodetool repair in Node2, it took 5 days to compaction but nothing changed either.

Actual status here: 

Datacenter: dc1
===============
Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID    Rack

UN  172.x.y.z     149.46 GB  256          100.0%            xxx  rack1
UN  172.x.y.k     1.04 TB    256          100.0%            xyz  rack1

Nodes are in AWS. What should we do?

Ekrem Gurdal
  • 1,118
  • 13
  • 14
  • Which version of cassandra do you use? What is your replication factor? Are there many updates or deletes? – Mandraenke Mar 22 '18 at 09:20
  • @Mandraenke cqlsh:5.0.1 cassandra: 3.0.9 CQL Spec: 3.4.0, Replication factor = 2, There are many updates ~ 1 TB – Ekrem Gurdal Mar 22 '18 at 10:40
  • 1
    Well, if there's 1TB of in-place updates which result in 100GB of actual data, then its entirely possible that y.z is fine. The real question, is can your application query at ALL or TWO without issue? – Aaron Mar 22 '18 at 11:50
  • I can fully agree with @Aron - I guess you mostly have updates in your sstables on node2 which are not compacted yet. – Mandraenke Mar 22 '18 at 14:43
  • My application cannot query at two nodes. It randomly selects one node and queries. If data is not there, it stops. There should be exact same "Load" after repair? @Mandraenke – Ekrem Gurdal Mar 23 '18 at 06:55

1 Answers1

1

We had a solution but don't know how it happened. Here what we did:

  • Took snapshot of both servers in case of data loss over AWS
  • We detached Volume where data is stored. (for instance 172.x.y.k)
  • We formatted instance and loaded Cassandra with new version. Then we attached Volume where data is stored.
  • Finally we started nodetool repair --full in terminal and it took 4 days.

Now our nodes are equal.

Ekrem Gurdal
  • 1,118
  • 13
  • 14