2

I have Cassandra cluster on AWS. I have it only for testing purposes. Because of that I need to stop my instances and start them some other time. I deployed cluster with Datastax AMI and everything seems alright. I stoped instances in EC2 Management console and waited for 'stoped' state. After that I started them. I connected to them and Cassandra did not work. Command nodetool could not reach localhost:7199. Security groups on AWS allow all inbound and outbound traffic. I tried to sudo service cassandra start, but I got the same error.

I need start all nodes of cluster without manual action on them.

32cupo
  • 850
  • 5
  • 18
  • 36
  • 4
    Please show the exact error from your output.log – phact Mar 09 '15 at 21:28
  • When I use command `nodetool stats` I got error `nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.` It is because of disabled service cassandra. When I tried to start service, nothing happened. No error. Problem is how to start cassandra properly after stopping and starting instance. – 32cupo Mar 09 '15 at 22:00
  • @phact is asking about your "output.log" typically found in "/var/log/cassandra/". This will probably show the error causing the service to not startup correctly. – BeepBoop Mar 09 '15 at 22:16
  • and you should do that via API or ec2 python or other script to avoid manual task. – BMW Mar 10 '15 at 03:04
  • Oh sorry, I found this log file but it is symbolic link pointing on /raid0/ and as @RomanTumaykin mentioned in answer below /raid0/ was deleted while stopping of instance. Because of that I have no output.log – 32cupo Mar 10 '15 at 20:25

1 Answers1

2

Ah. You stopped instances from the EC2 management console. This erased everything from the ephemeral storage. You should only restart instances if necessary, but never stop them. Or use sudo service dse restart to restart the dse service only without restarting the entire instance. Now I'd suggest that your best bet is to redeploy the cluster (or at least this node) from scratch....

Here is some explanation of what the instance store (ephemeral store) is: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html

The data in an instance store persists only during the lifetime of its associated instance. If an instance reboots (intentionally or unintentionally), data in the instance store persists. However, data on instance store volumes is lost under the following circumstances:

Failure of an underlying drive

Stopping an Amazon EBS-backed instance

Terminating an instance

Datastax uses the ephemeral storage to store data and commit logs, because this storage is much faster than the ebs storage. More details here: http://datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installAMI.html

Roman Tumaykin
  • 1,921
  • 11
  • 11
  • Thanks, is there any options how to deploy Cassandra cluster on EBS volumes? Have you any experience with manual deploying of cluster without datastax AMI? – 32cupo Mar 10 '15 at 20:30
  • 1
    Of course. Here is the documentation. http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installTOC.html – Roman Tumaykin Mar 10 '15 at 21:25