0

I am running a Cassandra cluster in Docker containers using fleet for management. I am able to get the cluster up and running, but if I bring the units down with fleet and then back up again, the containers fail. The Cassandra logs has this entry on the second start.

Cannot add table 'role_members' to non existing keyspace 'system_auth'.
Fatal configuration error; unable to start server.  See log for stacktrace.
INFO  20:59:34 InetAddress /172.17.8.102 is now DOWN
ERROR 20:59:34 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot add table 'role_members' to non existing keyspace 'system_auth'.
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:284) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:275) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.maybeAddTable(StorageService.java:1046) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1034) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:967) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:698) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) ~[apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:291) [apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:481) [apache-cassandra-2.2.0.jar:2.2.0]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:588) [apache-cassandra-2.2.0.jar:2.2.0]

I can't find any information on this particular error, and I really have no idea why it's happening. The closest information I can find is that the system_auth table needs to be configured specially if you are not using the default AllowAllAuthenticator, but I am using this. I haven't changed it in the cassandra.yaml file.

Does anyone know why this might be happening?

Community
  • 1
  • 1
Kris Harper
  • 5,672
  • 8
  • 51
  • 96

3 Answers3

0

Is it possible that you are using CassandraAuthorizer without using PasswordAuthenticator? I think that might not work and cause this particular error.

system_auth is not applicable to AllowAllAuthenticator, you need to use PasswordAuthenticator instead. If you configure cassandra.yaml in the following way:

authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer

And then restart cassandra, it should create the system_auth keyspace for you. If you don't want to set up authorization, you can always use AllowAllAuthorizer instead. More information can be found here.

Andy Tolbert
  • 11,418
  • 1
  • 30
  • 45
  • My cassandra.yaml file has `authenticator: AllowAllAuthenticator` `authorizer: AllowAllAuthorizer`. I think these are the defaults. I haven't changed them during the setup. I do think it's odd though, because other people have mentioned that `system_auth` shouldn't even be created with these settings. Could it be that they are automatically set somewhere else? – Kris Harper Sep 10 '15 at 22:03
  • Interesting, I agree, I wouldn't expect a system_auth table to be present unless you are explicitly setting those. What version of cassandra? – Andy Tolbert Sep 10 '15 at 22:27
  • Version is 2.2.0. I could try updating to 2.2.1 I guess. – Kris Harper Sep 10 '15 at 22:31
  • The system_auth keyspace won't be created when using AllowAllAuthorizer and AllowAllAuthenticator. The question is why Cassandra is trying to create the role_members table given your config. Have you changed anything in these files or is it straight out of the box? – Alec Collier Sep 11 '15 at 04:52
  • I have not changed the cassandra.yaml file directly. I have set a few cassandra environment variables in my dockerfile, which Cassandra then uses to update that file. The variables I have set are `ENV CASSANDRA_SEEDS`, `ENV CASSANDRA_CLUSTER_NAME`, `ENV CASSANDRA_DC`, `ENV CASSANDRA_RACK`, and `ENV CASSANDRA_ENDPOINT_SNITCH`. – Kris Harper Sep 11 '15 at 13:12
0

This turned out to be a rather unique configuration issue I had. I was mapping /var/lib/cassandra on the host to /var/lib/cassandra inside my docker container. But I was also inadvertently mapping /var/lib/cassandra/data to an auto-generated Docker directory on the host. As such when I stopped and restarted the containers, the data directory would disappear and Cassandra would fail as it tried to recreate data from the commitlog directory.

Kris Harper
  • 5,672
  • 8
  • 51
  • 96
0

I got the problem just following the Datastax "Initializing a multiple node cluster (single data center)" tutorial.

I solved the same problem deleting the whole content of /var/lib/cassandra and not only the content of /var/lib/cassandra/system/

Why? I think Kris got the real problem source: when restarting, the C* service found the commitLog full and recovered by trying to reconstruct the commits found there, failing due to a different configuration and a different table structure...

Vinz
  • 1