0

I had added 3 single token nodes to our existing DataCenter as specified in datastax document : http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddRplSingleTokenNodes.html

Currently all 3 new nodes are showing joining state.

Datacenter: SearchGraph
==========
Address     Rack        Status State   Load       Owns      Token                                       
10.10.1.46  rack1       Up     Normal  381.13 MiB      ?   -9223372036854775808                        
10.10.2.66  rack2       Up     Joining 277.94 MiB      ?   -6148914691236517206                        
10.10.1.36  rack1       Up     Normal  414.47 MiB      ?   -3074457345618258603                        
10.10.2.76  rack2       Up     Joining 251.52 MiB      ?   -2                                          
10.10.1.56  rack1       Up     Normal  409.32 MiB      ?    3074457345618258602                         
10.10.2.86  rack2       Up     Joining 332.89 MiB      ?    6148914691236517202

But we are getting following error in joining nodes.

AbstractSolrSecondaryIndex.java:1884 - Cannot find core chat.chat_history
AbstractSolrSecondaryIndex.java:1884 - Cannot find core chat.history
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.business_units
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.feeds
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.feeds_2
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.knowledegmodule
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.userdetails
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.userdetails_2
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.vault_details
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.workgroup
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.feeds
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.knowledgemodule
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.organizations
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.userdetails
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.vaults
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.workgroup

Post Updated :

Node joining failed with following error :

ERROR [main] 2017-08-10 04:22:08,449  DseDaemon.java:488 - Unable to start DSE server.
com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.plugin.SolrContainerPlugin


Caused by: java.lang.IllegalStateException: Cannot find secondary index for core ekamsearch.userdetails_2, did you create it? 
If yes, please consider increasing the value of the dse.yaml option load_max_time_per_core, current value in minutes is: 10

ERROR [main] 2017-08-10 04:22:08,450  CassandraDaemon.java:705 - Exception encountered during startup
java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin

Has anyone encountered these errors or warnings before?

Sreeraju V
  • 535
  • 2
  • 5
  • 19
  • Tried to increase value of load_max_time_per_core in dse.yaml still we are getting error – Sreeraju V Aug 10 '17 at 11:11
  • I'm pretty sure this is because you added the nodes but you have might not have loaded the Solr cores? what version of DSE are you running here? – markc Aug 10 '17 at 14:47
  • @markc Current DSE version is 5.1.2 .Can you please mention how to load solr core in new joining nodes. – Sreeraju V Aug 10 '17 at 15:29
  • @SreerajuV I would point you to this doc http://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/search/managingNodesSearch.html?hl=adding%2Cdecommissioning%2Crepairing%2Cdse%2Csearch%2Cnode – mando222 Aug 10 '17 at 16:34
  • @mando222 Thanks for pointing me to datastax doc.I followed the steps mentioned in the specified doc :http://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/operations/opsAddRplSingleTokenNodes.html . Is there any extra process need to be done for loading solr core in new nodes. We have enabled solr and graph in our cluster. Data is moving to new joining nodes, but we are facing error in above mentioned core – Sreeraju V Aug 10 '17 at 16:58
  • @mando222 Error show it cannot find solr core while migration and Cannot find secondary index for the core – Sreeraju V Aug 12 '17 at 05:00
  • @SreerajuV to create a solr core you can use `dsetool` commands like so: https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/dsetool.html#dsetool__create_core – markc Aug 15 '17 at 12:15
  • @markc I have created solr core and indexed data existing in old node, While new node joining time, isn't it normal process the solr core data migrated to new nodes ? . Did i need to reload the core in new nodes one by one while joining time to core get replicated in new node ? Thanks – Sreeraju V Aug 15 '17 at 14:38
  • 1
    @SreerajuV So when you create a solr core you need to create it on all DCs but only in one node. You might stream over the data from other DCs but you'll need to use `dsetool` to create the core in the new DC. Have you created the solr core in the new DC now? – markc Aug 16 '17 at 08:08

1 Answers1

0

Issue resolved : Increased load_max_time_per_core value in dse.yaml configuration file, still i was receving the error.Finalys solved the issue by following method

     1) Started the new nodes as non-solr and wait for all cassandra data  
        to migrate to joining nodes.
     2) Add the parameter auto_bootstrap: False directive to the 
        cassandra.yaml file
     3) Re-start the same nodes after enabling solr. Changed parameter 
        SOLR_ENABLED=1 in /etc/default/dse
     3) Re-index in all new joined nodes. I had to reloaded all core 
        required with the reindex=true and distributed=false parameters in 
        new  joined nodes. 
        Ref : http://docs.datastax.com/en/archived/datastax_enterprise/4.0/datastax_enterprise/srch/srchReldCore.html
Sreeraju V
  • 535
  • 2
  • 5
  • 19