1

I am setting up a cluster. I tried to join 3 nodes but while re-balancing. I got below error. So i extracted some info from debug.log and unable to identify the exact issue. Appreciate any help.

=========================CRASH REPORT=========================
  crasher:
    initial call: service_agent:-spawn_connection_waiter/2-fun-0-/0
    pid: <0.18486.7>
    registered_name: []
    exception exit: {no_connection,"index-service_api"}
      in function  service_agent:wait_for_connection_loop/3 (src/service_agent.erl, line 305)
    ancestors: ['service_agent-index',service_agent_children_sup,
                  service_agent_sup,ns_server_sup,ns_server_nodes_sup,
                  <0.170.0>,ns_server_cluster_sup,<0.89.0>]
    messages: []
    links: [<0.18481.7>,<0.18490.7>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 1195
  neighbours:

[ns_server:error,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:service_agent-index<0.18481.7>:service_agent:terminate:264]Terminating abnormally
[ns_server:debug,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:<0.18487.7>:ns_pubsub:do_subscribe_link:145]Parent process of subscription {ns_config_events,<0.18481.7>} exited with reason {linked_process_died,
                                                                                  <0.18486.7>,
                                                                                  {no_connection,
                                                                                   "index-service_api"}}
[error_logger:error,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server 'service_agent-index' terminating
** Last message in was {'EXIT',<0.18486.7>,
                               {no_connection,"index-service_api"}}
** When Server state == {state,index,
                         {dict,6,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[[{uuid,<<"55a14ec6b06d72205b3cd956e6de60e7">>}|
                              'ns_1@xuodf7.firebrand.com']],
                            [],
                            [[{uuid,<<"c5e67322a74826bef8edf27d51de3257">>}|
                              'ns_1@xuodf8.firebrand.com']],
                            [],
                            [[{uuid,<<"3b55f7739e3fe85127dcf857a5819bdf">>}|
                              'ns_1@xuodf9.firebrand.com']],
                            [],
                            [[{node,'ns_1@xuodf7.firebrand.com'}|
                              <<"55a14ec6b06d72205b3cd956e6de60e7">>],
                             [{node,'ns_1@xuodf8.firebrand.com'}|
                              <<"c5e67322a74826bef8edf27d51de3257">>],
                             [{node,'ns_1@xuodf9.firebrand.com'}|
                              <<"3b55f7739e3fe85127dcf857a5819bdf">>]],
                            [],[],[],[],[],[],[],[],[]}}},
                         undefined,undefined,<0.18626.7>,#Ref<0.0.5.56873>,
                         <0.18639.7>,
                         {[{<0.18646.7>,#Ref<0.0.5.56891>}],[]},
                         undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {linked_process_died,<0.18486.7>,{no_connection,"index-service_api"}}

[error_logger:error,2018-02-12T13:54:43.532-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: service_agent:init/1
    pid: <0.18481.7>
    registered_name: 'service_agent-index'
    exception exit: {linked_process_died,<0.18486.7>,
                        {no_connection,"index-service_api"}}
      in function  gen_server:terminate/6 (gen_server.erl, line 744)
    ancestors: [service_agent_children_sup,service_agent_sup,ns_server_sup,
                  ns_server_nodes_sup,<0.170.0>,ns_server_cluster_sup,
                  <0.89.0>]
    messages: [{'EXIT',<0.18639.7>,
                      {linked_process_died,<0.18486.7>,
                          {no_connection,"index-service_api"}}}]
    links: [<0.18487.7>,<0.4805.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 28690
    stack_size: 27
    reductions: 6334
  neighbours:

[error_logger:error,2018-02-12T13:54:43.533-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================SUPERVISOR REPORT=========================
     Supervisor: {local,service_agent_children_sup}
     Context:    child_terminated
     Reason:     {linked_process_died,<0.18486.7>,
                                      {no_connection,"index-service_api"}}
     Offender:   [{pid,<0.18481.7>},
                  {name,{service_agent,index}},
                  {mfargs,{service_agent,start_link,[index]}},
                  {restart_type,permanent},
                  {shutdown,1000},
                  {child_type,worker}]


[ns_server:error,2018-02-12T13:54:43.533-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index<0.18626.7>:service_rebalancer:run_rebalance:80]Agent terminated during the rebalance: {'DOWN',#Ref<0.0.5.56860>,process,
                                        <0.18481.7>,
                                        {linked_process_died,<0.18486.7>,
                                         {no_connection,"index-service_api"}}}
[error_logger:info,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
          supervisor: {local,service_agent_children_sup}
             started: [{pid,<0.20369.7>},
                       {name,{service_agent,index}},
                       {mfargs,{service_agent,start_link,[index]}},
                       {restart_type,permanent},
                       {shutdown,1000},
                       {child_type,worker}]

[ns_server:error,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_agent-index<0.20369.7>:service_agent:handle_call:186]Got rebalance-only call {if_rebalance,<0.18626.7>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index<0.18626.7>:service_agent:process_bad_results:815]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1@xuodf9.firebrand.com',nack}]
[ns_server:warn,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index<0.18626.7>:service_rebalancer:run_rebalance:89]Failed to unset rebalancer on some nodes:
{error,{bad_nodes,index,unset_rebalancer,
                  [{'ns_1@xuodf9.firebrand.com',nack}]}}
[error_logger:error,2018-02-12T13:54:43.535-05:00,ns_1@xuodf9.firebrand.com:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: service_rebalancer:-spawn_monitor/6-fun-0-/0
    pid: <0.18626.7>
    registered_name: 'service_rebalancer-index'
    exception exit: {linked_process_died,<0.18486.7>,
                        {no_connection,"index-service_api"}}
      in function  service_rebalancer:run_rebalance/7 (src/service_rebalancer.erl, line 92)
    ancestors: [cleanup_process,ns_janitor_server,ns_orchestrator_child_sup,
                  ns_orchestrator_sup,mb_master_sup,mb_master,<0.4893.0>,
                  ns_server_sup,ns_server_nodes_sup,<0.170.0>,
                  ns_server_cluster_sup,<0.89.0>]
    messages: [{'EXIT',<0.18640.7>,
                      {linked_process_died,<0.18486.7>,
                          {no_connection,"index-service_api"}}}]
    links: []
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 6359
  neighbours:

[ns_server:error,2018-02-12T13:54:43.536-05:00,ns_1@xuodf9.firebrand.com:cleanup_process<0.18625.7>:service_janitor:maybe_init_topology_aware_service:84]Initial rebalance for `index` failed: {error,
                                       {initial_rebalance_failed,index,
                                        {linked_process_died,<0.18486.7>,
                                         {no_connection,
                                          "index-service_api"}}}}
[ns_server:debug,2018-02-12T13:54:43.536-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {"projector-cbauth",<0.5099.0>} needs_update
[ns_server:debug,2018-02-12T13:54:43.538-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {"goxdcr-cbauth",<0.479.0>} needs_update
[ns_server:debug,2018-02-12T13:54:43.539-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {"cbq-engine-cbauth",<0.5124.0>} needs_update
[ns_server:debug,2018-02-12T13:54:43.540-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth<0.4796.0>:menelaus_cbauth:handle_cast:95]Observed json rpc process {"fts-cbauth",<0.5129.0>} needs_update

This is a blocker for cluster creation at this point.

slavoo
  • 5,798
  • 64
  • 37
  • 39
Mari U
  • 21
  • 2
  • What version of Couchbase Server? How are you joining the nodes to a cluster (CLI, UI, REST, etc)? – Matthew Groves Feb 12 '18 at 20:21
  • I using couchbase Community Edition 5.0.1 build 5003. I am joining / running rebalance using web UI. – Mari U Feb 12 '18 at 20:48
  • Could you provide some more details like: which operating system are you using? Are you running these in containers? On VMs? On separate machines? Have you opened up all the necessary ports? – Matthew Groves Feb 13 '18 at 13:46

1 Answers1

0

The rebalance error is coming due to index service. You can check indexer.log to see if there are any errors and the process is able to bootstrap correctly.

Please make sure the communication ports are open as mentioned here: https://developer.couchbase.com/documentation/server/current/install/install-ports.html

projector_port 9999 being blocked can lead to this.

deepkaran
  • 71
  • 4