1

Problem

I have a test setup running Couchbase Server 3.0.3 with one node on a remote server (Windows Server 2012), and one node on my local machine (Windows 8.1). Last time I checked, which was a few days ago, all nodes were up and running with all buckets healthy.

Today I noticed the node on my local machine in the "Pending" state. I first ignored because it is my local machine and figured it was probably something with the firewall or port forwarding not setup correctly.

I created a new bucket and tested it in my application (.net), adding a document to it. This failed with an error "Temporary failure".

When checking the Couchbase webadmin I noticed the newly created bucket had a state "unhealthy". After I searched online for a bit and went back to the webadmin suddenly all buckets were "unhealthy" and both nodes were "Pending".

couchbase node overview

There is nothing in the log during the time I created the new bucket and when I noticed both nodes were "Pending".

This is the log of the bucket creation:

couchbase logs

What I tried so far

I tried removing one node but rebalancing doesn't work because I guess there is no active node left:

Rebalance exited with reason {badmatch,
{error,
{failed_nodes,
['ns_1@couchbase.xxxxxxxxx.com']}}}

I restarted my local machine and the remote server.

I restarted the CouchbaseServer service.

Deleted some other buckets I didn't use anymore just in case (default & samples).

I removed the bucket I had created and tried the same steps.

There was a full disk warning so I freed up some disk space.

I'm not sure how to proceed now, so any help is appreciated.

redrobot
  • 133
  • 1
  • 2
  • 8
  • Since you are running the enterprise edition, this means that either you are a customer of Couchbase or a prospective customer. You might want to contact your account representative and someone from Couchbase can help you. That being said, the set up where you span a cluster over remote servers like this is a bad idea. I would start from the ground up and confirm all networking ports are open and communication is flowing between the nodes. Then step up to see the number of vBuckets are 1024. Then start digging more through logs. – Kirk Jun 17 '15 at 17:02
  • 2
    As far as I know you are allowed to freely run the Enterprise version on up to 2 nodes: http://stackoverflow.com/questions/12685801 . The problem was in fact caused by not having all ports forwarded correctly since my local IP had changed, but the first node only became unavailable ("pending") until after I created a new bucket. – redrobot Jun 18 '15 at 07:19

0 Answers0