glassfish 4.1.1 cluster ssh node error

Question

I`m trying to create a cluster with two nodes using glassfish 4.1.1 build 1.
One node is local, an the other one is ssh.The node is working as if I ping it it responds ok. ( Successfully made SSH connection to node node2 (gfNode2)) I have setup ssh, create the node, create one instance (i2) on that node but when i want to start the instance i get:

i2: Could not start instance i2 on node node2 (gfNode2). Command failed on node node2 (gfNode2): Previous synchronization failed at Sep 10, 2016 12:25:27 PM Will perform full synchronization. Removing all cached state for instance i2. CLI802 Synchronization failed for directory config, caused by: remote failure: SynchronizeFiles: Exception reading request Command start-local-instance failed. To complete this operation run the following command locally on host gfNode2 from the GlassFish install location /opt/glassfish4: lib/nadmin start-local-instance --node node2 --sync normal i2

if i run this command on node 2 machine i get:

 ./nadmin  start-local-instance --node node2 --sync normal i2
Previous synchronization failed at Sep 10, 2016 12:25:27 PM
Will perform full synchronization.
Removing all cached state for instance i2.
Enter admin user name>  admin
Enter admin password for user "admin">
CLI802 Synchronization failed for directory config, caused by:
remote failure: SynchronizeFiles: Exception reading request
Command start-local-instance failed.

any idea what to try next ?

Update:
The DAS is reachabe, the ssh is working properly (ping-node-ssh works from das).
What I have noticed is that even after i have installed (install-node-ssh) and create with(create-node-ssh), node 2 has no files inside.
At /glassfish4/glassfish/nodes/node2/i2 there is only one file: .syncstate which is empty. The node2/i2 directories are there but nothing in i2. Maybe due to : Removing all cached state for instance i2.

That is what i got in DAS logs:

    [2016-09-10T19:31:14.806+0000] [glassfish 4.1] [WARNING] []           [javax.enterprise.system.core] [tid: _ThreadID=106 _ThreadName=admin-   listener(5)] [timeMillis: 1473535874806] [levelValue: 900] [[
  Could not start instance i2 on node node2 (gfNode2).: Command ' /opt/glassfish4/glassfish/lib/nadmin --_auxinput - --interactive=false start-local-instance --node node2 --sync normal i2' failed on node node2 (gfNode2): Previous synchronization failed at Sep 10, 2016 12:25:27 PM
Will perform full synchronization.
Removing all cached state for instance i2.
Command start-local-instance failed.
CLI802 Synchronization failed for directory config, caused by:
  remote failure: SynchronizeFiles: Exception reading request: To complete this operation run the following command locally on host gfNode2 from the GlassFish install location /opt/glassfish4:

 lib/nadmin  start-local-instance --node node2 --sync normal i2]]

[2016-09-10T19:31:14.818+0000] [glassfish 4.1] [SEVERE] [] [org.glassfish.admingui] [tid: _ThreadID=102 _ThreadName=admin-listener(1)] [timeMillis: 1473535874818] [levelValue: 1000] [[
  RestResponse.getResponse() gives FAILURE.  endpoint = 'https://localhost:4848/management/domain/servers/server/i2/start-instance'; attrs = '{}']]

[2016-09-10T19:31:14.820+0000] [glassfish 4.1] [SEVERE] [] [org.glassfish.admingui] [tid: _ThreadID=102 _ThreadName=admin-listener(1)] [timeMillis: 1473535874820] [levelValue: 1000] [[
  Error in instanceAction ;
endpoint=https://localhost:4848/management/domain/servers/server/i2/start-instance;attrsMap=null]]

If I try to run the command from node2 I got what is showed on first code block of the post...

This is pretty weird. If you can install GlassFish on the remote node, it doesn't make sense that synchronisation is failing like that...it looks like the remote files were never created when the instance was created. A sync=normal wouldn't affect them if the DAS was unreachable, but a sync=full would wipe them, ready to start again. It's interesting that the last note in the log says it's going to `localhost` to run the command. If the remote instance is trying to contact the DAS by going to localhost instead of the proper hostname, that would explain it — Mike, Sep 10 '16 at 19:49
I think that the wiped files are as a result of me running a sync full, at some moment. I have noticed also that localhost but I do not know why it invoke it like this.. I`ll keep searching — Videanu Adrian, Sep 10 '16 at 19:52
found the problem! The issue was that my host file on the second machine (node2) was configured wrong, and by that I mean that both gfNode1 and gfNode2 had the same IP, and when the instance tried to sync it was going to itself. The error was due to a mistaken copy/paste :( — Videanu Adrian, Sep 10 '16 at 20:55

score 0 · Answer 1 · answered Sep 10 '16 at 16:01

The problem here is that the remote instance i2 can't communicate with the DAS to download its configuration.

You will need to verify:

Is the DAS online?
Is the server where the DAS is reachable by the remote node?
Is SSH communication working properly? (use the asadmin command ping-node-ssh

If you open the server.log file for the instance and on the DAS, that should give you a more detailed error message and indicate whether or not the request is reaching the DAS.

The instance logs are located in:

$GLASSFISH_HOME/glassfish/nodes/node2/i2/logs/server.log

The domain logs are located in:

$GLASSFISH_HOME/glassfish/domains/domain1/logs/server.log

glassfish 4.1.1 cluster ssh node error

1 Answers1