I'm trying to setup OrientDB in distributed mode on AWS, behind a ELB, in an ASG.
So far so good, but the initial DB replication have only happened in one out of fifteen tries! That is, after a new node have joined the cluster, only ONE DB is synchronized. And that only once.
On top of that, the master is unresponsive! I.e., OrientDB wouldn't answer a health check from the ELB, and had I not taken the instance out of the ELB (while I'm testing this), it would have been destroyed.
What I do and want
I have a(t least one) instance, where I load my database. I then start set the ASG to have +1 instance, which is then started. I want this to join the first one (this part works just fine - now) and get the DB from the first one. Additional instances should work the same - join the cluster, get the DB.
So my two questions:
How do I initiate/start the replication? As in, make it synchronize ALL of the DBs? I can't, for several reasons, use "scp" (et al) to copy the database(s)!
How do I configure OrientDB to NOT reject incoming queries while it syncs the DB(s)?
I've tried 2.1.7, 2.1.15 and 2.2.0-beta2 and in all three cases I had to upgrade the hazelcast-all.jar file from Hazelcast.com (v3.6.2) for the auto-discovery to work in the first place.
With the new jar file, "it just worked"! As in, the new instance joined the cluster as soon as it was started. Although, it wouldn't get the new database :(. Well, in some cases it would get only the first, sometimes only PART of the first one, but mostly nothing.
My files:
default-distributed-db-config.json
{
"replication": true,
"autoDeploy": true,
"hotAlignment": true,
"resyncEvery": 15,
"executionMode": "synchronous",
"readQuorum": 1,
"writeQuorum": 2,
"failureAvailableNodesLessQuorum": false,
"readYourWrites": true,
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"index": {
},
"*": {
"servers": ["<NEW_NODE>"]
}
}
}
hazelcast.xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- !!! This file is autogenerated by Saltstack. All changes will be overridden -->
<!-- ~ Copyright (c) 2008-2012, Hazel Bilisim Ltd. All Rights Reserved. ~
~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may
not use this file except in compliance with the License. ~ You may obtain
a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~
~ Unless required by applicable law or agreed to in writing, software ~ distributed
under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES
OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for
the specific language governing permissions and ~ limitations under the License. -->
<hazelcast
xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.0.xsd"
xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<properties>
<property name="hazelcast.icmp.enabled">true</property>
</properties>
<group>
<name>orientdb</name>
<password>sEcrEt</password>
</group>
<network>
<port auto-increment="true">2434</port>
<join>
<multicast enabled="false">
<multicast-group>235.1.1.1</multicast-group>
<multicast-port>2434</multicast-port>
</multicast>
<tcp-ip enabled="false">
<member>10.129.1.129:2434</member>
</tcp-ip>
<aws enabled="true">
<access-key>acCEss</access-key>
<secret-key>seCRet</secret-key>
<host-header>ec2.amazonaws.com</host-header>
<region>eu-west-1</region>
<!--
http://grepcode.com/file/repo1.maven.org/maven2/com.hazelcast/hazelcast-all/3.1.5/com/hazelcast/cluster/TcpIpJoinerOverAWS.java
There are 2 mechanisms for filtering out AWS instances and
these mechanisms can be combined (AND).
1. If a securityGroup is configured only instanced within
that security group are selected.
2. If a tag key/value is set only instances with that tag
key/value will be selected.
Once Hazelcast has figured out which instances are available,
it will use the private ip addresses of these instances to
create a tcp/ip-cluster.
<security-group-name>sg-orientdb</security-group-name>
<tag-key>aws:autoscaling:groupName</tag-key>
<tag-value>orientdb-orientdb</tag-value>
-->
</aws>
</join>
<interfaces enabled="true">
<interface>10.129.1.129</interface>
</interfaces>
</network>
<executor-service>
<pool-size>16</pool-size>
</executor-service>
</hazelcast>
orientdb-server-config.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<orient-server>
<handlers>
<handler class="com.orientechnologies.orient.graph.handler.OGraphServerHandler">
<parameters>
<parameter value="true" name="enabled"/>
<parameter value="50" name="graph.pool.max"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin">
<parameters>
<parameter value="true" name="enabled"/>
<parameter value="orientdb-i-8d93ec05" name="nodeName"/>
<parameter value="${ORIENTDB_HOME}/config/default-distributed-db-config.json" name="configuration.db.default"/>
<parameter value="${ORIENTDB_HOME}/config/hazelcast.xml" name="configuration.hazelcast"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.handler.OJMXPlugin">
<parameters>
<parameter value="false" name="enabled"/>
<parameter value="true" name="profilerManaged"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.handler.OAutomaticBackup">
<parameters>
<parameter value="false" name="enabled"/>
<parameter value="4h" name="delay"/>
<parameter value="23:00:00" name="firstTime"/>
<parameter value="backup" name="target.directory"/>
<parameter value="${DBNAME}-${DATE:yyyyMMddHHmmss}.zip" name="target.fileName"/>
<parameter value="9" name="compressionLevel"/>
<parameter value="1048576" name="bufferSize"/>
<parameter value="" name="db.include"/>
<parameter value="GratefulDeadConcerts" name="db.exclude"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.handler.OServerSideScriptInterpreter">
<parameters>
<parameter value="true" name="enabled"/>
<parameter value="SQL" name="allowedLanguages"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.token.OrientTokenHandler">
<parameters>
<parameter value="false" name="enabled"/>
<parameter value="" name="oAuth2Key"/>
<parameter value="60" name="sessionLength"/>
<parameter value="HmacSHA256" name="encryptionAlgorithm"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.plugin.livequery.OLiveQueryPlugin">
<parameters>
<parameter value="false" name="enabled"/>
</parameters>
</handler>
</handlers>
<network>
<sockets>
<socket implementation="com.orientechnologies.orient.server.network.OServerSSLSocketFactory" name="ssl">
<parameters>
<parameter value="false" name="network.ssl.clientAuth"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.keyStore"/>
<parameter value="password" name="network.ssl.keyStorePassword"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.trustStore"/>
<parameter value="password" name="network.ssl.trustStorePassword"/>
</parameters>
</socket>
<socket implementation="com.orientechnologies.orient.server.network.OServerSSLSocketFactory" name="https">
<parameters>
<parameter value="false" name="network.ssl.clientAuth"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.keyStore"/>
<parameter value="password" name="network.ssl.keyStorePassword"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.trustStore"/>
<parameter value="password" name="network.ssl.trustStorePassword"/>
</parameters>
</socket>
</sockets>
<protocols>
<protocol implementation="com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary" name="binary"/>
<protocol implementation="com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpDb" name="http"/>
</protocols>
<listeners>
<listener protocol="binary" socket="default" port-range="2424-2430" ip-address="0.0.0.0"/>
<listener protocol="http" socket="default" port-range="2480-2490" ip-address="0.0.0.0">
<commands>
<command implementation="com.orientechnologies.orient.server.network.protocol.http.command.get.OServerCommandGetStaticContent" pattern="GET|www GET|studio/ GET| GET|*.htm GET|*.html G\
ET|*.xml GET|*.jpeg GET|*.jpg GET|*.png GET|*.gif GET|*.js GET|*.css GET|*.swf GET|*.ico GET|*.txt GET|*.otf GET|*.pjs GET|*.svg GET|*.json GET|*.woff GET|*.woff2 GET|*.ttf GET|*.svgz" stateful="false">
<parameters>
<entry value="Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nPragma: no-cache" name="http.cache:*.htm *.html"/>
<entry value="Cache-Control: max-age=120" name="http.cache:default"/>
</parameters>
</command>
<command implementation="com.orientechnologies.orient.graph.server.command.OServerCommandGetGephi" pattern="GET|gephi/*" stateful="false"/>
</commands>
<parameters>
<parameter value="utf-8" name="network.http.charset"/>
<parameter value="true" name="network.http.jsonResponseError"/>
</parameters>
</listener>
</listeners>
</network>
<storages/>
<users>
<user resources="*" password="secret" name="root"/>
<user resources="connect,server.listDatabases,server.dblist" password="SecRet" name="guest"/>
<user resources="database.passthrough" password="SeCrEt" name="replicator"/>
</users>
<properties>
<entry value="1" name="db.pool.min"/>
<entry value="50" name="db.pool.max"/>
<entry value="true" name="profiler.enabled"/>
<entry value="info" name="log.console.level"/>
<entry value="fine" name="log.file.level"/>
</properties>
</orient-server>
Once adding a new node, I get this on the "master" (10.129.1.48):
2016-04-12 10:41:38:483 INFO [10.129.1.48]:2434 [orientdb] [3.6.2] Accepting socket connection from /10.129.1.129:36103 [SocketAcceptorThread]
2016-04-12 10:41:38:485 INFO [10.129.1.48]:2434 [orientdb] [3.6.2] Established socket connection between /10.129.1.48:2434 and /10.129.1.129:36103 [TcpIpConnectionManager]
2016-04-12 10:41:45:505 INFO [10.129.1.48]:2434 [orientdb] [3.6.2]
Members [3] {
Member [10.129.1.219]:2434
Member [10.129.1.48]:2434 this
Member [10.129.1.129]:2434
}
[ClusterService]
2016-04-12 10:41:45:505 SEVER [orientdb-i-346e11bc] Cannot find node with id '3082da3b-c329-49f3-8561-2053ac5bbe21' [OHazelcastPlugin]
2016-04-12 10:41:45:505 WARNI [orientdb-i-346e11bc] added new node id=Member [10.129.1.129]:2434 name=ext:3082da3b-c329-49f3-8561-2053ac5bbe21 [OHazelcastPlugin]
2016-04-12 10:41:47:558 INFO [orientdb-i-346e11bc]<-[orientdb-i-8d93ec05] added node configuration id=Member [10.129.1.129]:2434 name=orientdb-i-8d93ec05, now 3 nodes are configured [OHazelcastPlugin]
2016-04-12 10:41:47:569 INFO [orientdb-i-346e11bc] Current node started as MASTER for database 'db_1' [OHazelcastPlugin]
2016-04-12 10:41:47:569 INFO [orientdb-i-346e11bc] Current node started as MASTER for database 'db_2' [OHazelcastPlugin]
And on the new node:
2016-04-12 10:41:41:267 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Creating AWSJoiner [Node]
2016-04-12 10:41:41:272 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.129]:2434 is STARTING [LifecycleService]
2016-04-12 10:41:41:359 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads [NonBlockingIOThreadingModel]
2016-04-12 10:41:41:855 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.129:2435, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:857 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.1.129:2435. Reason: SocketException[Connection refused to address /10.129.1.129:2435] [InitConnectionTask]
2016-04-12 10:41:41:857 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.129]:2435 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:858 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.129:2436, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:866 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.219:2434, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:868 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.1.129:2436. Reason: SocketException[Connection refused to address /10.129.1.129:2436] [InitConnectionTask]
2016-04-12 10:41:41:880 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.219:2435, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:881 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.0.14:2434, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:881 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Established socket connection between /10.129.1.129:59102 and /10.129.1.219:2434 [TcpIpConnectionManager]
2016-04-12 10:41:41:882 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.219:2436, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:883 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.129]:2436 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:885 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.0.14:2434. Reason: SocketException[Connection refused to address /10.129.0.14:2434] [InitConnectionTask]
2016-04-12 10:41:41:885 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.0.14]:2434 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:887 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.1.219:2436. Reason: SocketException[Connection refused to address /10.129.1.219:2436] [InitConnectionTask]
2016-04-12 10:41:41:887 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.219]:2436 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:890 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.1.219:2435. Reason: SocketException[Connection refused to address /10.129.1.219:2435] [InitConnectionTask]
2016-04-12 10:41:41:890 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.219]:2435 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:892 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.48:2434, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:893 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Established socket connection between /10.129.1.129:36103 and /10.129.1.48:2434 [TcpIpConnectionManager]
2016-04-12 10:41:41:896 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.48:2435, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:897 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.1.48:2435. Reason: SocketException[Connection refused to address /10.129.1.48:2435] [InitConnectionTask]
2016-04-12 10:41:41:897 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.48]:2435 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:899 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.0.14:2436, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:901 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.1.48:2436, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:902 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.1.48:2436. Reason: SocketException[Connection refused to address /10.129.1.48:2436] [InitConnectionTask]
2016-04-12 10:41:41:902 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.48]:2436 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:906 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Connecting to /10.129.0.14:2435, timeout: 0, bind-any: true [InitConnectionTask]
2016-04-12 10:41:41:906 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.0.14:2435. Reason: SocketException[Connection refused to address /10.129.0.14:2435] [InitConnectionTask]
2016-04-12 10:41:41:907 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.0.14]:2435 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:41:910 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Could not connect to: /10.129.0.14:2436. Reason: SocketException[Connection refused to address /10.129.0.14:2436] [InitConnectionTask]
2016-04-12 10:41:41:910 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.0.14]:2436 is added to the blacklist. [TcpIpJoinerOverAWS]
2016-04-12 10:41:48:921 WARNI [10.129.1.129]:2434 [orientdb] [3.6.2] Ignoring received partition table, startup is not completed yet. Sender: Address[10.129.1.219]:2434 [InternalPartitionService]
2016-04-12 10:41:48:924 INFO [10.129.1.129]:2434 [orientdb] [3.6.2]
Members [3] {
Member [10.129.1.219]:2434
Member [10.129.1.48]:2434
Member [10.129.1.129]:2434 this
}
[ClusterService]
2016-04-12 10:41:50:942 INFO [10.129.1.129]:2434 [orientdb] [3.6.2] Address[10.129.1.129]:2434 is STARTED [LifecycleService]
2016-04-12 10:41:50:943 INFO Starting distributed server 'orientdb-i-8d93ec05' (hzID=3082da3b-c329-49f3-8561-2053ac5bbe21) dbDir='/opt/orientdb-enterprise-2.1.15/databases/'... [OHazelcastPlugin]
2016-04-12 10:41:50:985 INFO [orientdb-i-8d93ec05] found no previous messages in queue orientdb.node.orientdb-i-8d93ec05.response [OHazelcastDistributedMessageService]
2016-04-12 10:41:50:992 INFO [orientdb-i-8d93ec05] loaded database configuration from active cluster [OHazelcastPlugin]
2016-04-12 10:41:51:002 WARNI [orientdb-i-8d93ec05] updated distributed configuration for database: db_1:
----------
{
"version": 1,
"replication": true,
"autoDeploy": true,
"hotAlignment": true,
"resyncEvery": 15,
"executionMode": "synchronous",
"readQuorum": 1,
"writeQuorum": 2,
"failureAvailableNodesLessQuorum": false,
"readYourWrites": true,
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"index": {
},
"*": {
"servers": ["orientdb-i-8d93ec05","<NEW_NODE>"]
}
}
}
---------- [OHazelcastPlugin]
2016-04-12 10:41:51:009 INFO [orientdb-i-8d93ec05] Current node started as MASTER for database 'db_1' [OHazelcastPlugin]
2016-04-12 10:41:51:108 WARNI [orientdb-i-8d93ec05]->[[]] requesting deploy of database 'db_1' on local server... [OHazelcastPlugin]
2016-04-12 10:41:51:112 SEVER [orientdb-i-8d93ec05] No nodes configured for partition 'db_1.null' request: id=-1 from=orientdb-i-8d93ec05 task=deploy_db [OHazelcastPlugin][orientdb-i-8d93ec05] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: No nodes configured for partition 'db_1.null' request: id=-1 from=orientdb-i-8d93ec05 task=deploy_db
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:347)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.requestDatabase(OHazelcastPlugin.java:944)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:906)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1484)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:175)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:993)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:336)
at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
PS: 10.129.1.193 and 10.129.1.213 is the ELBs, 10.129.0.14 is the NAT instance and 10.129.1.129/orientdb-i-8d93ec05 is the new node, 10.129.1.48/orientdb-i-346e11bc is (or is supposed to be) the "master" node.
What "worries" me is the No nodes configured for partition 'db_1'... part!
Even bigger problem: If I then shutdown all other nodes, and restart the master, it won't come up either!
com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'hasfieldset_orientdb-i-346e11bc' in class 'HasFieldset'
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1663)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1313)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.loadDistributedDatabases(OHazelcastPlugin.java:1457)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:177)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:993)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:336)
at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Quorum cannot be reached because there are no nodes available
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.calculateQuorum(OHazelcastDistributedDatabase.java:417)
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:121)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:358)
at com.orientechnologies.orient.server.distributed.ODistributedStorage.command(ODistributedStorage.java:326)
at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1333)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2172)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1081)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1656)
... 6 more
2016-04-12 11:49:53:989 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /127.0.0.1:36167. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
2016-04-12 11:49:54:090 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /127.0.0.1:36168. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
Which leads me to question three:
- How do I now recover my setup, without removing the DB and import it from backup?
