Index creation error when starting embedded Neo4j HA Slave

Question

I am trying to create a cluster of Java applications running the embedded Neo4J HA server but I am getting an error when starting any slave servers

Caused by: org.springframework.dao.InvalidDataAccessResourceUsageException: Error executing statement CREATE INDEX ON :`DeviceNode`(`deviceType`); nested exception is org.springframework.dao.InvalidDataAccessResourceUsageException: Error executing statement CREATE INDEX ON :`DeviceNode`(`deviceType`); nested exception is org.neo4j.cypher.CypherExecutionException: Modifying the database schema can only be done on the master server, this server is a slave. Please issue schema modification commands directly to the master.
        at org.springframework.data.neo4j.support.query.CypherQueryEngine.query(CypherQueryEngine.java:56) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.support.schema.SchemaIndexProvider.createIndex(SchemaIndexProvider.java:36) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.support.mapping.EntityIndexCreator$1.doWithPersistentProperty(EntityIndexCreator.java:45) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.support.mapping.EntityIndexCreator$1.doWithPersistentProperty(EntityIndexCreator.java:41) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:261) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.support.mapping.EntityIndexCreator.ensureEntityIndexes(EntityIndexCreator.java:41) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.support.mapping.Neo4jMappingContext.updateStoredEntityType(Neo4jMappingContext.java:77) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.support.mapping.Neo4jMappingContext.addPersistentEntity(Neo4jMappingContext.java:71) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.support.mapping.Neo4jMappingContext.addPersistentEntity(Neo4jMappingContext.java:49) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:170) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:139) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:65) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.repository.query.CypherQueryBuilder.<init>(CypherQueryBuilder.java:37) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.repository.query.CypherQueryCreator.create(CypherQueryCreator.java:72) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.repository.query.CypherQueryCreator.create(CypherQueryCreator.java:35) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.repository.query.parser.AbstractQueryCreator.createCriteria(AbstractQueryCreator.java:109) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.repository.query.parser.AbstractQueryCreator.createQuery(AbstractQueryCreator.java:88) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.repository.query.parser.AbstractQueryCreator.createQuery(AbstractQueryCreator.java:73) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.repository.query.DerivedCypherRepositoryQuery.<init>(DerivedCypherRepositoryQuery.java:59) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.repository.query.GraphQueryMethod.createQuery(GraphQueryMethod.java:146) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.repository.GraphRepositoryFactory$1.resolveQuery(GraphRepositoryFactory.java:113) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.<init>(RepositoryFactorySupport.java:304) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport.getRepository(RepositoryFactorySupport.java:161) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.initAndReturn(RepositoryFactoryBeanSupport.java:224) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.afterPropertiesSet(RepositoryFactoryBeanSupport.java:210) ~[spring-data-commons-1.7.0.RELEASE.jar:na]
        at org.springframework.data.neo4j.repository.GraphRepositoryFactoryBean.afterPropertiesSet(GraphRepositoryFactoryBean.java:69) ~[spring-data-neo4j-3.0.0.RELEASE.jar:na]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571) ~[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) ~[spring-beans-3.2.8.RELEASE.jar:3.2.8.RELEASE]
        ... 38 common frames omitted

The most important bit seems to be Modifying the database schema can only be done on the master server, this server is a slave. Please issue schema modification commands directly to the master. I believe that the index is being created as the Repository interfaces are parsed by SDN.

Have I done something crazy somewhere, can I tell the slaves just to hang back a little and pull their schema from the Master which will get there in the end? I believe that my config is fairly standard:

<util:map id="config">
    <entry key="org.neo4j.server.database.mode" value="HA"/>

    <entry key="enable_remote_shell" value="true"/> 
    <entry key="ha.server_id" value="1"/>
    <entry key="ha.initial_hosts" value="neo1:5001,neo2:5001,neo3:5001"/>
    <entry key="ha.allow_init_cluster" value="true"/>
    <entry key="ha.cluster_join_timeout" value="60s"/>
    <entry key="ha.slave_only" value="false"/>
</util:map>

<bean id="graphDbFactory" class="org.neo4j.graphdb.factory.HighlyAvailableGraphDatabaseFactory"/>

<bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newHighlyAvailableDatabaseBuilder">
    <constructor-arg value="#{neoloc}"/>
</bean>

<bean id="graphDbBuilderFinal" factory-bean="graphDbBuilder" factory-method="setConfig">
    <constructor-arg ref="config"/>
</bean>

<bean id="graphDatabaseService" factory-bean="graphDbBuilderFinal" factory-method="newGraphDatabase" destroy-method="shutdown" />

<neo4j:config graphDatabaseService="graphDatabaseService"/>
<neo4j:repositories base-package="com.clarifimedia.data.repository"></neo4j:repositories>

<tx:annotation-driven transaction-manager="transactionManager"/>

<bean id="conversionService" class="org.springframework.context.support.ConversionServiceFactoryBean">
    <property name="converters">
        <set>
            <bean class="com.clarifimedia.data.converter.DateTimeToLongConverter"/>
            <bean class="com.clarifimedia.data.converter.LongToDateTimeConverter"/>
        </set>
    </property>
</bean>

Configuration values are in fact read from JNDI but I have hardcoded above as a load of #{xyz} wouldn't be so handy.

Versions
neo4j-2.0.1
neo4j-ha-2.0.1
spring-data-neo4j-3.0.0.RELEASE
spring-general-stuff-3.2.8.RELEASE

[Edit, tangent, bug?]
I tried to start three isolated masters by breaking the network, with the intent of fixing the network and seeing if an election took place. However on startup the webapp fails to start due to a TimeoutException.

Caused by: java.util.concurrent.TimeoutException: null
        at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:300) ~[neo4j-cluster-2.0.1.jar:2.0.1]
        at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:158) ~[neo4j-cluster-2.0.1.jar:2.0.1]
        at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:91) ~[neo4j-cluster-2.0.1.jar:2.0.1]
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:503) ~[neo4j-kernel-2.0.1.jar:2.0.1]

ClusterJoin catches both InterruptedException and ExecutionException but not the TimeoutException as thrown by StateMachineProxyFactory$ResponseFuture. Given the while(true) in the ClusterJoin class I think this is a bug or do I somehow have a bad mix of jar files (although both these classes are in neo4j-cluster-2.0.1).

Interesting question. I think you should separate out the second issue into a second question. — Michael Hunger, Apr 11 '14 at 21:41
Done: http://stackoverflow.com/questions/23056095/starting-first-node-in-heo4j-ha-cluster-fails-even-when-allowed-to-create-cluste — JohnMark13, Apr 14 '14 at 09:06

Wouter · Answer 1 · 2014-04-24T14:05:29.880

I got a workaround for this. Notice that I use spring Java config, but configuring it via XML shouldn't be a big deal.

Create two classes that extend standard sdn classes, first one adds a method check whether the current jvm is running with neo as master or not:

public class OurDelegatingGraphDatabase extends DelegatingGraphDatabase {

    public OurDelegatingGraphDatabase(GraphDatabaseService delegate) {
        super(delegate);
    }

    public boolean isMaster(){
        if (delegate instanceof HighlyAvailableGraphDatabase){
            HighlyAvailableGraphDatabase highlyAvailableGraphDatabase = (HighlyAvailableGraphDatabase) delegate;
            return highlyAvailableGraphDatabase.isMaster();
        }
        return true;
    }
}

Then a custom SchemaIndexProvider that only does indexing when it is on a master (single mode is also considered master):

public class HACheckingSchemaIndexProvider extends SchemaIndexProvider {

    private OurDelegatingGraphDatabase graphDatabase;

    public HACheckingSchemaIndexProvider(GraphDatabase gd) {
        super(gd);
        if (gd instanceof OurDelegatingGraphDatabase){
            this.graphDatabase = (OurDelegatingGraphDatabase) gd;
        } else {
            throw new IllegalStateException("OurDelegatingGraphDatabase was not configured");
        }
    }

    @Override
    public void createIndex(Neo4jPersistentProperty property) {
        if (!graphDatabase.isMaster()){
            return;
        }
        super.createIndex(property);
    }

    @Override
    public void createIndex(String label, String prop, boolean unique) {
        if (!graphDatabase.isMaster()){
            return;
        }
        super.createIndex(label, prop, unique);
    }
}

Then configure these using spring:

@Configuration
//..whatever else config you have
public class Neo4jConfig extends Neo4jConfiguration {

//other config stuf..

    @Bean
    @Autowired
    @DependsOn("graphDatabaseService")
    public GraphDatabase graphDatabase() {
        return new OurDelegatingGraphDatabase(getGraphDatabaseService());
    }

    @Bean
    @Override
    public SchemaIndexProvider schemaIndexProvider() throws Exception {
        return new HACheckingSchemaIndexProvider(graphDatabase());
    }

Nice workaround and should translate easily to XML. Having traced through the exception stack I considered something similar but worried about the ground moving under my feet as SDN develops. Whilst I also accept that SDN is not Neo4J and is community supported this is a rather fundamental feature for anyone who might be considering paying a licence fee! — JohnMark13, Apr 24 '14 at 15:55

score 2 · Answer 2 · answered Apr 11 '14 at 21:44

2

SDN is not aware of a database being HA or not.

Actually I didn't know that Slaves could not accept schema updates.

So this is probably something that we have to make configurable on the spring config for Neo4j, to disable index creation for certain setups explicitly.

Could you raise a JIRA issue about this?

answered Apr 11 '14 at 21:44

Michael Hunger

41,339
3
57
80

Issue raised: https://jira.spring.io/browse/DATAGRAPH-459 I thought that this was probably the case as I traced through the code and thought why should SDN know/care about the state of the underlying graph?! It does present me with an interesting challenge though. Thanks for looking, any workarounds you can think of (other than just coding using Neo APIs directly)? – JohnMark13 Apr 12 '14 at 18:17
I can see this is still an open issue in JIRA. Has there been any progress on this? I'm facing the same issue and not keen on implementing the workaround. – BtySgtMajor Jul 16 '15 at 02:35
(But I did anyway. :) ) – BtySgtMajor Jul 16 '15 at 04:39
you can now configure it on the neo4j configuration there is a flag that says create-index=true or so. (don't have the code right now, but it's also documented) – Michael Hunger Jul 18 '15 at 09:26

score 0 · Answer 3 · answered Jul 16 '15 at 04:38

For those in a hurry to translate Wouter's Spring Java config into its XML equivalent, hopefully this helps (note that graphDatabaseServiceis defined as the OP has):

<bean depends-on="graphDatabaseService" id="graphDatabase" class="com.path.to.OurDelegatingGraphDatabase">
    <constructor-arg ref="graphDatabaseService" /> 
</bean>       

<bean id="schemaIndexProvider" class="com.path.to.HACheckingSchemaIndexProvider">
    <constructor-arg ref="graphDatabase" />
</bean>

This worked for me. Of course, if I've made any mistakes or the above can be improved, please let me know. Thanks!

Index creation error when starting embedded Neo4j HA Slave

3 Answers3

Linked