App-Engine JDO consistent reading not working, maybe caching?

Question

Today it's the first time I'm using GWT and JDO. I am running it with Eclipse in the local debug mode.

I do the following thing:

    public Collection<MyObject> add(MyObject o) {
PersistenceManager pm = PMF.get().getPersistenceManager();
try {
    pm.makePersistent(o);
    Query query = pm.newQuery(MyObject.class);// fetch all objects incl. o. But o only sometimes comes...
List<MyObject> rs = (List<MyObject>) query.execute();
ArrayList<MyObject> list= new ArrayList<MyObject>();
for (MyObject r : rs) {
    list.add(r);
}
return list; 
} finally {
    pm.close();
}
}

I already set <property name="datanucleus.appengine.datastoreReadConsistency" value="STRONG" /> in my jdoconfig.xml. Do I have to set some other transaction stuff in the config? Was somebody got a working jdoconfig.xml? Or is the problem somewhere else? Some caching inbetween?

EDIT: Things I have tried:

• Setting NontransactionalRead/Write to false
• Using the same/a different PersistenceManager though calling PMF.get().getPersistenceManager() multiple times
• Using transactions
• ignoreCache = true on PersistenceManager
• calling flush and checkConsistency

The jdoconfig:

    <persistence-manager-factory name="transactions-optional">
<property name="datanucleus.appengine.datastoreReadConsistency" value="STRONG" />
    <property name="javax.jdo.PersistenceManagerFactoryClass"
        value="org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory"/>
    <property name="javax.jdo.option.ConnectionURL" value="appengine"/>
    <property name="javax.jdo.option.NontransactionalRead" value="true"/>
    <property name="javax.jdo.option.NontransactionalWrite" value="true"/>
    <property name="javax.jdo.option.RetainValues" value="true"/>
    <property name="datanucleus.appengine.autoCreateDatastoreTxns" value="true"/>
</persistence-manager-factory>

I must be missing something central here because all approaches fail...

EDIT2: When I split the job into two transaction the log says that the write transaction fished and then the read transaction starts. But it doesn't find the just persited object. It always says Level 1 Cache of type "weak" initialised aswell. Is week bad or good?

It about 30% of requests that go wrong... Might I be some lazy query loading issue?

Answer 1

Calling the makePersistent() method doesn't write to the datastore; closing the PersistenceManager or committing your changes does. Since you haven't done this when you run your query, you're getting all objects from the datastore which does not, yet, include the object you just called makePersistent on.

Read about object states here: http://db.apache.org/jdo/state_transition.html

There are two ways around this, you can put this inside a transaction since the commit writes to the datastore (keep in mind GAE 5 transaction/entity type limit on transactions) and commit before running your query; Example using transaction...

public Collection<MyObject> add(MyObject o) {
    PersistenceManager pm = PMF.get().getPersistenceManager();
    ArrayList<MyObject> list = null;
    try {
        Transaction tx=pm.currentTransaction();
        try {
            tx.begin();
            pm.makePersistent(o);
            tx.commit(); 
        } finally {
            if (tx.isActive()) {
                tx.rollback();
            }
        }

        Query query = pm.newQuery(MyObject.class);
        List<MyObject> rs = (List<MyObject>) query.execute();
        ArrayList<MyObject> list = new ArrayList<MyObject>();
        for (MyObject r : rs) {
            list.add(r);
        }
    } finally {
        pm.close();
    }

    return list; 
}

or you could close the persistence manager after calling makePersistent on o and then open another one to run your query on.

// Note that this only works assuming the makePersistent call is successful
public Collection<MyObject> add(MyObject o) {
    PersistenceManager pm = PMF.get().getPersistenceManager();
    try {
        pm.makePersistent(o);
    } finally {
        pm.close();
    }

    pm = PMF.get().getPersistenceManager();
    ArrayList<MyObject> list = null;
    try {

        Query query = pm.newQuery(MyObject.class);
        List<MyObject> rs = (List<MyObject>) query.execute();
        list= new ArrayList<MyObject>();
        for (MyObject r : rs) {
            list.add(r);
        }

    } finally {
        pm.close();
    }

    return list; 
}

NOTE: I originally said you could just add o to the result list before returning; but that isn't a smart thing to do since in the event that there is a problem writing o to the datastore; then the returned list wouldn't reflect the actual data in the datastore. Doing what I now have (committing a transaction or closing the pm and then getting another one) should work since you have your datastoreReadPolicy set to STRONG.

Answer 2

Franz, the Default read consistency in the JDO Config is STRONG. so if you are trying to approach it in that direction, it wont lead you anywhere

Check this out as i think it mentions something similar to the scenario which you are encountering, with the committed data not returned back in the query. It isnt concurrent as mentioned, but it explains the commit process.

http://code.google.com/appengine/articles/transaction_isolation.html

Also, another approach would be to query using Extents and find out if that solves the particular use case you are looking at, since i believe you are pulling out all the records in the table.

EDIT :

Since in the code snippet that you have mentioned, it queries the entire table. And if that is what you need, you can use an Extent... The way to use it is by calling

Extent ext = getExtent(<Entity Class name>)

on the persistenceManager singleton object. You can then iterate through the Extent

Check out the documentation and search for Extents on the page here. http://code.google.com/appengine/docs/java/datastore/jdo/queries.html

Answer 3

I encountered the same problem and this didn't help. Since it seems to be the top result on Google for "jdo app engine consistency in eclipse" I figured I would share the fix for me!

Turns out I was using multiple instances the PersistenceManagerFactory which led to some bizarre behaviour. The fix is to have a singleton that every piece of code accesses. This is in fact documented correctly on the GAE tutorials but I think it's importance is understated.

Getting a PersistenceManager Instance

An app interacts with JDO using an instance of the PersistenceManager class. You get this instance by instantiating and calling a method on an instance of the PersistenceManagerFactory class. The factory uses the JDO configuration to create PersistenceManager instances.

Because a PersistenceManagerFactory instance takes time to initialize, an app should reuse a single instance. An easy way to manage the PersistenceManagerFactory instance is to create a singleton wrapper class with a static instance, as follows:

PMF.java

import javax.jdo.JDOHelper;
import javax.jdo.PersistenceManagerFactory;

    public final class PMF {
        private static final PersistenceManagerFactory pmfInstance =
            JDOHelper.getPersistenceManagerFactory("transactions-optional");

        private PMF() {}

        public static PersistenceManagerFactory get() {
            return pmfInstance;
        }
    }