JDO transactions + many GAE instances = overriding data

Question

I have very simple PersistenceCapable class:

import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;
@PersistenceCapable
public class TheEntity {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Long id;
public Long getId() {
    return id;
}
@Persistent
private int version;
public int getVersion() {
    return version;
}
public void setVersion(int version)
{
    this.version=version;
}
}

And I simply update version by incrementing it (and I added some computations to make GAE warmup new instances sometimes)

public void test2(){
    PersistenceManager mgr = getPersistenceManager();
    mgr.currentTransaction().begin();
    TheEntity test = mgr.getObjectById(TheEntity.class, 5081359164899328L);
    test.setVersion(test.getVersion()+1);
    log.log(Level.SEVERE, "VERSION: " + test.getVersion());
        for (int i = 0; i < 10000; i++)
        {
            long x = i*i;
            x++;
        }
    mgr.currentTransaction().commit();
    mgr.close();}

So, I called the code 9 times, it never failed. The logs below:

W1. I 02:28:21.331 2015-01-10  200       0 B  6839ms /_ah/warmup
1. I 02:28:38.407 2015-01-10  204       0 B  4343ms /_ah/spi/com.vaktu.MyEndpoint.test1
   188.122.13.128 - - [09/Jan/2015:17:28:38 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=4343 cpu_ms=2940 cpm_usd=0.000022 instance=00c61b117c4063829b2bf7ef35e1e6e8a0e41978 app_engine_release=1.9.17
   I 02:28:38.012 com.vaktu.MyEndpoint test2: VERSION: 1
2. I 02:28:42.493 2015-01-10  204       0 B   115ms /_ah/spi/com.vaktu.MyEndpoint.test1
   188.122.13.128 - - [09/Jan/2015:17:28:42 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=115 cpu_ms=70 instance=00c61b117c4063829b2bf7ef35e1e6e8a0e41978 app_engine_release=1.9.17
   I 02:28:42.444 com.vaktu.MyEndpoint test2: VERSION: 2
W2. I 02:28:42.495 2015-01-10  200       0 B  7061ms /_ah/warmup
3. I 02:28:54.196 2015-01-10  204       0 B  4647ms /_ah/spi/com.vaktu.MyEndpoint.test1
   188.122.13.128 - - [09/Jan/2015:17:28:54 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=4647 cpu_ms=3173 cpm_usd=0.000022 instance=00c61b117c31a94949f82af423e0f1a3c1ac35 app_engine_release=1.9.17
   I 02:28:53.814 com.vaktu.MyEndpoint test2: VERSION: 3
4. I 02:28:59.126 2015-01-10  204       0 B   163ms /_ah/spi/com.vaktu.MyEndpoint.test1
   188.122.13.128 - - [09/Jan/2015:17:28:59 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=163 cpu_ms=70 instance=00c61b117c31a94949f82af423e0f1a3c1ac35 app_engine_release=1.9.17
   I 02:28:59.036 com.vaktu.MyEndpoint test2: VERSION: 4
5. I 02:29:02.426 2015-01-10  204       0 B   119ms /_ah/spi/com.vaktu.MyEndpoint.test1
  188.122.13.128 - - [09/Jan/2015:17:29:02 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=119 cpu_ms=117 instance=00c61b117c31a94949f82af423e0f1a3c1ac35 app_engine_release=1.9.17
   I 02:29:02.373 com.vaktu.MyEndpoint test2: VERSION: 5
6. I 02:29:06.947 2015-01-10  204       0 B   114ms /_ah/spi/com.vaktu.MyEndpoint.test1
  188.122.13.128 - - [09/Jan/2015:17:29:06 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=114 cpu_ms=70 instance=00c61b117c31a94949f82af423e0f1a3c1ac35 app_engine_release=1.9.17
   I 02:29:06.896 com.vaktu.MyEndpoint test2: VERSION: 6
7. I 02:29:11.705 2015-01-10  204       0 B   101ms /_ah/spi/com.vaktu.MyEndpoint.test1
   188.122.13.128 - - [09/Jan/2015:17:29:11 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=101 cpu_ms=47 instance=00c61b117c31a94949f82af423e0f1a3c1ac35 app_engine_release=1.9.17
   I 02:29:11.665 com.vaktu.MyEndpoint test2: VERSION: 7
8. I 02:29:30.133 2015-01-10  204       0 B   111ms /_ah/spi/com.vaktu.MyEndpoint.test1
  188.122.13.128 - - [09/Jan/2015:17:29:30 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=111 cpu_ms=117 instance=00c61b117c4063829b2bf7ef35e1e6e8a0e41978 app_engine_release=1.9.17
   I 02:29:30.085 com.vaktu.MyEndpoint test2: VERSION: 3
9. I 02:29:33.598 2015-01-10  204       0 B   158ms /_ah/spi/com.vaktu.MyEndpoint.test1
  188.122.13.128 - - [09/Jan/2015:17:29:33 -0800] "POST /_ah/spi/com.vaktu.MyEndpoint.test1 HTTP/1.1" 204 0 - "Google-HTTP-Java-Client/1.18.0-rc (gzip)" "mmorpgmonsters.appspot.com" ms=158 cpu_ms=93 instance=00c61b117c4063829b2bf7ef35e1e6e8a0e41978 app_engine_release=1.9.17
   I 02:29:33.509 com.vaktu.MyEndpoint test2: VERSION: 4

test1 just calls test2, and the code of test2 is above. W1 = warmup of instance 1 W2 = warmup of instance 2 Requests run on instance 1: 1, 2, 8, 9 Requests run on instance 2: 3, 4, 5, 6, 7 Now, after 9 updates that should increase the version to 9, I have version = 4 in my database. Seriously, it is a disaster for me.. Any help very appreciated. If anyone implemented JPA and could test if it works better please give feedback.

My jdoconfig.xml:

   <property name="datanucleus.appengine.datastoreEnableXGTransactions" value="true"/>
   <property name="javax.jdo.PersistenceManagerFactoryClass" value="org.datanucleus.api.jdo.JDOPersistenceManagerFactory"/>
   <property name="javax.jdo.option.ConnectionURL" value="appengine"/>
   <property name="javax.jdo.option.NontransactionalRead" value="true"/>
   <property name="javax.jdo.option.NontransactionalWrite" value="true"/>
   <property name="javax.jdo.option.RetainValues" value="true"/>
   <property name="datanucleus.appengine.autoCreateDatastoreTxns" value="true"/>
   <property name="datanucleus.appengine.singletonPMFForName" value="true"/>

My persistence.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence         http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
<persistence-unit name="transactions-optional">
    <provider>org.datanucleus.api.jpa.PersistenceProviderImpl</provider>
    <properties>
        <property name="datanucleus.NontransactionalRead" value="true"/>
        <property name="datanucleus.NontransactionalWrite" value="true"/>
        <property name="datanucleus.ConnectionURL" value="appengine"/>
    </properties>
</persistence-unit>
</persistence>

JARS in WEB-INF/lib:

appengine-api-1.0-sdk-1.9.17.jar
appengine-api-labs.jar
appengine-api.jar
appengine-endpoints-deps.jar
appengine-endpoints.jar
appengine-jsr107cache-1.9.17.jar
asm-4.0.jar
datanucleus-api-jdo-3.1.3.jar
datanucleus-api-jpa-3.1.3.jar
datanucleus-appengine-2.1.2.jar
datanucleus-cache-3.1.3.jar //I want to use memcache after I finally make it work without it
datanucleus-core-3.1.3.jar
gcm-server.jar
geronimo-jpa_2.0_spec-1.0.jar
jdo-api-3.0.1.jar
json_simple-1.1.jar
jsr107cache-1.1.jar
jta-1.1.jar

I'll be very, very grateful for help...

IDENTITY is assigned by Google's datastore. Any JPA implementation should simply make use of the value they have provided. Suggest you look at the log at debug level and see what it says, or at the code for their plugin https://code.google.com/p/datanucleus-appengine/ Maybe their datastore doesn't cope with multithreaded access? You could try using SEQUENCE strategy on the id field — Neil Stockton, Jan 10 '15 at 09:01
Thank you for your comment, but you did not understand the situation. I dont change the ID that is allocated automatically. I created the object, and GAE assigned its ID = 5081359164899328L. Then I fetch the object by its ID, and change the field version - here in this test case I always increase it by one for simplicity, in my real application I do more stuff, but it just gets overriden. I need some way of persisting data that will cope with: 1) multithreaded access / many GAE instances (as the whole point of GAE is using many server instances), and 2) will deal with concurrent modificatios — user2855896, Jan 10 '15 at 10:17
GAE datastore has this "eventual consistency" thing, that changes don't get sent immediately, so simply not like an RDBMS, and cross-over in transactions could occur easily. Can't help further since I don't use GAE/Datastore — Neil Stockton, Jan 10 '15 at 10:45
I read about it but thought that eventual consistency means it will be consistent, eventually. If I have stored integer x=0, I successfully, transactionally increase it 9 times, then after 1 - 5 seconds all my reads will say x=9. I think "no consistency" is appropriate in my case. Its either a bug in my code (e.g. I miss some crucial method or miss some entry in xml), or a bug in JDO. There must be some way around, if not I think GAE is useless. If you have cloud database, you need transactions, multi-instance access and concurrent modification handling. Thanks for trying to help though. — user2855896, Jan 10 '15 at 13:53
"A bug in JDO" : you aren't using JDO according to this question. And besides that would be a bug in Google's plugin if it is that - so see the source code and the log. — Neil Stockton, Jan 10 '15 at 14:36
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/68550/discussion-between-user2855896-and-neil-stockton). — user2855896, Jan 10 '15 at 15:06
Hi @user2855896, did you ever get this to work or found an explanation as to why this failed? — kanghj91, Jul 11 '16 at 04:19
@kanghj91, I did not. I had to rewrite the whole application to get rid of JDO / JPA, I use the low level api and it works fine. As Neil Stockton clarified, it is a bug in JDO plugin from Google that makes it totally unreliable, unusable and useless. I cannot imagine using it by anyone with a bug like this. Here is my suggestion for another person with a problem with JDO plugin for GAE, you might find it interesting: http://stackoverflow.com/a/27891239/2855896 — user2855896, Jul 12 '16 at 06:46

JDO transactions + many GAE instances = overriding data

0 Answers0

Linked