GAE JDO Query sometimes returns with newest persisted entities missing

Question

I am trying to build a simple app using google app engine, with java and JDO.

my application creates Sale objects and persists them to the datastore, once persisted i attempt to list using a query, ALL the Sales in the datastore.

however sometimes the list is missing one or two of the last persisted Sales, refreshing the browser a few times will eventually make all Sales listed.

even when i click the "add sale" button very slowly (once every 3 seconds or so) there is still no guarantee that ALL persisted Sales will be retrieved from the query.

there must be a way to ensure that a query returns ALL Sales, i have tried using transactions and no luck, however i may have used it wrong, any advice would be appreciated.

Here is my Sale object:

import java.util.Date;

import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;

import com.google.appengine.api.datastore.Key;

@PersistenceCapable(detachable="true")
public class Sale {

    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Key key;

    @Persistent
    private String title;

    @Persistent
    private Date valuationDate;

    public Sale(){

    }

    //getters and setters

This is my utility class:

public class SaleUtil {

    public static void addSale(){
        PersistenceManager pm = PMF.get().getPersistenceManager();

        Sale s = new Sale();
        s.setTitle("new sale");
        s.setValuationDate(new Date());

        try{
            pm.makePersistent(s);
        } catch (Exception e){
            System.out.println(e);
        } finally {
            pm.close();
        }
    }

    public static List<Sale> getSales(){
        List<Sale> sales = new ArrayList<Sale>();
        PersistenceManager pm = PMF.get().getPersistenceManager();
        Query query = pm.newQuery(Sale.class);
        query.setOrdering("valuationDate desc");
        try{
            List<Sale> results = (List<Sale>)query.execute();
            if(!results.isEmpty()){
                for(Sale s: results){
                    Sale detached = pm.detachCopy(s);
                    sales.add(detached);
                }
            }
        } catch (Exception e){
            System.out.println(e);
        } finally {
            query.closeAll();
            pm.close();
        }
        return sales;
    }

}

This is my home page:

<%@ page language="java" contentType="text/html; charset=ISO-8859-1" pageEncoding="ISO-8859-1"%>
<%@ page import="java.util.List" %>
<%@ page import="javax.jdo.PersistenceManager" %>
<%@ page import="javax.jdo.Query" %>
<%@ page import="com.grc.tradesharx.*" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"    "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Insert title here</title>
</head>
<body>
    <form action="/tradesharx" method="post">
        <input type="submit" value="add sale"/>
    </form>
    <%
        List<Sale> sales = SaleUtil.getSales();
        out.println(sales.size()+"<br/>");
        for(Sale s : sales){
            out.println(s+"<br/>");
        }
    %>
</body>
</html>

and this is my servlet:

package com.grc.tradesharx;

import java.io.IOException;
import javax.servlet.http.*;

@SuppressWarnings("serial")
public class TradeSharxServlet extends HttpServlet {
    public void doPost(HttpServletRequest req, HttpServletResponse resp) throws     IOException {
        SaleUtil.addSale();
        resp.sendRedirect("index.jsp");
    }
}

please help me, i know i am missing something.

thejaz · Accepted Answer · 2011-08-19T07:45:57.533

1

If you are using the High Replication Datastore (HRD), queries outside of entity groups are eventual consistent. It means it can take some time before new objects are replicated and returned in a regular non-ancestor query.

The solution for you is to use entity groups/ancestor queries for the data you need to be consistent.

... in the High Replication datastore, queries across entity groups (in other words, non-ancestor queries) may return stale results. In order to return strongly consistent query results in the High Replication environment, you need to query over a single entity group. This type of query is called an ancestor query.

Ancestor queries work because entity groups are a unit of consistency: all operations are applied to the entire group. Ancestor queries won't return data until the entire entity group is up to date. Thus, the data returned from ancestor queries on entity groups is strongly consistent.

http://code.google.com/intl/sv-SE/appengine/docs/java/datastore/hr/overview.html

edited Aug 19 '11 at 07:45

answered Aug 19 '11 at 07:38

thejaz

2,763
2
26
40

thank you for pointing me in the correct direction. i am using eclipse to build my application, is there any way to find out if the local datastore is is using HRD or Master/Slave? – Arthur Aug 19 '11 at 12:49
By default it works like Master/Slave, but you can simulate HRD. http://code.google.com/intl/sv-SE/appengine/docs/java/tools/devserver.html "You can configure the local datastore to simulate the consistency model of the High Replication Datastore. This will give you a good idea how an application configured to use the High Replication Datastore will operate in production." – thejaz Aug 19 '11 at 12:59
Some how my default was set to High Replication - unapplied job percentage: 50 i assume this means that half the time my commits would not be applied? – Arthur Aug 19 '11 at 13:08

GAE JDO Query sometimes returns with newest persisted entities missing

1 Answers1