Is there a maximum size for index numbers in Google Datastore

Question

I have an application that is storing and indexing numeric fields in Google Datastore. Internally these are being created and indexed using Java/Objectify with the Entity that is being persisted annotated with the @Index annotation for the fields that we wish to search on...

@Index
private Long fieldOne;

@Index
private Long fieldTwo;

All works well until I try to search for entities using these indexed fields. fieldOne has a value of 5709068098338816 and searching on that works fine using the GQL:

SELECT * 
FROM MyEntity
WHERE fieldOne = 5709068098338816

However, fieldTwo has a value of 3572054303678568400 and this is where my issues start. For some reason I am unable to search using this value in either GQL or using objectify. The following GQL returns no results even though I can see the value in google datastore:

SELECT * 
FROM MyEntity
WHERE fieldTwo = 3572054303678568400

I can see that this is both stored and indexed in Google Datastore.

My question is this, Is there a limit to the size of an indexed number in Google Datastore? Is it possible that this value is somehow to large to be indexed correctly or could I have missed something else completely? This value is below the maximum Long value so I am not sure what is going on.

Any help would be much appreciated. Cheers!

score 1 · Answer 1 · answered Jan 09 '15 at 19:48

1

It may be due to your fieldTwo being over 16 digits long. The App Engine Datastore can use Long values for primary keys - but in the documentation linked below - it states that:

Each ID can be up to 16 decimal digits long.

Your field is not a primary key, but I am guessing it is following the same pattern as a Long PrimaryKey when it is indexed.

https://cloud.google.com/appengine/docs/java/datastore/entities

answered Jan 09 '15 at 19:48

Jeff Deskins

1,650
1
10
9

2

You are misreading the documentation. It says that the IDs generated by the generator will be up to 16 digits long. It does not say that Long primary keys cannot be longer than that. In fact, we know that they can - the reason the generator has this restriction is because the first version of the scattered generator did not have this restriction, and it caused failures when large numbers were passed to many Javascript applications. – stickfigure Jan 09 '15 at 20:53
1

Thanks for clarifying. I now remember those failures when they first switched to the scattered generator. The first thing I noticed with the values above was that one was over 16 digits and thought that may be why. – Jeff Deskins Jan 09 '15 at 20:59

James Faulkner · Accepted Answer · 2015-01-22T14:30:35.647

I think I have managed to figure out what the issue was. It appears that when using the App Engine console, for some reason the wrong value is being returned for the value of these long entities. I was able to replicate this using the new app engine console...

I have an entity called FooBar which has an indexed property called fieldTwo . If I want to search for all FooBars that have a fieldTwo of the value 3032316359359434752 through the new App Engine interface, It results in a request like.

"https://console.developers.google.com/m/datastore/entities?filters="%"5B"%"7B"%"22property"%"22:"%"22fieldTwo"%"22,"%"22type"%"22:"%"22number"%"22,"%"22operator"%"22:"%"22equal"%"22,"%"22value"%"22:"%"223032316359359434752"%"22"%"7D"%"5D&kind=FooBar&namespace=&pid=some-id"

The problem is the result of this request. The request appears to be returning the correct entities, however, the value of fieldTwo in the returned entities is incorrect. The result of the previous request was...

{
    "tos": [],
    "cursor": "E-ABAIICQ2oMc35mb29kaXQtZGV2cjMLEg1PcmRlckZlZWRiYWNrGICAgPyq8IQIDAsSDE1lYWxGZWVkYmFjaxiAgICAgK6ZCgyIAgAU",
    "results": [
        {
            "properties": [
                // some properties removed
                {
                    "indexed": true,
                    "name": "fieldTwo",
                    "value": 3032316359359435000,
                    "type": "number"
                },
                // some properties removed
            ],
            "key": {
                "label": "5629499534213120",
                "object": "FooBar id:5629499534213120",
                "type": "id"
            }
        }
    ],
    "status": 0
}

As you can see the value of fieldTwo is incorrect. It is 3032316359359435000 when it should be 3032316359359434752. This is what is being returned from the server so I don't think this is a Javascript issue though, since the incorrect value is coming back from the server when a make a cURL request for this query.

At the moment I'm leaning towards this being a bug in Google Datastore.

Edited

Google got back to me and confirmed this was indeed a bug in the interface for datastore. It has since been fixed.

Is there a maximum size for index numbers in Google Datastore

2 Answers2