3

I'm using the bulkloader to upload data into my App Engine data storage.

I can't seem to store a dictionary into a JsonProperty and I am getting the following error:

BadValueError: Unsupported type for property nearby_countries: <type 'dict'>

My model defines this property as a JsonProperty:

nearby_countries = ndb.JsonProperty()

The only workaround I found seems to store a json.dumps() of my value instead but I guess this basically storing the string representation of the dictionary rather than the dictionary itself.

My understanding of the JsonProperty is that it takes a python object as the value and that I should not be bothered about the JSON serialization which ndb will take care of. Am I correct?

Value is a Python object (such as a list or a dict or a string) that is serializable using Python's json module; the Datastore stores the JSON serialization as a blob.

Erwan
  • 3,733
  • 30
  • 25

2 Answers2

3

After lots of try & error as well as googling around for similar posts, I manage to find the following post which lead me the below solution:

http://blog.thekensta.com/2012/06/google-app-engine-bulk-loader-and-ndb.html

In short, JsonProperties are stored as blobs and we need to pass the bulkloader the correct transform method to generate a blob from the json string. We can use transform.blobproperty_from_base64 (from the google.appengine.ext.bulkload.transform module)

So I convert my list or dict to a string JSON string representation which then gets converted to a blob so that the bulkloader can store it:

import_transform: "lambda x: transform.blobproperty_from_base64(base64.b64encode(bytes(json.dumps(x.strip(' ,').split(',')))))"

The same reasoning fixes the TextProperty saved as Strings (mentioned in my comment above). You need to use db.Text as the transform function:

import_transform: db.Text

And to save a repeated=True TextProperty, I actually had to transform it to a blob as well:

import_transform: "lambda x: transform.blobproperty_from_base64(base64.b64encode(bytes(json.dumps(x.strip(' ,').split(',')))))"

(in the exemple above I actually turn a coma separated string into a list of Text objects to be stored in a TextProperty(repeated=True)

doru
  • 9,022
  • 2
  • 33
  • 43
Erwan
  • 3,733
  • 30
  • 25
1

In general you are correct about the JsonProperty. However, the bulk loader is special. Honestly, I don't know much about how it works, but in that context I wouldn't be surprised if it required you to call json.dumps() yourself.

Guido van Rossum
  • 16,690
  • 3
  • 46
  • 49
  • Thanks. The problem is that when I call json.dumps() myself, the string get stored in the Data storage, not a blob. I can see the stored values in the admin console although the property is defined as JsonProperty which should result in a binary being stored. And when loading this property later, it complains that it is not a JSON object. – Erwan Mar 01 '13 at 01:39
  • I'm actually thinking that it could be related to the bulkloader validation. Anyone knows where to find it, I'll try to see if we can losen it to make sure that my gets accepted in a JsonProperty. The only other workaround I found is to first import the string into a TextProperty through bulkloader, and then run a script which will create the back from the string and store it into the JsonProperty. Not optimal though. – Erwan Mar 01 '13 at 02:04
  • There is definitely an issue with the some validation of the bulkloader. I am also getting the following error: Property XXXX is too long. Maximum length is 500. For a property XXXX declared as TextProperty (not indexed by default and 500 max length should not apply here) – Erwan Mar 01 '13 at 03:32
  • I found the answer the my last comment regarding the TextProperty here: http://stackoverflow.com/questions/3434090/app-engine-badvalueerror-on-bulk-data-upload-textproperty-being-construed-as-s You basically need to use db.Text to transform the string upon importing with bulkloader in order to make sure it gets saved as a TextProperty – Erwan Mar 01 '13 at 14:05