1

I am trying to create something like a folder structure for saving to GAE ndb datastore. I will be saving a top root folder (MjlistitemFolder) to the datastore. A MjlistitemFolder can have a number of other subitems (Mjlistitem) in its items property. Ultimately the content of a folder items will be one of these: MjlistitemFolder, MjlistitemJobGeneric, MjlistitemJobApp

This all works if I create this structure in memory. But after put()ing it away and reloading the root folder from the datastore, I don't get the same structure back.

class Mjlistitem(ndb.Model):
    pass

class MjlistitemFolder(Mjlistitem):
    title = ndb.StringProperty()
    items = ndb.StructuredProperty(Mjlistitem, repeated=True)

class MjlistitemJob(Mjlistitem):
    pass

class MjlistitemJobGeneric(MjlistitemJob):
    jobtype = ndb.IntegerProperty()

class MjlistitemJobApp(MjlistitemJob):
    appleid = ndb.StringProperty()

I get these warnings:

WARNING  2014-04-04 07:14:17,506 model.py:2359] Skipping unknown structured subproperty (items.items.appleid) in repeated structured property (items of MjlistitemFolder)
WARNING  2014-04-04 07:14:17,506 model.py:2359] Skipping unknown structured subproperty (items.items.jobtype) in repeated structured property (items of MjlistitemFolder)
WARNING  2014-04-04 07:14:17,506 model.py:2359] Skipping unknown structured subproperty (items.items.appleid) in repeated structured property (items of MjlistitemFolder)

It seems like the db→instance process renders the stuff in items to be of Mjlistitem class only. How do I make them appear as their real inherited classes?

This is how I create a test structure:

        rootfolder = MjlistitemFolder(title="root")

        subfolder = MjlistitemFolder(title="Cool things")
        subfolder.items.append(MjlistitemJobApp(appleid="281796108")) # evernote
        subfolder.items.append(MjlistitemJobGeneric(jobtype=3)) # phone number
        subfolder.items.append(MjlistitemJobApp(appleid="327630330")) # dropbox

        rootfolder.items.append(MjlistitemJobGeneric(jobtype=15)) # passbook
        rootfolder.items.append(subfolder)
        rootfolder.items.append(MjlistitemJobGeneric(jobtype=17)) # appstore
        rootfolder.put()
Jonny
  • 15,955
  • 18
  • 111
  • 232
  • Added how I am creating it in code, it's just for testing atm. – Jonny Apr 04 '14 at 07:21
  • Datastore entity inheritance doesn't work the way you think. You may also want to look Polymodel. I don't believe you will want to use StructuredProperty here – Tim Hoffman Apr 04 '14 at 07:29
  • If you do the entire heirarchy will be pulled in when you fetch the root object. – Tim Hoffman Apr 04 '14 at 07:29
  • "the entire heirarchy" - from using Polymodel or from using StructuredProperty? I think I am actually wanting to get the whole hierarchy. Maybe I can just restructure it all, scrap the "items" and use parent on subitems to point to parent folder... like here http://stackoverflow.com/a/10075612/129202 I want to save the root folder and be able to retrieve it with its whole structure in one go later... – Jonny Apr 04 '14 at 07:34
  • 1
    That won't scale if you have 100's or more entities. – Tim Hoffman Apr 04 '14 at 09:25
  • If you use parent keys (entity groups) then you can't re-arrange the hierarchy without rekeying everything. – Tim Hoffman Apr 04 '14 at 09:25

1 Answers1

1

use Polymodel with repeated KeyProperty

The StructuredProperty need to be changed to KeyProperty because:

TypeError: This StructuredProperty cannot use repeated=True because its model class (Mjlistitem) contains repeated properties (directly or indirectly).

The Model

from google.appengine.ext import ndb
from google.appengine.ext.ndb import polymodel

class Mjlistitem(polymodel.PolyModel):
    pass

class MjlistitemFolder(Mjlistitem):
    title = ndb.StringProperty()
    # StructuredProperty won't allow you to apply on nested model, use key property instead
    items = ndb.KeyProperty(kind=Mjlistitem, repeated=True)

class MjlistitemJob(Mjlistitem):
    pass

class MjlistitemJobGeneric(MjlistitemJob):
    jobtype = ndb.IntegerProperty()

class MjlistitemJobApp(MjlistitemJob):
    appleid = ndb.StringProperty()

The Usage

 def test():
    rootfolder = MjlistitemFolder(title="root")

    subfolder = MjlistitemFolder(title="Cool things")
    subfolder.items.append(MjlistitemJobApp(appleid="281796108").put()) # evernote
    subfolder.items.append(MjlistitemJobGeneric(jobtype=3).put()) # phone number
    subfolder.items.append(MjlistitemJobApp(appleid="327630330").put()) # dropbox

    rootfolder.items.append(MjlistitemJobGeneric(jobtype=15).put()) # passbook
    rootfolder.items.append(subfolder.put())
    rootfolder.items.append(MjlistitemJobGeneric(jobtype=17).put()) # appstore
    rootfolder.put()

another thing should watch out.

the repeated property won't perform well if there are too many items in one folder, so it would be better if 

class Mjlistitem(polymodel.PolyModel):
    parent = ndb.KeyProperty(kind="Mjlistitem")

    def set_parent(self, parent):
        assert isinstance(parent, MjlistitemFolder)
        self.parent = parent.key

class MjlistitemFolder(Mjlistitem):
    title = ndb.StringProperty()

class MjlistitemJob(Mjlistitem):
    pass

class MjlistitemJobGeneric(MjlistitemJob):
    jobtype = ndb.IntegerProperty()

class MjlistitemJobApp(MjlistitemJob):
    appleid = ndb.StringProperty()

The Usage

def test():
    rootfolder = MjlistitemFolder(title="root")
    rootfolder.put()

    subfolder = MjlistitemFolder(title="Cool things")
    subfolder.set_parent(rootfolder)
    subfolder.put()

    a = MjlistitemJobApp(appleid="281796108")
    a.set_parent(subfolder)
    a.put()

    b = MjlistitemJobGeneric(jobtype=3)
    b.set_parent(subfolder)
    b.put()
    c = MjlistitemJobApp(appleid="327630330")
    c.set_parent(subfolder)
    c.put()
lucemia
  • 6,349
  • 5
  • 42
  • 75