Serializing Python Object to JSON Created with eulxml.xmlmap.XmlObject

Question

I have a set of python objects that I create using eulxml.xmlmap.XmlObject (I use this method primarily because I'm working with an eXistDB server and eulxml offers a pretty easy mapping function). I am able to successfully query my eXistDB and load an xquery result set into some python objects I've created. My problem is that I then want to be able to write these objects out as JSON when I pass them to the webserver (using Angular for the front end).

I've tried using jsonpickle but it seems as though eulxml is doing some kind of lazy loading magic. A standard call to jsonpickle to serialize my object to json gives me this result:

python code:

jsonpickle.encode(myObject)

result:

"py/object": "models.alcalaPage.AlcalaPage", "context": 
{"namespaces": 
   {"exist": "http://exist.sourceforge.net/NS/exist"}
}, 
"node":  {
   "py/object": "lxml.etree._Element", 
   "py/seq": [
       {"py/object": "lxml.etree._Comment", "py/seq": []},
       {"py/object": "lxml.etree._Element", "py/seq": []},
       ...
    ]
   }...

it seems to be only outputting the type of the attribute but not the value of the attribute itself. If I change my jsonpickle code to set unpickable=False, all I get is an empty set of json (meaning that the structure is there in terms of the right number of curly braces and brackets but there is literally no data. The json output is just curly braces and brackets).

I thought perhaps if I attempted to access a value in the field and then output the json that might work (at least for the field I accessed) but no luck. I get the same result as stated above (and yes I've double checked that there is data in the object itself).

I'm sort of at a loss at this point. I could migrate to something like BeautifulSoup but it means writing A LOT more code (eulxml lets me simply specify the xpath to the value I want to fill my attribute with and bing, I'm done). Is there something I'm missing with jsonpickle? Or is there another json package I should look at? Or maybe I'm making this way more difficult than I need to and there is some other way to query an eXistDB using python and then send the information to a front end application built using Angular. I'm open to suggestions.

I'll include samples of my code below (I won't include all of it because there are probably 10+ objects I'm working with):

Sample object code with eulxml:

import jsonpickle
from eulxml.xmlmap import XmlObject


class AlcalaBase(XmlObject):

    def to_xml(self):
        return self.serializeDocument(pretty=True)

    def to_json(self):
        return jsonpickle.encode(self)

from eulxml import xmlmap
from models.alcalaBase import AlcalaBase

class AlcalaPage(AlcalaBase):
    ROOT_NAME = 'page'
    id = xmlmap.StringField('pageID')
    archive_page_number = xmlmap.StringField('archivistsPageNumber')
    year = xmlmap.IntegerField('content/@yearID')

score 0 · Accepted Answer · answered Apr 25 '18 at 16:56

I was able to figure out the issue (sort of). So I'm posting it here in case others have the same issue.

The problem seems to be that the attributes are not added to dict so the actual values are not being output during the json process. I wrote my to_json() method in my base class in order to output the appropriate objects. NOTE: While I tried to keep this as generic as possible, it is somewhat specific to my data structure (in that I know what to expect in given scenarios and since I'm dealing with static data, I don't have to "future proof" the solution. Anyone adopting this code should adapt it to their given scenario.

from eulxml import xmlmap
import inspect
import lxml
import json as JSON

class AlcalaBase(xmlmap.XmlObject):

    def to_json(self, skipBegin=False):
        json = list()
        if not skipBegin:
            json.append('{')
        json.append(str.format('"{0}": {{', self.ROOT_NAME))
        for attr, value in inspect.getmembers(self):
            if (attr.find("_") == -1
                and attr.find("serialize") == -1
                and attr.find("context") == -1
                and attr.find("node") == -1
                and attr.find("schema") == -1):
                if type(value) is xmlmap.fields.NodeList:
                    if len(value) > 0:
                        json.append(str.format('"{0}": [', attr))
                        for v in value:
                            json.append(v.to_json())
                            json.append(",")
                        json = json[:-1]
                        json.append("]")
                    else:
                        json.append(str.format('"{0}": null', attr))
                elif (type(value) is xmlmap.fields.StringField
                        or type(value) is str
                        or type(value) is lxml.etree._ElementUnicodeResult):
                        value = JSON.dumps(value)
                        json.append(str.format('"{0}": {1}', attr, value))
                elif (type(value) is xmlmap.fields.IntegerField
                    or type(value) is int
                    or type(value) is float):
                    json.append(str.format('"{0}": {1}', attr, value))
                elif value is None:
                    json.append(str.format('"{0}": null', attr))
                elif type(value) is list:
                    if len(value) > 0:
                        json.append(str.format('"{0}": [', attr))
                        for x in value:
                            json.append(x)
                            json.append(",")
                        json = json[:-1]
                        json.append("]")
                    else:
                        json.append(str.format('"{0}": null', attr))
                else:
                    json.append(value.to_json(skipBegin=True))
                json.append(",")
        json = json[:-1]
        if not skipBegin:
            json.append('}')
        json.append('}')
        return ''.join(json)

Anything that inherits from this class will be able to serialise out to json. This also assumes that all object collections inherit from this base class (in my particular model, this is true so it's not an issue).

score 0 · Answer 2 · answered Oct 22 '19 at 12:56

Yes, jsonpickle calls the dict method so that it works, you can use the following in a meta class:

class MyXmlObject(XmlObject):
    @property
    def __dict__(self):
        d = { 'ROOT_NAME': self.ROOT_NAME }
        for key, value in self._fields.items():
            if isinstance(value, fields.Field):
                if isinstance(value, fields.NodeListField):
                    d[key] = [x.__dict__ for x in getattr(self, key)]
                elif isinstance(value, fields.NodeField):
                    d[key] = getattr(self, key).__dict__
                else:
                    d[key] = getattr(self, key)
        return d

so the dict method will directly return the values of fields

Serializing Python Object to JSON Created with eulxml.xmlmap.XmlObject

2 Answers2