Adding a pickle-able attribute to a subclass of numpy.ndarray

Question

I would like to add a property (.csys) to a subclass of numpy.ndarray:

import numpy as np

class Point(np.ndarray):

    def __new__(cls, arr, csys=None):
        obj = np.asarray(arr, dtype=np.float64).view(cls)
        obj._csys = csys
        return obj

    def __array_finalize__(self, obj):
        if obj is None: return
        self._csys = getattr(obj, '_csys', None)

    @property
    def csys(self):
        print('Getting .csys')
        return self._csys

    @csys.setter
    def csys(self, csys):
        print('Setting .csys')
        self._csys = csys

However, when I run this test code:

pt = Point([1, 2, 3])
pt.csys = 'cmm'
print("pt.csys:", pt.csys)

# Pickle, un-pickle, and check again
import pickle

pklstr = pickle.dumps(pt)
ppt = pickle.loads(pklstr)

print("ppt.csys:", ppt.csys)

it appears that the attribute cannot be pickled:

Setting .csys
Getting .csys
pt.csys: cmm
Getting .csys
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
C:\Rut\Vanes\bin\pointtest.py in <module>()
     39     ppt = pickle.loads(pklstr)
     40 
---> 41     print("ppt.csys:", ppt.csys)

C:\Rut\Vanes\bin\point.py in csys(self)
     15     def csys(self):
     16         print('Getting .csys')
---> 17         return self._csys
     18 
     19     @csys.setter

AttributeError: 'Point' object has no attribute '_csys'

I tried doing the same thing without using decorators (e.g. defining get_csys() and set_csys(), plus csys = property(__get_csys, __set_csys), but had the same result with that.

I'm using numpy 1.13.3 under Python 3.6.3

Cannot reproduce... this works for me on python 3.6.0 and numpy 1.14.0. Are you sure you're using python3? — cs95, May 17 '18 at 04:48
The opening banner when I run Jupyter QtConsole 4.3.1 tells me so, yes. Is it a numpy 1.13 problem? — subnivean, May 17 '18 at 04:58
I don't know. You should try to upgrade and see if it persists. — cs95, May 17 '18 at 04:59
[This](https://stackoverflow.com/questions/26598109/preserve-custom-attributes-when-pickling-subclass-of-numpy-array) seems to be related. In conclusion, subclassing `__reduce__` and `__setstate__` is needed to tell the ndarray subclass how to pickle itself in diff to a normal ndarray. — Jeronimo, May 17 '18 at 05:25
@Jeronimo Yes, thanks, just found that myself off to the right and it works perfectly. I'll write that up as my own answer soon. — subnivean, May 17 '18 at 05:34

score 0 · Accepted Answer · answered May 17 '18 at 11:43

This question has already been asked and answered here. In a nutshell, numpy uses __reduce__ and __setstage__ to pickle itself. The overrides, adapted to the case above, look like this:

def __reduce__(self):
    # Get the parent's __reduce__ tuple
    pickled_state = super().__reduce__()
    # Create our own tuple to pass to __setstate__
    new_state = pickled_state[2] + (self._csys,)
    # Return a tuple that replaces the parent's __setstate__ tuple with our own
    return (pickled_state[0], pickled_state[1], new_state)

def __setstate__(self, state):
    self._csys = state[-1]  # Set the _csys attribute
    # Call the parent's __setstate__ with the other tuple elements.
    super().__setstate__(state[0:-1])

Also note that the getter and setter methods (under the @property and @csys.getter decorators, respectively) are not strictly required in this simple case. If they are dispensed with, access .csys directly, rather than through the 'private' ._csys attribute.

Adding a pickle-able attribute to a subclass of numpy.ndarray

1 Answers1