Python ORM to NumPy arrays

Question

I am building data simulation framework with numpy ORM, where it is much more convenient to work with classes and objects instead of numpy arrays directly. Nevertheless, output of the simulation should be numpy array. Also blockz is quite interesting as a backend here.

I would like to map all object attributes to numpy arrays. Thus, numpy arrays work like a column-oriented "persistent" storage for my classes. I also need to link "new" attributes to objects which I can calculate using numpy(pandas) framework. And then just link them to objects accordingly using the same back-end.

Is there any solution for such approach? Would you recommend any way to build it in a HPC way? I have found only django-pandas. PyTables is quite slow on adding new columns-attributes.

Something like (working on pointers to np_array):

class Instance()
    def __init__(self, np_array, np_position):
        self.np_array = np_array
        self.np_position = np_position

    def get_test_property():
        return(self.np_array[np_position])

    def set_test_property(value):
        self.np_array[np_position] = value

I would use pandas dataframes as much as possible. Might you be reinventing the wheel somewhat? — YXD, Nov 17 '14 at 13:34
@Mr E, So far I haven't find anyway to map values from pandas Series to object attributes. Am I missing something? — Cron Merdek, Nov 17 '14 at 14:55
I meant, can you use a [DataFrame](http://pandas.pydata.org/pandas-docs/dev/dsintro.html#dataframe) instead of writing your own class? — YXD, Nov 17 '14 at 16:00

score 0 · Accepted Answer · answered Nov 17 '14 at 22:47

In fact there is a way to change NumPy or bcolz arrays by reference. Simple example can be found in the following code.

a = np.arange(10)
class Case():
    def __init__(self, gcv_pointer):
        self.gcv = gcv_pointer

    def gcv(self):
        return(self.gcv)

    def gcv_set(self, value):
        self.gcv[:] = value
pass
#===============================================================================
# NumPy
#===============================================================================
caseList = []
for i in range(1, 10):
    case = Case(a[i-1:i])
    caseList.append(case)
gcvs = [case.GetGCV() for case in caseList]
caseList[1].SetGCV(5)
caseList[1].SetGCV(13)
caseList[1].gcv[:] = 6

setattr(caseList[1], 'dpd', a[5:6])

caseList[1].dpd
caseList[1].dpd[:] = 888

Python ORM to NumPy arrays

1 Answers1