1

Accumulation stage

In the script, the same-sized data matrix X is re-estimated by some model (here just a random number generator (RNG)) and accumulated/saved in a matrix Y over the course of a finite number of trials t.

import numpy as np
from numpy.random import random
import pandas as pd

k = 3 #shape
t = 5 #trials
Y = np.zeros((t,k,k)) 

for i in range(5):
    X = random((k,k))   #2D estimate
    X = pd.DataFrame(X)
    Y[i,:,:] = X        #3D tensor

Reduction stage

Afterwards, how do I then apply an element-wise reduction of all accumulated 2d X arrays inside the 3d Y tensor into a single 2d matrix Z that is the same shape as X? An example reduction is the average of all the individual X elements reduced into Z:

Z[0,0] = average of: {the first Z[0,0], second Z[0,0], ... , fifth Z[0,0]}

I'd prefer no element-by-element loops, if possible. I showed the accumulation stage using numpy arrays because I don't think pandas DataFrames can be 3d tensors, being restricted to 2d inputs only, but can the arithmetic reduction stage (averaging across accumulated arrays) be done as a pandas DataFrame operation?

develarist
  • 1,224
  • 1
  • 13
  • 34

1 Answers1

1

Is this what you're looking for?

Toy example:

test = np.arange(12).reshape(-1,2,3)
array([[[ 0,  1,  2],
        [ 3,  4,  5]],

       [[ 6,  7,  8],
        [ 9, 10, 11]]])

Solution

np.apply_over_axes(np.mean,test,0).reshape(test.shape[1],test.shape[2])
array([[3., 4., 5.],
    [6., 7., 8.]])

And IIRC I think you're correct, pandas cannot really handle 3d tensors unless you mess about with Multi-indices, so personally I would rather deal with this operation in numpy first and then convert it to a dataframe. You can convert dataframes to numpy via to_numpy().

Jeff
  • 610
  • 4
  • 12