4D array numpy into pandas

Question

I'd like to transform my numpy array ( shape=(27, 77, 77) ) :

   [[1., 1., 1., ..., 1., 1., 1.],
    [1., 1., 1., ..., 1., 1., 1.],
    [1., 1., 1., ..., 1., 1., 1.],
    ...,
    [1., 1., 1., ..., 2., 2., 2.],
    [1., 1., 1., ..., 2., 2., 2.],
    [1., 1., 1., ..., 1., 2., 2.]],

   ...,

   [[1., 1., 1., ..., 1., 1., 0.],
    [1., 1., 1., ..., 1., 1., 0.],
    [1., 1., 1., ..., 1., 1., 0.],
    ...,
    [1., 1., 1., ..., 1., 1., 1.],
    [1., 1., 1., ..., 1., 1., 1.],
    [1., 1., 1., ..., 1., 1., 1.]])

into a pandas dataframe with columns 'x' = index 2 (right), 'y' = index 1 (down), 'z' = index 0 ( the 27 "different" arrays) and 'v' = the values in it. df.columns=['x','y','z','v']

I'm relatively new to python, do you know how I should code this?

Thanks !

Welcome to SO! Can you produce a MWE? https://stackoverflow.com/help/minimal-reproducible-example . It is not clear what you are exactly looking for. — SKPS, Apr 01 '21 at 21:41
Such a table will have 160,000 rows. Is that really what you want? — Tim Roberts, Apr 01 '21 at 21:49

score 2 · Accepted Answer · answered Apr 01 '21 at 21:53

This does it in a primitive way.

import numpy as np
import pandas as pd

data = np.ones( (27,77,77) )

rows = []
for i,plane in enumerate(data):
    for j,row in enumerate(plane):
        for k,col in enumerate(row):
            rows.append( [k,j,i,col] )

df = pd.DataFrame( rows, columns=['x','y','z','val'])
print(df)

Output:

C:\tmp>python x.py
         x   y   z  val
0        0   0   0  1.0
1        1   0   0  1.0
2        2   0   0  1.0
3        3   0   0  1.0
4        4   0   0  1.0    
...     ..  ..  ..  ...
160078  72  76  26  1.0
160079  73  76  26  1.0
160080  74  76  26  1.0
160081  75  76  26  1.0
160082  76  76  26  1.0

[160083 rows x 4 columns]

C:\tmp>

score 2 · Answer 2 · answered Apr 01 '21 at 22:26

As a "simple" one-liner for arbitrary number of dimensions:

>>> import itertools as it; import numpy as np; import pandas as pd

# analogous test data
>>> arr = np.random.rand(27, 77, 77)

# np.nditer(arr) + v.item() using no additional memory
# arr.flatten() is slightly faster but uses additional memory
>>> df = pd.DataFrame(data=[(*axes, v.item()) for axes, v in zip(it.product(*[range(i) for i in arr.shape]), np.nditer(arr))], columns=tuple('xyzv'))
>>> df
         x   y   z         v
0        0   0   0  0.375027
1        0   0   1  0.511405
2        0   0   2  0.645937
3        0   0   3  0.229538
4        0   0   4  0.274867
...     ..  ..  ..       ...
160078  26  76  72  0.404251
160079  26  76  73  0.010852
160080  26  76  74  0.048079
160081  26  76  75  0.426528
160082  26  76  76  0.723565

4D array numpy into pandas

2 Answers2