converting 3D pandas dataframe to 2d

Question

I have a 3D dataframe with 2 levels of index and one column that looks like this:

          col1
0   0      67.23
0   1      7382
0   2      43
    .
    .
0   8002   54
0   8003   87
1   0      348
1   1      83
1   2      234
    .
    .
1   8002   23
1   8003   87
....
9   0      348
9   1      833
9   2      43433
    .
    .
9   8002   23
9   8003   87

The first level has 10 indexes and each of the second levels has 8004 elements. I need to reshape it to a 2d dataframe as follow:

    0 | 1 | 2 | 3 | ... | 8000 | 8001 | 8002 | 8003
--------------------------------------------------
0     |   |   |   |     |      |      |      | 
1     |   |   |   |     |      |      |      |       
2     |   |   |   |     |      |      |      |       
.     |   |   |   |     |      |      |      | 
.     |   |   |   |     |      |      |      | 
8     |   |   |   |     |      |      |      |       
9     |   |   |   |     |      |      |      |       

To do that I created a new index: new_idx = pd.Index(range(0,10))

and new columns as: cols = range(0,8004)

and then tried to use pivot as follow:

2d_df = df.pivot(index=new_idx, columns=cols, values='var1')

But that gives me an error: Int64Index([0,1,2,3,4,5,6,7,8,9], dtype='int64') not in index. How should I do this?

Answer 1

Try using DataFrame.unstack - bear in mind that this will not work if you have duplicate indices.

df_2d = df.unstack(1)

Then fix column level using:

  df_2d.columns = df_2d.columns.droplevel(0)