Pandas - Unpack column of lists of varying lengths of tuples

Question

I would like to take a Pandas Dataframe named df which has an ID column and a lists column of lists that have variable number of tuples, all the tuples have the same length. Looks like this:

ID  list
1   [(0,1,2,3),(1,2,3,4),(2,3,4,NaN)]
2   [(Nan,1,2,3),(9,2,3,4)]
3   [(Nan,1,2,3),(9,2,3,4),(A,b,9,c),($,*,k,0)]

And I would like to unpack each list into columns 'A','B','C','D' representing the fixed positions in each tuple.

The result should look like:

ID  A   B   C   D
1   0   1   2   3
1   1   2   3   4
1   2   3   4   NaN
2   NaN 1   2   3
2   9   2   3   4
3   NaN 1   2   3
3   9   2   3   4
3   A   b   9   c
3   $   *   k   0

I have tried df.apply(pd.Series(list) but fails as the len of the list elements is different on different rows. Somehow need to unpack to columns and transpose by ID?

score 6 · Accepted Answer · answered Jul 31 '16 at 18:15

6

In [38]: (df.groupby('ID')['list']
            .apply(lambda x: pd.DataFrame(x.iloc[0], columns=['A', 'B', 'C', 'D']))
            .reset_index())
Out[38]: 
   ID  level_1    A  B  C    D
0   1        0    0  1  2    3
1   1        1    1  2  3    4
2   1        2    2  3  4  NaN
3   2        0  NaN  1  2    3
4   2        1    9  2  3    4
5   3        0  NaN  1  2    3
6   3        1    9  2  3    4
7   3        2    A  b  9    c
8   3        3    $  *  k    0

answered Jul 31 '16 at 18:15

chrisb

49,833
8
70
70

What kind of sorcery is this! Excellent! – clg4 Jul 31 '16 at 18:53
1

.@chrisb When I run this I get >> DataFrame constructor not properly called! – Merlin Jul 31 '16 at 20:27
Works perfectly for me. Version of pandas? – clg4 Aug 02 '16 at 01:41

score 0 · Answer 2 · answered Oct 15 '17 at 08:33

A vectorized way would be

In [2237]: dff = pd.DataFrame(np.concatenate(df['list'].values), columns=list('ABCD'))

In [2238]: dff['ID'] = df.ID.repeat(df['list'].str.len()).values

In [2239]: dff
Out[2239]:
     A    B    C    D  ID
0  0.0  1.0  2.0  3.0   1
1  1.0  2.0  3.0  4.0   1
2  2.0  3.0  4.0  nan   1
3  nan  1.0  2.0  3.0   2
4  9.0  2.0  3.0  4.0   2
5  nan    1    2    3   3
6    9    2    3    4   3
7    A    b    9    c   3
8    $    *    k    0   3

Details

In [2240]: df
Out[2240]:
   ID                                               list
0   1       [(0, 1, 2, 3), (1, 2, 3, 4), (2, 3, 4, nan)]
1   2                     [(nan, 1, 2, 3), (9, 2, 3, 4)]
2   3  [(nan, 1, 2, 3), (9, 2, 3, 4), (A, b, 9, c), (...

Pandas - Unpack column of lists of varying lengths of tuples

2 Answers2