Trouble using ttest_ind on pandas dataframe

Question

I'm currently working with a dataframe in the form:

import pandas as pd
import numpy as np
df = pd.DataFrame([['A', 12.1, 11.4, 15.1, 9.9], ['B', 8.3, 10.3, 16.6, 7.8], ['B', 7.8, 11.1, 16.3, 8.4], 
                   ['B', 8.6, 10.9, 16.4, 8.1], ['A', 12.25, 11.6, 16.25, 8.9], ['B', 8.13, 11.6, 16.7, 7.4]
                  ], columns = ['Symbol', 'C1','C2', 'C3', 'C4'])

And a list of lists that includes the comparisons across columns I'd like to make:

lst = [['C1','C2'], ['C1','C3'], ['C3','C4']]

I'm trying to calculate the difference of the means (repeated symbols) for each comparison as well as do a ttest_ind then return a new dataframe with the results that would look like:

df2 = pd.DataFrame([['A', 0.675, 'pval here', -3.5, 'pval here',6.275,'pval here'], 
                    ['B', -2.7675, 'pval here', -8.2925, 'pval here', 8.575 , 'pval here']], 
                   columns = ['Symbol', 'C1-C2','C1-C2 pval', 'C1-C3', 'C1-C3 pval', 'C3-C4','C3-C4 pval'])

Finding the difference between the means is somewhat straightforward using groupby to get the means then loop over the pairs of the list as:

df = df.groupby('Symbol').agg(np.mean)
for pair in lst:
    df[pair[0]+'-'+pair[1]] = df[pair[0]] - df[pair[1]]

But I've been stuck in applying ttest_ind and then returning the p-vaule into another column.

Any assistance is greatly appreciated.

score 1 · Answer 1 · answered Mar 16 '19 at 14:03

You can use the method scipy.stats.ttest_ind for this.

The method returns a tuple with (t-statistic, p-value). So we can access the p-value with index 1 like following:

# Dataframe I start with, given by OP
df = df.groupby('Symbol').agg(np.mean)
for pair in lst:
    df[pair[0]+'-'+pair[1]] = df[pair[0]] - df[pair[1]]

print(df)
             C1      C2      C3     C4   C1-C2   C1-C3  C3-C4
Symbol                                                       
A       12.1750  11.500  15.675  9.400  0.6750 -3.5000  6.275
B        8.2075  10.975  16.500  7.925 -2.7675 -8.2925  8.575

from scipy.stats import ttest_ind

lst = [['C1','C2'], ['C1','C3'], ['C3','C4']]

df_group = df.groupby('Symbol').sum()

for l in lst:
    df_group[l[0]+'-'+l[1]+' pval'] = ttest_ind(df_group[l[0]], df_group[l[1]])[1]

# Drop columns not needed anymore    
df = df_group.drop(['C1', 'C2', 'C3', 'C4'],axis=1)

# Sort columns to get expected output
df = df.reindex(sorted(df.columns), axis=1).reset_index()

print(df)
  Symbol   C1-C2  C1-C2 pval   C1-C3  C1-C3 pval  C3-C4  C3-C4 pval
0      A  0.6750    0.653228 -3.5000    0.100586  6.275    0.012706
1      B -2.7675    0.653228 -8.2925    0.100586  8.575    0.012706

Thank you for your response Erfan. I think this gets me closer to a possible solution, however, I would expect the p-values to be different for 'A' and 'B' in each comparison so it looks like the ttest was done on everything rather than each group? — user987443, Mar 17 '19 at 14:32

score 0 · Accepted Answer · answered Mar 17 '19 at 18:47

Eureka!

Starting from what I posted in the question:

import pandas as pd
import numpy as np
from scipy.stats import ttest_ind
df = pd.DataFrame([['A', 12.1, 11.4, 15.1, 9.9], ['B', 8.3, 10.3, 16.6, 7.8], ['B', 7.8, 11.1, 16.3, 8.4], 
                   ['B', 8.6, 10.9, 16.4, 8.1], ['A', 12.25, 11.6, 16.25, 8.9], ['B', 8.13, 11.6, 16.7, 7.4]
                  ], columns = ['Symbol', 'C1','C2', 'C3', 'C4'])
lst = [['C1','C2'], ['C1','C3'], ['C3','C4']]

I first find the difference between the pairs in the list:

df2 = df.groupby('Symbol').agg(np.mean)
for pair in lst:
    df2[pair[0]+'-'+pair[1]] = df2[pair[0]] - df2[pair[1]]

Then I make list of the 'Symbols' and loop through it to make a new dataframes containing only the same symbol for the t-test which I then append to the dataframe that has the differences:

lst2 = list(set(df.Symbol))
for item in lst2:
    df3 = df[df.Symbol == item]
    for pair in lst:
        df2.loc[item, pair[0]+'-'+pair[1]+' pval'] = ttest_ind(df3[pair[0]], df3[pair[1]])[1]

This results in the dataframe (df2):

    C1  C2  C3  C4  C1-C2   C1-C3   C3-C4   C1-C2 pval  C1-C3 pval  C3-C4 pval
Symbol                                      
A   12.1750 11.500  15.675  9.400   0.6750  -3.5000 6.275   0.032625    2.636815e-02    1.442745e-02
B   8.2075  10.975  16.500  7.925   -2.7675 -8.2925 8.575   0.000124    9.784611e-09    2.636731e-08

To which then I can drop the columns with the averages (C1, C2...) to get my desired output.

Trouble using ttest_ind on pandas dataframe

2 Answers2