Aggregate a bunch of different data in a single groupby with multiple columns

Question

I have large dataframe of data in Pandas (let's say of courses at a university) looking like:

ID          name        credits     enrolled    ugrad/grad  year        semester
1           Math        4           62          ugrad       2016        Fall
2           History     3           15          ugrad       2016        Spring
3           Adv Math    3           8           grad        2017        Fall
...

and I want to group it by year and semester, and then get a bunch of different aggregate data on it, but all at one time if I can. For example, I want a total count of courses, count of only undergraduate courses, and sum of enrollment for a given semester. I can do each of these individually using value_counts, but I'd like to get an output such as:

year        semester    count       count_ugrad total_enroll
2016        Fall        #           #           #
            Spring      #           #           #
2017        Fall        #           #           #
            Spring      #           #           #
...

Is this possible?

score 3 · Accepted Answer · answered Jun 20 '18 at 18:38

Here I added a new subject for Python and provided as a dict to load into dataframe.

Solution is a combination of the agg() method on a groupby, where the aggregations are provided in a dictionary, and then the use of a custom aggregation function for your ugrad requirement:

def my_custom_ugrad_aggregator(arr):
    return sum(arr == 'ugrad')

dict = {'name': {0: 'Math', 1: 'History', 2: 'Adv Math', 3: 'Python'}, 'year': {0: 2016, 1: 2016, 2: 2017, 3: 2017}, 'credits': {0: 4, 1: 3, 2: 3, 3: 4}, 'semester': {0: 'Fall', 1: 'Spring', 2: 'Fall', 3: 'Spring'}, 'ugrad/grad': {0: 'ugrad', 1: 'ugrad', 2: 'grad', 3: 'ugrad'}, 'enrolled': {0: 62, 1: 15, 2: 8, 3: 8}, 'ID': {0: 1, 1: 2, 2: 3, 3: 4}}
df  =pd.DataFrame(dict)


   ID  credits  enrolled      name semester ugrad/grad  year
0   1        4        62      Math     Fall      ugrad  2016
1   2        3        15   History   Spring      ugrad  2016
2   3        3         8  Adv Math     Fall       grad  2017
3   4        4         8   Python   Spring       ugrad  2017

print df.groupby(['year','semester']).agg({'name':['count'],'enrolled':['sum'],'ugrad/grad':my_custom_ugrad_aggregator})

gives:

               name                 ugrad/grad enrolled
              count my_custom_ugrad_aggregator      sum
year semester                                          
2016 Fall         1                          1       62
     Spring       1                          1       15
2017 Fall         1                          0        8
     Spring       1                          1        8

score 1 · Answer 2 · answered Jun 20 '18 at 18:42

Use agg with dictionary on how to rollup/aggregate each column:

df_out = df.groupby(['year','semester'])[['enrolled','ugrad/grad']]\
           .agg({'ugrad/grad':lambda x: (x=='ugrad').sum(),'enrolled':['sum','size']})\
           .set_axis(['Ugrad Count','Total Enrolled','Count Courses'], inplace=False, axis=1)

df_out

Output:

               Ugrad Count  Total Enrolled  Count Courses
year semester                                            
2016 Fall                1              62              1
     Spring              1              15              1
2017 Fall                0               8              1

Aggregate a bunch of different data in a single groupby with multiple columns

2 Answers2