Pandas groupby quantile values

Question

I tried to calculate specific quantile values from a data frame, as shown in the code below. There was no problem when calculate it in separate lines.

When attempting to run last 2 lines, I get the following error:

AttributeError: 'SeriesGroupBy' object has no attribute 'quantile(0.25)'

How can I fix this?

import pandas as pd
df = pd.DataFrame(
    {
        'x': [0, 1, 0, 1, 0, 1, 0, 1],
        'y': [7, 6, 5, 4, 3, 2, 1, 0],
        'number': [25000, 35000, 45000, 50000, 60000, 70000, 65000, 36000]
    }
)
f = {'number': ['median', 'std', 'quantile']}
df1 = df.groupby('x').agg(f)
df.groupby('x').quantile(0.25)
df.groupby('x').quantile(0.75)

# code below with problem:
f = {'number': ['median', 'std', 'quantile(0.25)', 'quantile(0.75)']}
df1 = df.groupby('x').agg(f)

Related: https://stackoverflow.com/a/17578653/4686625 – miradulo Dec 04 '17 at 16:55 — miradulo, Dec 04 '17 at 16:55

score 54 · Accepted Answer · edited Nov 17 '21 at 11:42

54

I prefer def functions

def q1(x):
    return x.quantile(0.25)

def q3(x):
    return x.quantile(0.75)

f = {'number': ['median', 'std', q1, q3]}
df1 = df.groupby('x').agg(f)
df1
Out[1643]: 
  number                            
  median           std     q1     q3
x                                   
0  52500  17969.882211  40000  61250
1  43000  16337.584481  35750  55000

edited Nov 17 '21 at 11:42

igorkf

3,159
2
22
31

answered Dec 04 '17 at 16:32

BENY

317,841
20
164
234

1

I tried lambda didn't work thought of function you posted already – Bharath M Shetty Dec 04 '17 at 16:34
Nope since lambda will return a column name lambda it would lead to specification error. Wonder how you would make it – Bharath M Shetty Dec 04 '17 at 16:35
@Dark something like fl = lambda x : x.quantile(0.25) fl.__name__ = 'f1' – BENY Dec 04 '17 at 16:38
1

I meant with this one `f = {'number': ['median', 'std', lambda x: x.quantile(0.25), lambda x: x.quantile(0.75)]}` , certainly didn't thinkg of assigning to a variable. – Bharath M Shetty Dec 04 '17 at 16:39
@Dark this one can not , return the duplicated lambda name :-) , you are right – BENY Dec 04 '17 at 16:41
Just a note: Q2 is the median, not the 75th percentile. Your second function should be named `q3` – igorkf Jul 28 '21 at 01:54

score 17 · Answer 2 · answered Oct 24 '19 at 06:54

@WeNYoBen's answer is great. There is one limitation though, and that lies with the fact that one needs to create a new function for every quantile. This can be a very unpythonic exercise if the number of quantiles become large. A better approach is to use a function to create a function, and to rename that function appropriately.

def rename(newname):
    def decorator(f):
        f.__name__ = newname
        return f
    return decorator

def q_at(y):
    @rename(f'q{y:0.2f}')
    def q(x):
        return x.quantile(y)
    return q

f = {'number': ['median', 'std', q_at(0.25) ,q_at(0.75)]}
df1 = df.groupby('x').agg(f)
df1

Out[]:
number                            
  median           std  q0.25  q0.75
x                                   
0  52500  17969.882211  40000  61250
1  43000  16337.584481  35750  55000

The rename decorator renames the function so that the pandas agg function can deal with the reuse of the quantile function returned (otherwise all quantiles results end up in columns that are named q).

Nice! No need for the decorator, though: you can just rename `q` before returning it from `q_at`. — z0r, Oct 28 '20 at 12:02

igorkf · Answer 3 · 2022-02-10T00:48:26.237

3

There's a nice way if you want to give names to aggregated columns:

df1.groupby('x').agg(
    q1_foo=pd.NamedAgg('number', q1),
    q2_foo=pd.NamedAgg('number', q2)
)

where q1 and q2 are functions.

Or even simpler:

df1.groupby('x').agg(
    q1_foo=('number', q1),
    q2_foo=('number', q2)
)

edited Feb 10 '22 at 00:48

answered Jul 28 '21 at 01:57

igorkf

3,159
2
22
31

Pandas groupby quantile values

3 Answers3

Linked