3

I have been using polars but it seems like it lacks qcut functionality as pandas do.

I am not sure about the reason but is it possible to achieve the same effect as pandas qcut using current available polars functionalities?

The following shows an example about what I can do with pandas qcut.

import pandas as pd

data = pd.Series([11, 1, 2, 2, 3, 4, 5, 1, 2, 3, 4, 5])
pd.qcut(data, [0, 0.2, 0.4, 0.6, 0.8, 1], labels=['q1', 'q2', 'q3', 'q4', 'q5'])

The results are as follows:

0     q5
1     q1
2     q1
3     q1
4     q3
5     q4
6     q5
7     q1
8     q1
9     q3
10    q4
11    q5
dtype: category

So, I am curious how can I get the same result by using polars?

Thanks for your help.

lebesgue
  • 837
  • 4
  • 13

1 Answers1

6

Update: Series.qcut was added in polars version 0.16.15

data = pl.Series([11, 1, 2, 2, 3, 4, 5, 1, 2, 3, 4, 5])
 
data.qcut([0.2, 0.4, 0.6, 0.8], labels=['q1', 'q2', 'q3', 'q4', 'q5'], maintain_order=True)

shape: (12, 3)
┌──────┬─────────────┬──────────┐
│      ┆ break_point ┆ category │
│ ---  ┆ ---         ┆ ---      │
│ f64  ┆ f64         ┆ cat      │
╞══════╪═════════════╪══════════╡
│ 11.0 ┆ inf         ┆ q5       │
│ 1.0  ┆ 2.0         ┆ q1       │
│ 2.0  ┆ 2.0         ┆ q1       │
│ 2.0  ┆ 2.0         ┆ q1       │
│ …    ┆ …           ┆ …        │
│ 2.0  ┆ 2.0         ┆ q1       │
│ 3.0  ┆ 3.6         ┆ q3       │
│ 4.0  ┆ 4.8         ┆ q4       │
│ 5.0  ┆ inf         ┆ q5       │
└──────┴─────────────┴──────────┘

Old answer:

From what I can tell .qcut() uses the linear quantile of the bin values?

If so, you could implement that part "manually":

import polars as pl

data = pl.Series([11, 1, 2, 2, 3, 4, 5, 1, 2, 3, 4, 5])
bins = [0.2, 0.4, 0.6, 0.8]
labels = ["q1", "q2", "q3", "q4", "q5"]

pl.cut(data, bins=[data.quantile(val, "linear") for val in bins], labels=labels)

shape: (12, 3)
┌──────┬─────────────┬──────────┐
│      | break_point | category │
│ ---  | ---         | ---      │
│ f64  | f64         | cat      │
╞══════╪═════════════╪══════════╡
│ 1.0  | 2.0         | q1       │
│ 1.0  | 2.0         | q1       │
│ 2.0  | 2.0         | q1       │
│ 2.0  | 2.0         | q1       │
│ 2.0  | 2.0         | q1       │
│ 3.0  | 3.6         | q3       │
│ 3.0  | 3.6         | q3       │
│ 4.0  | 4.8         | q4       │
│ 4.0  | 4.8         | q4       │
│ 5.0  | inf         | q5       │
│ 5.0  | inf         | q5       │
│ 11.0 | inf         | q5       │
└──────┴─────────────┴──────────┘
jqurious
  • 9,953
  • 1
  • 4
  • 14
  • One follow up question, is there anyway I can do pl.cut in either select or agg context? Thanks! – lebesgue Jan 25 '23 at 02:58
  • As it works on a Series - It's somewhat awkward: `df.with_columns(pl.col("value").map(lambda col: pl.cut(col, bins=[...], labels=[...]).to_struct("")).over("group").suffix("_cut")).drop("value").unnest("value_cut")` – jqurious Jan 25 '23 at 03:20
  • Thanks! It worked if I put in pre-calculated bins, say bins=[1, 3, 5]. But it does not work when I put in bins=[pl.col("x").quantile(val, "linear") for val in bins] (here bins = [0.2, 0.4, 0.6, 0.8] as in my example). Is there any way around? – lebesgue Jan 25 '23 at 04:25
  • By the way, looks like I cannot do pl.cut just within select context, i.e., this failed: df.with_columns(pl.col("value").map(lambda col: pl.cut(col, bins=[...], labels=[...]).to_struct(""))). How am I supposed to do it correctly? – lebesgue Jan 25 '23 at 04:30
  • 1
    Perhaps it deserves its own question - or you can edit this one. Something like: `df.with_columns(pl.col("x").map(lambda col: pl.cut(col, bins=[col.quantile(val, "linear") for val in col], labels=["q1", "q2", "q2"]).to_struct(""))).unnest("x") ` – jqurious Jan 25 '23 at 04:49
  • Great! Thanks a lot. I may ask a separate question for my remaining questions. – lebesgue Jan 25 '23 at 05:10
  • Good idea. Also, I just noticed that [`pl.cut()` does not retain order](https://github.com/pola-rs/polars/issues/4286) - so my previous attempts at using `.map()` here are broken - and should be disregarded. – jqurious Jan 25 '23 at 05:52