4

I am trying to compute percentile of two columns using the pandas qcut method like below:

my_df['float_col_quantile'] = pd.qcut(my_df['float_col'], 100, labels=False)
my_df['int_col_quantile'] = pd.qcut(my_df['int_col'].astype(float), 100, labels=False)

The column float_col_quantile works fine, but the column int_col_quantile has the following error. Any idea what I did wrong here? And how can I fix this problem? Thanks!

ValueError                                Traceback (most recent call last)
<ipython-input-19-b955e0b00953> in <module>()
      1 my_df['float_col_quantile'] = pd.qcut(my_df['float_col'], 100, labels=False)
----> 2 my_df['int_col_quantile'] = pd.qcut(my_df['int_col'].astype(float), 100, labels=False)


/usr/local/lib/python3.4/dist-packages/pandas/tools/tile.py in qcut(x, q, labels, retbins, precision)
    173     bins = algos.quantile(x, quantiles)
    174     return _bins_to_cuts(x, bins, labels=labels, retbins=retbins,
--> 175                          precision=precision, include_lowest=True)
    176 
    177 

/usr/local/lib/python3.4/dist-packages/pandas/tools/tile.py in _bins_to_cuts(x, bins, right, labels, retbins, precision, name, include_lowest)
    192 
    193     if len(algos.unique(bins)) < len(bins):
--> 194         raise ValueError('Bin edges must be unique: %s' % repr(bins))
    195 
    196     if include_lowest:

ValueError: Bin edges must be unique: array([  1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,
         1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,   1.,
         1.,   1.,   1.,   1.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,
         2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,
         2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,
         2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,   2.,
         2.,   2.,   2.,   4.,   4.,   4.,   4.,   4.,   4.,   4.,   4.,
         4.,   4.,   4.,   4.,   4.,   4.,   4.,   4.,   4.,   4.,   4.,
         4.,   4.,   4.,   4.,   4.,   4.,   4.,   4.,   8.,   8.,   8.,
         8.,  10.])
Edamame
  • 23,718
  • 73
  • 186
  • 320
  • 2
    Possible duplicate of [How to qcut with non unique bin edges?](https://stackoverflow.com/questions/20158597/how-to-qcut-with-non-unique-bin-edges) There are some workarounds there and also note that updating pandas to version 0.20 might solve your problem – JohnE Jun 19 '17 at 21:07

1 Answers1

7

The problem is pandas.qcut chooses the bins so that you have the same number of records in each bin/quantile, but the same value cannot fall in multiple bins/quantiles.

Here is a list of solutions.

luca
  • 7,178
  • 7
  • 41
  • 55