0

I'm using the OptBinning package to bin some numeric data. I'm following this example to do this. And from this tutorial I read that "... the best way to view BinningProcess is as a wrapper for OptimalBinning", which implies that they should both give the same outputs. However, I'm seeing they give different outputs for some features and the same for others. Why is this the case? Below is an example showing how the two methods lead to the same output for 'mean radius' but not 'worst radius' using the breast cancer data in sklearn.

import pandas as pd
import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

from optbinning import BinningProcess
from optbinning import OptimalBinning

# Load data
data = load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)

# Bin 'mean radius' data using OptimalBinning method
var = 'mean radius'

x = df[var]
y = data.target
optb = OptimalBinning(name=var, dtype="numerical")
optb.fit(x, y)
binning_table = optb.binning_table
binning_table.build()['WoE']
0         -3.12517
1         -2.71097
2         -1.64381
3        -0.839827
4        -0.153979
5          2.00275
6          5.28332
7                0
8                0
Totals            
Name: WoE, dtype: object

# Bin 'mean radius' using BinningProcess method
var = ['mean radius']
bc_pipe = Pipeline([('WOE Binning', BinningProcess(variable_names=var))])
preprocessor = ColumnTransformer([('Numeric Pipeline', bc_pipe, var)], remainder='passthrough')
preprocessor.fit(df, y)
df_processed = preprocessor.transform(df)
df_processed = pd.DataFrame(df_processed, columns=df.columns)
df_processed[var[0]].unique()
array([ 5.28332344, -3.12517033, -1.64381421, -0.15397917,  2.00275405,
       -0.83982705, -2.71097154])
## We see that the Weight of Evidence (WoE) values are the same for 'mean radius' using both methods (except for the 0's, which we can ignore for now)

# Bin 'worst radius' using OptimalBinning process
var = 'worst radius'
x = df[var]
y = data.target

optb = OptimalBinning(name=var, dtype="numerical")
optb.fit(x, y)

binning_table = optb.binning_table
binning_table.build()['WoE']
0         -4.56645
1          -2.6569
2        -0.800606
3        -0.060772
4          1.61976
5           5.5251
6                0
7                0
Totals            
Name: WoE, dtype: object

# Bin 'worst radius' using BinningProcess method
var = ['worst radius']
bc_pipe = Pipeline([('WOE Binning', BinningProcess(variable_names=var))])
preprocessor = ColumnTransformer([('Numeric Pipeline', bc_pipe, var)], remainder='passthrough')
preprocessor.fit(df, y)
df_processed = preprocessor.transform(df)
df_processed = pd.DataFrame(df_processed, columns=df.columns)
df_processed[var[0]].unique()
array([0.006193 , 0.003532 , 0.004571 , 0.009208 , 0.005115 , 0.005082 ,
       0.002179 , 0.005412 , 0.003749 , 0.01008  , 0.003042 , 0.004144 ,
       0.01284  , 0.003002 , 0.008093 , 0.005466 , 0.002085 , 0.004142 ,
       0.001997 , 0.0023   , 0.002425 , 0.002968 , 0.004394 , 0.001987 ,
       0.002801 , 0.007444 , 0.003711 , 0.004217 , 0.002967 , 0.003742 ,
       0.00456  , 0.005667 , 0.003854 , 0.003896 , 0.003817 , ... ])
## We now see that for 'worst radius' the two WoE's are not the same. Why?
Gaurav Bansal
  • 5,221
  • 14
  • 45
  • 91

1 Answers1

1

I think the problem is due to the default behaviour of ColumnTransformer option remainder="passthrough". The remaining columns are concatenated, and that's why the position of the transformed variables changes. If you look at the dataframe, the first column contains the WoE values of the feature "worst radius". As an example, please try the following:

binning_process = BinningProcess(variable_names=var)
binning_process.fit(df[var], y)
np.unique(binning_process.transform(df[var]).values)

The binning process, as expected, will return the same WoE values. See also: https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html

By default, only the specified columns in transformers are transformed and combined in the output, and the non-specified columns are dropped. (default of 'drop'). By specifying remainder='passthrough', all remaining columns that were not specified in transformers will be automatically passed through. This subset of columns is concatenated with the output of the transformers.

Guillermo Navas-Palencia
  • 61
  • 3