8

I have a dataframe with all categorical columns which i am encoding using a oneHotEncoder from sklearn.preprocessing. My code is as below:

from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline


steps = [('OneHotEncoder', OneHotEncoder(handle_unknown ='ignore')) ,('LReg', LinearRegression())]

pipeline = Pipeline(steps)

As seen inside the OneHotEncoder the handle_unknown parameter takes either error or ignore. I want to know if there is a way to selectively ignore unknown categories for certain columns whereas give error for the other columns?

import pandas as pd

df = pd.DataFrame({'Country':['USA','USA','IND','UK','UK','UK'],
                   'Fruits':['Apple','Strawberry','Mango','Berries','Banana','Grape'],
                   'Flower':   ['Rose','Lily','Orchid','Petunia','Lotus','Dandelion'],
                   'Result':[1,2,3,4,5,6,]})

from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

steps = [('OneHotEncoder', OneHotEncoder(handle_unknown ='ignore')) ,('LReg', LinearRegression())]

pipeline = Pipeline(steps)

from sklearn.model_selection import train_test_split

X = df[["Country","Flower","Fruits"]]
Y = df["Result"]
X_train, X_test, y_train, y_test = train_test_split(X,Y,test_size=0.3, random_state=30, shuffle =True)

print("X_train.shape:", X_train.shape)
print("y_train.shape:", y_train.shape)
print("X_test.shape:", X_test.shape)
print("y_test.shape:", y_test.shape)

pipeline.fit(X_train,y_train)

y_pred = pipeline.predict(X_test)

from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score

#Mean Squared Error:
MSE = mean_squared_error(y_test,y_pred)

print("MSE", MSE)

#Root Mean Squared Error:
from math import sqrt

RMSE = sqrt(MSE)
print("RMSE", RMSE)

#R-squared score:
R2_score = r2_score(y_test,y_pred)

print("R2_score", R2_score)

In this case for all the columns that is Country, Fruits and Flowers if there is a new value which comes the model would still be able to predict an output.

I want to know if there is a way to ignore unknown categories for Fruits and Flowers but however raise an error for unknown value in Country column?

Venkatachalam
  • 16,288
  • 9
  • 49
  • 77
sayo
  • 207
  • 4
  • 18

2 Answers2

11

I think ColumnTransformer() would help you to solve the problem. You can specify the list of columns for which you want to apply OneHotEncoderwith ignore for handle_unknown and similarly for error.

Convert your pipeline to the following using ColumnTransformer

from sklearn.compose import ColumnTransformer

ct = ColumnTransformer([("ohe_ignore", OneHotEncoder(handle_unknown ='ignore'), 
                              ["Flower", "Fruits"]),
                        ("ohe_raise_error",  OneHotEncoder(handle_unknown ='error'),
                               ["Country"])])

steps = [('OneHotEncoder', ct),
         ('LReg', LinearRegression())]

pipeline = Pipeline(steps)

Now, when we want to predict 

>>> pipeline.predict(pd.DataFrame({'Country': ['UK'], 'Fruits': ['Apple'], 'Flower': ['Rose']}))

array([2.83333333])

>>> pipeline.predict(pd.DataFrame({'Country': ['UK'], 'Fruits': ['chk'], 'Flower': ['Rose']}))

array([3.66666667])


>>> pipeline.predict(pd.DataFrame({'Country': ['chk'], 'Fruits': ['Apple'], 'Flower': ['Rose']}))

> ValueError: Found unknown categories ['chk'] in column 0 during
> transform

Note: ColumnTransformer is available from version 0.20.

cs95
  • 379,657
  • 97
  • 704
  • 746
Venkatachalam
  • 16,288
  • 9
  • 49
  • 77
  • 2
    Interesting, I had no idea this was directly possible. The syntax is a mouthful but definitely worth going with something that ships by default with the API. – cs95 Jun 18 '19 at 14:17
  • 1
    Brilliant answer – Chuck Mar 16 '20 at 13:39
1

From v0.20 onwards, you can use the ColumnTransformer API. However, for older versions, you can easily roll out your own implementation of a preprocessor that handles columns selectively.

Here's a simple prototype I've implemented which extends OneHotEncoder. You will need to specify a list of columns to raise an error on to the raise_error_cols argument. Any column not specified to this argument is implicitly handled as "ignored".

Sample Run

# Setup data
X_train

  Country     Flower  Fruits
2     IND     Orchid   Mango
0     USA       Rose   Apple
4      UK      Lotus  Banana
5      UK  Dandelion   Grape

X_test

  Country   Flower      Fruits
3      UK  Petunia     Berries
1     USA     Lily  Strawberry

X_test2 = X_test.append(
    {'Country': 'SA', 'Flower': 'Rose', 'Fruits': 'Tomato'}, ignore_index=True)
X_test2

  Country   Flower      Fruits
0      UK  Petunia     Berries
1     USA     Lily  Strawberry
2      SA     Rose      Tomato


from selective_handler_ohe import SelectiveHandlerOHE

she = SelectiveHandlerOHE(raise_error_cols=['Country'])
she.fit(X_train)

she.transform(X_test).toarray()
# array([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]])


she.transform(X_test2)
# ---------------------------------------------------------------------------
# ValueError: Found unknown categories SA in column Country during fit
cs95
  • 379,657
  • 97
  • 704
  • 746