OrdinalEncoder and keeping Nans

Question

I have a 2d numpy array that was created with:

array = dataset.to_numpy()
X = array[:, 1:]

I want to use OrdinalEncoder, but there are some Nans in X that I want to impute. I can't run OrdinalEncoder because it doesn't like the Nans and I can't run the KNNImputer until I encode.

I know I can replace the Nan with something like '?', etc and then OrdinalEncoder() will work, but then I have to go through and replace the numbers that the '?' turned into back to Nan. That means looping through the OrdinalEncoder internals to figure out what the '?' was mapped to in each column and then doing a replace on that column.

Isn't there a better way to do this? I was trying to get masking to work, but couldn't figure it out. I need to operate on X and not the dataset.

[PR19069](https://github.com/scikit-learn/scikit-learn/pull/19069) allows `np.nan` to pass through the `OrdinalEncoder`, so building from the main branch or waiting for the release of v1.0, this should work out-of-the-box. For similar questions doing this manually (for `OrdinalEncoder` or `OneHotEncoder`), see https://stackoverflow.com/a/64946429/10495893, https://stackoverflow.com/q/66635031/10495893, https://stackoverflow.com/q/54444260/10495893, https://stackoverflow.com/q/62868129/10495893 — Ben Reiniger, Jun 06 '21 at 00:02

score 2 · Answer 1 · answered Jun 05 '21 at 06:18

Too long for a comment, but if you don't mind some copying you can simply shuffle the NaNs out of the array temporarily.

array = dataset.to_numpy()
X = array[:, 1:]

nan_free_mask = ~np.isnan(X)
nan_free_X = X[nan_free_mask]

nan_free_encoded = OrdinalEncoder.fit_transform(nan_free_X, ...)
X_encoded = X.copy()
X_encoded[nan_free_mask] = nan_free_encoded
X_encoded = KNNImputer(...).fit_transform(X_encoded)

There is also nothing wrong with your idea of replacing nan with ? either. You simply need to remember where it happened. As far as I am aware, the OrdinalEncoder doesn't shuffle your data, but I could be wrong:

array = dataset.to_numpy()
X = array[:, 1:]

nan_mask = np.isnan(X)
X[nan_mask] = '?'
X_encoded = OrdinalEncoder.fit_transform(X, ...)
X_encoded[nan_mask] = np.nan  # restore NaN
X_encoded = KNNImputer(...).fit_transform(X_encoded)

Then again, you may have thought of this already ... if so, please update the question and specify what you have tried.

When I posted the question, I wasn't able to work out the syntax for creating the mask as you did above with native numpy, so thanks on that. I was able to create the mask correctly with pandas.isnull(). Wasn't a big fan of mixing numpy and pandas multiple times, but that's just me being picky. Either way seems kind of "hacky" to me of patching the arrays multiple times. What I ended up using was the OrdinalEncoder at https://github.com/scikit-learn-contrib/category_encoders... works with nans out of the box :). — SledgeHammer, Jun 05 '21 at 14:07

score 0 · Answer 2 · answered Jun 05 '21 at 08:18

If you don't need the encoder anymore, then you can use pd.factorize, which converts all np.nan to -1 and you need to put that back as np.nan :

import pandas as pd
import numpy as np
from sklearn.preprocessing import OrdinalEncoder

X = pd.DataFrame({'x1':['a','b',np.NaN],'x2':[np.NaN,'c','d']})

    x1   x2
0    a  NaN
1    b    c
2  NaN    d

X.apply(lambda x:x.factorize()[0]).replace(-1,np.nan)

    x1   x2
0  0.0  NaN
1  1.0  0.0
2  NaN  1.0

OrdinalEncoder and keeping Nans

2 Answers2