3

I am trying to create a pipeline to help me process some data by: Imputing the mean, scaling the data, and then fitting a regressor.

I am having some trouble with the Imputer, and may be using it wrong. I know that my data contains NaN's; but when I try to Imput them inside a Pipeline, I get a ValueError:

Traceback (most recent call last):

File "<ipython-input-124-8517b294cb64>", line 1, in <module>
modelBuild(df)

File "C:/Users/tmori/Google Drive/Projects/Product Dimension Accuracy/Qubert_PredictiveModel/qubert_predictive_model.py", line 81, in modelBuild
clf_x = pipeline.fit_transform(df[['OverallHeight-ToptoBottom', 'OverallDepth-FronttoBack']], df['OverallWidth-SidetoSide'])

File "C:\Program Files\Anaconda\lib\site-packages\sklearn\pipeline.py", line 303, in fit_transform
return last_step.fit_transform(Xt, y, **fit_params)

File "C:\Program Files\Anaconda\lib\site-packages\sklearn\base.py", line 497, in fit_transform
return self.fit(X, y, **fit_params).transform(X)

File "C:\Program Files\Anaconda\lib\site-packages\sklearn\ensemble\forest.py", line 248, in fit
y = check_array(y, accept_sparse='csc', ensure_2d=False, dtype=None)

File "C:\Program Files\Anaconda\lib\site-packages\sklearn\utils\validation.py", line 407, in check_array
_assert_all_finite(array)

File "C:\Program Files\Anaconda\lib\site-packages\sklearn\utils\validation.py", line 58, in _assert_all_finite
" or a value too large for %r." % X.dtype)

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

My code looks like this so far:

def modelBuild(df):  
    imp = Imputer()  
    scl = StandardScaler()  
    clf = RandomForestRegressor()      
    pipeline = Pipeline([('imputer', imp),  
                         ('scaler', scl),  
                         ('clf', clf)])
    clf_x = pipeline.fit_transform(df[['OverallHeight-ToptoBottom', 'OverallDepth-FronttoBack']], df['OverallWidth-SidetoSide'])

and an example of the DataFrame data:

StagName   OverallDepth-FronttoBack  OverallHeight-ToptoBottom  \
PtagPrSKU                                                        
AABP1004                        NaN                       48.0   
AAI2179                        28.0                       32.0   
AAI2180                        28.0                       32.0   
AAI2181                        36.0                       32.0   
AAI2182                        36.0                       32.0   

StagName   OverallWidth-SidetoSide  
PtagPrSKU                           
AABP1004                      64.0  
AAI2179                       55.0  
AAI2180                       55.0  
AAI2181                       71.0  
AAI2182                       71.0

I am pretty sure I am just using the Imputer incorrectly, but I cannot for the life of me pinpoint where.

Thanks in advance for all the help!

Best, Tom

Quickbeam2k1
  • 5,287
  • 2
  • 26
  • 42
Tom Mori
  • 131
  • 1
  • 1
  • 3

2 Answers2

0

Try to remove the line PtagPrSKU.

So after the column names you should just have their values. The easy way to do this is using pandas and defining skiprows when loading the data.

The following works fine for me.

The problem

The PtagPrSKU line inserts an empty cell for each column (this is the problem).

The file that I used for this example can be found here link

from sklearn.preprocessing import Imputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
import pandas as pd



def buildit(df):
    imp = Imputer()  
    scl = StandardScaler()  
    clf = RandomForestRegressor()      
    pipeline = Pipeline([('imputer', imp), ('scaler', scl),  ('clf', clf)])
    clf_x = pipeline.fit_transform(df[['OverallHeight-ToptoBottom', 'OverallDepth-FronttoBack']], df['OverallWidth-SidetoSide'])

    return clf_x



df = pd.read_excel('t.xlsx',skiprows=[1])
print(df)
buildit(df)
seralouk
  • 30,938
  • 9
  • 118
  • 133
-1

Change your missing value identifier from 'np.nan' to something else (maybe 0 or a very big number). I had the same issue and this worked for me.

Tola
  • 1
  • 1