2

So, I have 8 columns in my dataframe: 5 features and other 3 are targets. After following these process, the results obtained are not good. Can anyone provide any feedback in the steps followed?

Here I am defining 2 minmax scaling variables, one for features and other for targets columns. Once model predicts the values, we run reverse scaling on features and predicted targets again to obtain the results.

#smoothening and minMax scaling
smoother=tsmoothie.KalmanSmoother(component='level_trend',  component_noise={'level':0.1, 'trend':0.1})
scaler_features = MinMaxScaler(feature_range=(0,1))
scaler_targets = MinMaxScaler(feature_range=(0,1))

#setting up features and targets from the df
df_norm_feature = scaler_features.fit_transform(raw_df.iloc[:,:5])
df_norm_target = scaler_targets.fit_transform(raw_df.iloc[:,5:])

#smoothening features and targets
smoother.smooth(df_norm_feature)
smoothed_features = smoother.smooth_data

smoother.smooth(df_norm_target)
smoothed_targets = smoother.smooth_data

#split into train test and train the data, and prepare the model on train.
#for reverse transformation I am using the following code. 
test_resultsForAll = mode.predict(test_data)
transformed_test_resultsForAll = scaler_targets.inverse_transform(test_resultsForAll))

but the results obtained via this method are not good. Are there any mistakes in the order of steps or do I need to perform minMax scaling & smoothening on the whole dataset at once?

  • Hi, could you provide a bit more context around the task you are trying to solve ? It would help us understand if the decisions you made are adapted. Concerning the use of scaling or any transformation technique, never tune it (select the min and max in your case) on the whole dataset, it will create "data-leak" between your train and test sets and may give you overly optimistic results. – Bruce Swain Nov 26 '21 at 09:34
  • Yes, I was asking whether to apply scaling on the train and test separately, or shall I apply the scaling on each column one by one. I shall keep those variables for inverse_transform once the results are obtained. – Sandeep Kumar Kushwaha Nov 26 '21 at 17:07

0 Answers0