5

I have a dataset and I want to build a model, preferably with the caret package. My data is actually a time series but the question is not specific to time series, it's just that I work with CreateTimeSlices for the data partition.

My data has a certain amount of missing values NA, and I imputed them separately of the caret code. I also kept a record of their locations:

# a logical vector same size as the data, which obs were imputed NA
imputed=c(FALSE, FALSE, FALSE, TRUE, FALSE, FALSE)
imputed[imputed] <- NA; print(imputed)
#### [1] FALSE FALSE FALSE    NA FALSE FALSE

I know there is an option in Caret train function to either exclude the NA or impute them with different techniques. That's not what I want. I need to build the model on the already imputed dataset but I want to exclude the imputed points from the calculation of the error indicators (RMSE, MAE, ...).

I don't know how to do this in caret. In my first script I tried to do the whole cross validation manually, and then I had a customized error measure:

actual = c(5, 4, 3, 6, 7, 5)
predicted = c(4, 4, 3.5, 7, 6.8, 4)
Metrics::rmse(actual, predicted) # with all the points
#### [1] 0.7404953
sqrt(mean( (!imputed)*(actual-predicted)^2 , na.rm=T)) # excluding the imputed
#### [1] 0.676757

How can I handle this way of doing in caret? Or is there another way to avoid coding everything by hand?

agenis
  • 8,069
  • 5
  • 53
  • 102

1 Answers1

4

I don't know if you are looking for this but here is a simple solution by creating a function:

i=which(imputed==F) ## As you have index for NA values

metric_na=function(fun, actual, predicted, index){
    fun(actual[index], predicted[index])
}

metric_na(Metrics::rmse, actual, predicted, index = i)
0.676757
metric_na(Metrics::mae, actual, predicted, index = i)
0.54

Also you can just use the index directly while calculating the desired metrics:

Metrics::rmse(actual[i], predicted[i])
halfer
  • 19,824
  • 17
  • 99
  • 186
Chirayu Chamoli
  • 2,076
  • 1
  • 17
  • 32
  • Thanks, that's a nice function that really helps, I will surely implement it it noone thinks its possible with caret :-) – agenis Oct 21 '16 at 21:46
  • I think it would be possible but it would require some effort. Also look at [this](http://stackoverflow.com/questions/28232658/how-to-change-metrics-using-the-librarycaret/28278334#28278334) post if you haven't. – Chirayu Chamoli Oct 22 '16 at 04:04