2

I want to do a custom sampling with caret. My specifications are the following: I have 1 observation per day, and my grouping factor is the month (12 values); so in the first step I create 12 resamples with 11 months in the training (11*30 points) and 1 in the testing (30 points). This way I get 12 resamples in total.

But that's not enough to me and I would like to make it a little more complex, by adding some bootstrapping of the training points of each partition. So, instead of having 11*30 points in Resample01, I would have several bootstrapped resamples of these 330 points. So in the end, I want a lot of resamples, but with one of the months NEVER in the training set.

How to specify this in a call to train? What I tried: 

library(caret)
x = rep(1:12, each=30)
folds = groupKFold(x, k=12)
folds2 = lapply(folds, createResample, times=10)

but this is wrong because 1/ i get a nested list, 2/ the initial indices are lost at the second step. enter image description here

Thanks for your help (and don't hesitate to tell me if you think it's a XY pb)

jmuhlenkamp
  • 2,102
  • 1
  • 14
  • 37
agenis
  • 8,069
  • 5
  • 53
  • 102
  • 1
    I initially suggested `purrr::map_df(folds, createResample, times = 10, .id = "id")`, but this isn't workable with `caret::trainControl()` – Phil Jan 08 '18 at 01:00
  • @Phil thanks for your help anyway, I was glad to discover this `map` function – agenis Jan 08 '18 at 01:05

1 Answers1

1

I trust this will solve your problem

library(caret)
x <- rep(1:12, each = 30)
folds <- groupKFold(x, k = 12)

provide 10 bootstrap replicates in a nested list for each of the groups in folds - this solves the lost indexes problem.

folds2 <- lapply(folds, function(x) lapply(1:10, function(i) sample(x, size = length(x), replace = TRUE)))

convert nested list to a one dimensional list - this solves the nested list problem.

folds2 <- unlist(folds2 , recursive = FALSE, use.names = TRUE)

does it work?

df <- data.frame(y = rnorm(360), x = rnorm(360))

lm_formula <- train(
  y ~ ., df,
  method = "lm",
  trControl = trainControl(method = "boot" , index = folds2)
)

looks like it does.

The only issue is perhaps in the intended indexOut for each resample, in the example all indexes not present in the fold were used as test. If I understood you would like to test on the held out months and not on all the held out samples. To solve this:

folds_out <- lapply(folds, function(x) setdiff(1:360, x))
folds_out <- rep(folds_out, each = 10)
names(folds_out) <- names(folds2)

lm_formula <- train(
  y ~ ., df,
  method = "lm",
  trControl = trainControl(method = "boot" , index = folds2, indexOut = folds_out)
)
missuse
  • 19,056
  • 3
  • 25
  • 47
  • Thanks for your answer @missuse. You're absolutely right, I indeed wanted to have only the 12th month in the test set. Nice use of `setdiff` by the way. If I get it right, there is no built-in function in caret to achieve what I want. – agenis Jan 08 '18 at 08:55
  • @agenis Glad to help. I am unsure if there is a built in caret function, from what I know this functionality is not inbuilt, but you might want to check `createTimeSlices` and `trainControl(method = "timeslice"...` To me it looks like they are able to form sets so only prediction on future data is possible, I have never performed serious ts analyses so my knowledge is lacking in this area. Also from what I can see your example does not fall strictly into ts analyses since you would like to predict previous time on later observations too. – missuse Jan 08 '18 at 10:00
  • you're right i"m not in the time series framework, and I don"t use timeseries-specific models here. – agenis Jan 08 '18 at 10:10