17

Similar to Custom cross validation split sklearn I want to define my own splits for GridSearchCV for which I need to customize the built in cross-validation iterator.

I want to pass my own set of train-test indices for cross validation to the GridSearch instead of allowing the iterator to determine them for me. I went through the available cv iterators on the sklearn documentation page but couldn't find it.

For example I want to implement something like this Data has 9 samples For 2 fold cv I create my own set of training-testing indices

>>> train_indices = [[1,3,5,7,9],[2,4,6,8]]
>>> test_indices = [[2,4,6,8],[1,3,5,7,9]]
                 1st fold^    2nd fold^
>>> custom_cv = sklearn.cross_validation.customcv(train_indices,test_indices)
>>> clf = GridSearchCV(X,y,params,cv=custom_cv)

What can be used to work like customcv?

Community
  • 1
  • 1
tangy
  • 3,056
  • 2
  • 25
  • 42
  • Could you add a question? Also I am not aware of the exsistence of `customcv` in `sklearn.cross_validation`, so you probably shouldn't put it. Are you sure that `LeaveOneLabelOut` does not work in your case? – eickenberg Nov 24 '14 at 10:53
  • 1
    I gave customcv as an example of what I wanted to implement... it isnt in sklearn. Ill try the method you gave in your answer – tangy Nov 24 '14 at 15:39

2 Answers2

16

Actually, cross-validation iterators are just that: Iterators. They give back a tuple of train/test fold at each iteration. This should then work for you:

custom_cv = zip(train_indices, test_indices)

Also, for the specific case you are mentioning, you can do

import numpy as np
labels = np.arange(0, 10) % 2
from sklearn.cross_validation import LeaveOneLabelOut
cv = LeaveOneLabelOut(labels)

Observe that list(cv) yields

[(array([1, 3, 5, 7, 9]), array([0, 2, 4, 6, 8])),
 (array([0, 2, 4, 6, 8]), array([1, 3, 5, 7, 9]))]
eickenberg
  • 14,152
  • 1
  • 48
  • 52
  • 1
    Amendment: This is for pre-0.18 releases of scikit-learn. The `cross_validation` module functionality is now in `model_selection`, and cross-validation splitters are now classes which need to be explicitly asked to split the data using the method `split`. This is to make nested cross-validation easier. – eickenberg Sep 05 '17 at 17:47
5

Actually the above solution returns each row as a fold what one really needs is:

    [(train_indices, test_indices)] # for one fold

    [(train_indices, test_indices), # 1stfold
    (train_indices, test_indices)] # 2nd fold etc
Cibic
  • 316
  • 5
  • 14