Questions tagged [cross-validation]

Cross-Validation is a method of evaluating and comparing predictive systems in statistics and machine learning.

Cross-Validation is a statistical method of evaluating and comparing learning algorithms by dividing data into two segments: one used to learn or train a model and the other used to validate the model.

In typical cross-validation, the training and validation sets must cross-over in successive rounds such that each data point has a chance of being validated against. The basic form of cross-validation is k-fold cross-validation.

Other forms of cross-validation are special cases of k-fold cross-validation or involve repeated rounds of k-fold cross-validation.

2604 questions
1
vote
1 answer

Kfold cross validation different correct rate values

I have a problem with the K fold cross validation in Matlab, actually I have used the function mentioned here. The problem is that I get different Cp values every time I run my program. I use the function to perform a feature selection procedure…
Tarek
  • 11
  • 1
1
vote
1 answer

Cross validation accuracy rates with R

I am using e1071 (svm) package of R. If I give cross parameter to the builder method, it cross validates and chooses the best model. However, if I do the cross validation manually, I would have accuracy rates of all models which are built for each…
Ricardo Cristian Ramirez
  • 1,194
  • 3
  • 20
  • 42
1
vote
1 answer

train() function and rate model (poisson regression with offset) with caret

I fitted a rate model using glm() (poisson link with offset, like y ~ offset(log(x1)) + x2 + x3 the response is y/x1 in this case). Then I wanted to do cross validation using caret package so I used 'train()' function with k-fold CV control. It…
zhifff
  • 199
  • 1
  • 4
  • 15
1
vote
0 answers

Cross Validation - One class classification

I am trying to perform one-class classification such that I only have positive data during training time and negative data is only available at testing time. If I use, One-class SVM or any other one-class method, there is a common problem to…
Shehroz
  • 347
  • 2
  • 9
  • 25
1
vote
0 answers

indices to column vectros of a matrix using "crossvalind"

i got a 3x50 matrix. i would like to obtain the 50 3x1 column vectors and assign to them indices using the crossvalind command seen above Indices = crossvalind('Kfold',N, 10); so lets hypotheticaly say that i had a 3x3 matrix named "input"…
1
vote
1 answer

Simulate Missing Data (i.e. Mask Data) in R to Test Imputation Accuracy

I want to determine a program's imputation accuracy using SNP genotype data, so I need to mask a portion of the SNP calls to simulate missing data. I've been testing my code on this subset of marker data (see below). Column names are names of…
1
vote
1 answer

Labeling one class for cross validation in libsvm matlab

I want to use one-class classification using LibSVM in MATLAB. I want to train data and use cross validation, but I don't know what I have to do to label the outliers. If for example I have this data: trainData = [1,1,1; 1,1,2; 1,1,1.5; 1,1.5,1;…
jessica
  • 379
  • 8
  • 23
1
vote
1 answer

Cross-validation and a separate "supplied test set"

After performing a 10 fold cross-validation, is it proper to perform a re-evaluation using a separate "supplied test set"?
1
vote
1 answer

ROC curve plot: 0.50 significant and cross-validation

I have got two problems of using pROC package to plot the ROC curve. A. The Significance level or P-value is the probability that the observed sample Area under the ROC curve is found when in fact, the true (population) Area under the ROC curve is…
Samo Jerom
  • 2,361
  • 7
  • 32
  • 38
1
vote
1 answer

how to create leave one out cross validation in matlab?

I am still confused with my code. I tried to implement leave one out cross validation in matlab for classification. so in here . I take out one data from training become testing data. I already make a code in matlab. but Iam not sure it's correct…
user2157806
  • 319
  • 2
  • 8
  • 17
1
vote
0 answers

R gbm: How does cross validation work in this package?

Can someone provide a work flow about this? For instance, suppose I am doing binary classification, For each iteration of the algorithm: Randomly sample k*N rows, where k is the bag.fraction, and N is nrow(dataset). Build a classifier using this…
Boxuan
  • 4,937
  • 6
  • 37
  • 73
1
vote
1 answer

Inconsistency in cross-validation results

I have a set of dataset recorded from subjects as they perform some particular cognitive task. The data consists of 16 channels and a number of sample points per channel and I want to classify this data according to the cognitive task being…
user2036690
  • 111
  • 2
1
vote
1 answer

Error rate in multidimensional contingency tables R

Good evening, I found the classAgreement function from package e1071 in R very useful and informative when working with contingency tables. It works great with 2-dimensions. But if I have 3 or more dimensions it doesn't work. I have reviewed several…
Diego
  • 824
  • 2
  • 9
  • 18
1
vote
2 answers

Cross validation of PCA+lm

I'm a chemist and about an year ago I decided to know something more about chemometrics. I'm working with this problem that I don't know how to solve: I performed an experimental design (Doehlert type with 3 factors) recording several analyte…
Ndr
  • 550
  • 3
  • 15
1
vote
1 answer

MATLAB - Difference between "k" and 'kfold" partition type for cvpartition

Can someone please explain to me the difference and its effect on partitioning data using either the stratified 10-fold cross validation procedure (as signified by 'k' partition type) or a 10-fold cross validation procedure (as indicated by "kfold"…