Questions tagged [cross-validation]

Cross-Validation is a method of evaluating and comparing predictive systems in statistics and machine learning.

Cross-Validation is a statistical method of evaluating and comparing learning algorithms by dividing data into two segments: one used to learn or train a model and the other used to validate the model.

In typical cross-validation, the training and validation sets must cross-over in successive rounds such that each data point has a chance of being validated against. The basic form of cross-validation is k-fold cross-validation.

Other forms of cross-validation are special cases of k-fold cross-validation or involve repeated rounds of k-fold cross-validation.

2604 questions
0
votes
0 answers

LSTM windowsize

I have an LSTM code but when I increase the window size, it is not working anymore. is there someone who knows how to fix this? I think it is because of [i+i+window_size], but I do not know how I can change this. def df_to_X_y(df,…
J1999
  • 3
  • 2
0
votes
1 answer

How to use cross_cal_score for MNIST dataset? How to fix: "ValueError: Shapes (None, 1) and (None, 10) are incompatible"

How to use cross_cal_score for MNIST dataset? Shapes between the output of my model and the y in cross_cal_score are incompatible. I would like to perform k-fold cross validation for MNIST dataset (an example in DEEP LEARNING: FROM BASICS TO…
0
votes
0 answers

XGBoost good in cross-validation but bad in testing

Dealing with overfitting problem I try to perform xgboost classification with Stratified Cross Validation and SMOTE sampling, and also randomized search. The result of cross validation XGBoost is great, but when I test it with testing dataset, the…
Lim
  • 11
  • 2
0
votes
0 answers

having trouble with measuring metrics in cross_validation

I have a simple KNN model and doing a cross_vaidation. model = KNeighborsClassifier(n_neighbors = 2, weights='uniform') cv = StratifiedKFold(n_splits = 5) scores = cross_val_score(model, trainX, trainY, scoring=acc_score, cv = cv) scores the…
william_Li
  • 21
  • 4
0
votes
1 answer

How to apply TimeSeries Cross Validation in Python for Data with irregular Dates of Observations and uneven Observations per Date?

Best Way to Perform TimeSeries Cross Validation with irregular Dates of Observations and uneven Observations per Date? I have a dataset that I have been trying to utilize for XGBoost Regression. The problem I am encountering is how best to apply…
0
votes
0 answers

Getting variable of importance for ANN through leave-one-out cross validation

I want to get the VIPs for a ANN using LOO-CV. The code bellow works well when the vips is not being requested, but when I run the example leaving out a group it works well. I wonder why this is happening and how to fix this…
Natz
  • 77
  • 6
0
votes
0 answers

How to use cross validation correctly for deep learning?

How to use cross validation correctly for a deep learning model? How to use cross validation correctly for a deep learning model? How to use cross validation correctly for a deep learning model? X_train, X_test, y_train, y_test…
0
votes
0 answers

How to handle misssing classes in leave-one-group-out CV strategy

I have a multi-class classification dataset that has readings from several places, and multi-class output. To get understanding of model performance, i am utilizing sklearn's cross-validate method. I am also utilizing leave-one-group out as a cv…
sp2006
  • 59
  • 3
0
votes
0 answers

how to preserve validation set in caret for hyperparameter tuning instead of cross validation?

trainControl(method=) I have a large time series dataset, and I would like to split into training,validation and testing set. However, I couldn't find the setting for prespecified validation set in caret trainControl for hyperparameter tuning. I…
Grace T
  • 1
  • 1
0
votes
0 answers

glmnet ununderstandable: Error in lognet(xd, is.sparse, ix, jx, y, weights, offset, alpha, nobs,

I have installed R and Rstudio just to use glmnet for a multinomial logistic regression. I keep getting the following error: Error in lognet(xd, is.sparse, ix, jx, y, weights, offset, alpha, nobs, : one multinomial or binomial class has 1 or 0…
0
votes
0 answers

Calculating cross-validation classification accuracy with stan/loo

I'm fitting a custom model to binary classification data using RStan, and assessing it with the "loo" cross-validation package. I want to estimate the classification accuracy (i.e., % of out-of-sample choices predicted correctly), rather than the…
Adam Morris
  • 293
  • 3
  • 8
0
votes
0 answers

getting Rank deficiency error in cross validation

Hi I am trying to cross validate my multiple logistic predictive model. I have used the following code before on the same data before and it never gave me an issue but now its giving the rank deficiency error. Any help would be…
0
votes
1 answer

Calculating LOOCV with adjusted R-square scoring parameter

I have dataset where I transformed categorical dataset into numerical by dummies and I ran simple linear regression model to predict dependent variable. I got adjusted R-square as 0.66. Now I want to cross validate my model with leave one out method…
0
votes
1 answer

Cross Validation on R Ranger Library

Hello I have the following ranger model: X <- train_df[, -1] y <- train_df$Price rf_model <- ranger(Price ~ ., data = train_df, mtry = 11 ,splitrule = "extratrees" ,min.node.size = 1, num.trees =100) I am trying to accomplish two things, Give me…
Cindy Burker
  • 117
  • 9
0
votes
0 answers

Train-test split in panel data

I have a panel dataset (multiple time series indexed on IDs and time) in which I want to perform multi-step forecasts (e.g. 5-steps forecasts). An example of the dataset (pandas dataframe) is the following: IDs, time, f1, f2, ... 1 0 4.1 50…