0

I'm trying to train a model with the following code in Colab, using keras:

  model.fit(
      x_tr, y_tr, 
      batch_size = 1,
      callbacks=[model_checkpoint_callback],
      validation_data = (x_val, y_val), 
      epochs = num_training)

The point is that using Colab it says that there are problem with the RAM so what I want to do is split the dataset (x_tr), and train the model on the different splits like this:

num_divisione_dataset=8
div_tr = int(16640/num_divisione_dataset)
div_val = int(2160/num_divisione_dataset) 

num_training = int(math.ceil(100/num_divisione_dataset))

for i in range(0,num_divisione_dataset-1):
  print("Training number: \n\n ", num_divisione_dataset)
  model.fit(
      x_tr[div_tr*i:div_tr*(i+1)], y_tr[div_tr*i:div_tr*(i+1)], 
      batch_size = 1,
      callbacks=[model_checkpoint_callback],
      validation_data = (x_val[div_val*i:div_val*(i+1)], y_val[div_val*i:div_val*(i+1)]), 
      epochs = num_training)

So what I'm doing is taking each split (8 splits in total) of x_tr and training the model till the final split.

For each split I'm training for 13 epochs.

Is it the right way to train the model on the whole dataset? The point is that when I try to predict using the model I have the 20% of accuracy, against the 96% on the evaluation set.

Diego Rando
  • 85
  • 8
  • pretty sure that `batch_size = 1,` is not a great choice – Alberto Sinigaglia Aug 02 '22 at 21:27
  • @AlbertoSinigaglia do you think however that this is a good solution to train the model on the whole dataset? Despite all the parameters? – Diego Rando Aug 02 '22 at 22:11
  • I changed however from batch_size=1 to batch_size=32 – Diego Rando Aug 02 '22 at 22:45
  • Why don't you try using tensorflow datasets to do this, or any other appropriate solution (like flow_from_directory if it's images)? If it's not fitting in the RAM, you are fetching each split of the data from somewhere I presume. – Vishal Balaji Aug 04 '22 at 06:58
  • @VishalBalaji I already use it and gives me problems. But however what I'm trying to understand is: is this solution ok? like does it make sense? – Diego Rando Aug 04 '22 at 10:29

0 Answers0