FileNotFoundError: No such file: -> Error occuring due to TimeOut of Google Drive?

Question

I created a DataGenerator with Sequence class.

import tensorflow.keras as keras
from skimage.io import imread
from skimage.transform import resize
import numpy as np
import math
from tensorflow.keras.utils import Sequence

Here, `x_set` is list of path to the images and `y_set` are the associated classes.

class DataGenerator(Sequence):
    def __init__(self, x_set, y_set, batch_size):
    self.x, self.y = x_set, y_set
    self.batch_size = batch_size

def __len__(self):
    return math.ceil(len(self.x) / self.batch_size)

def __getitem__(self, idx):
    batch_x = self.x[idx * self.batch_size:(idx + 1) *
    self.batch_size]
    batch_y = self.y[idx * self.batch_size:(idx + 1) *
    self.batch_size]

    return np.array([
        resize(imread(file_name), (224, 224))
           for file_name in batch_x]), np.array(batch_y)

Then, I applied this to my training and validation data. X_train is a list of strings which contains the image paths to the training data. y_train are onehotencoded labels of the training data. The same for validation data.

I created the image paths using this code:

X_train = []
for name in train_FileName:
  file_path = r"/content/gdrive/My Drive/data/2017-IWT4S-CarsReId_LP-dataset/" + name
  X_train.append(file_path)

After that, I applied the DataGenerator to the training and validation data:

training_generator = DataGenerator(X_train, y_train, batch_size=32)
validation_generator = DataGenerator(X_val, y_val, batch_size=32)

Afterwards I used the fit_generator method to run a model:

model.fit_generator(generator=training_generator,
                    validation_data=validation_generator,
                    steps_per_epoch = num_train_samples // 32,
                    validation_steps = num_val_samples // 32,
                    epochs = 10,
                    use_multiprocessing=True,
                    workers=2)

On CPU it worked fine the first times, my model was initialized and the first epoch started. Then, I changed the runtime type in Google Colab to GPU and ran the model again.

And got the following error:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-79-f43ade94ee10> in <module>()
      5                     epochs = 10,
      6                     use_multiprocessing=True,
----> 7                     workers=2)

16 frames
/usr/local/lib/python3.6/dist-packages/imageio/core/request.py in _parse_uri(self, uri)
    271                 # Reading: check that the file exists (but is allowed a dir)
    272                 if not os.path.exists(fn):
--> 273                     raise FileNotFoundError("No such file: '%s'" % fn)
    274             else:
    275                 # Writing: check that the directory to write to does exist

FileNotFoundError: No such file: '/content/gdrive/My Drive/data/2017-IWT4S-CarsReId_LP-dataset/s01_l01/1_1.png'

Today, I got this error also when running the program without the usage of GPU. When running the program, Colab told me that there was Google Drive Time Out. So, is this error due to this timeout of Google Drive? And if yes, how can I solve this? Does anyone know what I should change in the program?

Answer 1

You can write this code to avoid timeout in google colab in console

ConnectButton() {     
    console.log("Connect pushed");      
    document.querySelector("#top-toolbar > colab-connect- 
        button").shadowRoot.querySelector("#connect").click()  
} 
setInterval(ConnectButton,60000);

Source: How to prevent Google Colab from disconnecting?

Answer 2

The problem seems to be the input. Your model cannot find the input file. If you change runtime then there's gonna be a factory reset. All your disk content in the session will be erased.

Run cells from the beginning if you change runtime in between.