Failed copying input tensor from CPU to GPU in order to run GatherVe: Dst tensor is not initialized. [Op:GatherV2]

Question

    from random import sample
    index=sample(range(0, len(result)), len(result)//5*4)
    description_train=[child[0] for i, child in enumerate(result) if i in index]
    ipc_train=[child[1] for i, child in enumerate(result) if i in index]
    description_test=[child[0] for i, child in enumerate(result) if i not in index]
    ipc_test=[child[1] for i, child in enumerate(result) if i not in index]
    
    import numpy as np
    
    def to_onehot(li):
        result=np.zeros(8)
        if 'A' in li:
            result[0]=1
        if 'B' in li:
            result[1]=1
        if 'C' in li:
            result[2]=1
        if 'D' in li:
            result[3]=1
        if 'E' in li:
            result[4]=1
        if 'F' in li:
            result[5]=1
        if 'G' in li:
            result[6]=1
        if 'H' in li:
            result[7]=1
        return result
            
            
    
    from tensorflow.python.keras.preprocessing.text import Tokenizer
    
    
    max_words=100000
    num_classes=8
    
    t=Tokenizer(num_words=max_words)
    t.fit_on_texts(description_train)
    X_train=t.texts_to_matrix(description_train, mode='binary')
    X_test=t.texts_to_matrix(description_test, mode='binary')
    Y_train=np.array([to_onehot(child) for child in ipc_train], dtype=np.int32)
    Y_test=np.array([to_onehot(child) for child in ipc_test], dtype=np.int32)
    
    
    from tensorflow.python.keras.models import Sequential
    from tensorflow.python.keras.layers import Dense, Dropout
    
    
    model = Sequential()
    model.add(Dense(1024, input_shape=(max_words,), activation='relu'))
    model.add(Dense(128, activation='relu'))
    model.add(Dense(num_classes, activation='sigmoid'))
    
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit(X_train, Y_train, batch_size=128, epochs=5, validation_split=0.1)

the last line (model.fit) result in a following error.

InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run GatherV2: Dst tensor is not initialized. [Op:GatherV2]

How can I fix it? thank you in advance.

Answer 1

I had this error very often, even with high-RAM EC2 instances. The only solution for me was to use generators:

from tensorflow.keras.utils import Sequence
import numpy as np   

class DataGenerator(Sequence):
    def __init__(self, x_set, y_set, batch_size):
        self.x, self.y = x_set, y_set
        self.batch_size = batch_size

    def __len__(self):
        return int(np.ceil(len(self.x) / float(self.batch_size)))

    def __getitem__(self, idx):
        batch_x = self.x[idx * self.batch_size:(idx + 1) * self.batch_size]
        batch_y = self.y[idx * self.batch_size:(idx + 1) * self.batch_size]
        return batch_x, batch_y

train_gen = DataGenerator(X_train, y_train, 32)
test_gen = DataGenerator(X_test, y_test, 32)


history = model.fit(train_gen,
                    epochs=6,
                    validation_data=test_gen)

In the above example, we assume that X and y are numpy arrays.

My guess on what's happening: even though I'm using a high-RAM instance, I suspect the problem is a limitation in the GPU memory, and even though I'm training in batches, when not using generators, TensorFlow is trying to load the full array into the GPU memory.

Answer 2

it may be because of RAM shortage, and you can do one of the followings to solve the problem:

1. decreasing the batch_size can help most of the times and this is the best option when the training speed is not matters for you(as you know with decreasing the batch_size , it takes more long time for model to be trained).
2. you can run your code on a system with a large amount of RAM or run your code on a VM or Google Colab for free(Google Colab gives you 16 GB of RAM for free with Tesla K80 GPUs and TPU.
3. reduce the number of samples or reduce your data dimension with methods such as PCA or feature selection.
4. also if your model`s hidden layers size is so large, you can decrease it to solve problem in some situations.because with decreasing hidden layer size the model complexity and parameters decrease and it occupies less memory.

Answer 3

One solution is to reduce the size of input images to fit the capacity of GPU. For me, I reduced from (224,224,10) to (128,128,10).

Answer 4

One memory-costly process is the internal conversion from (usually) numpy to tf.tensor. If you are sure that your GPU should be able to handle the batches: Manually convert the data to tf tensors using the CPU RAM, and only then pass it to your model (eventually using GPU).

with tf.device('/cpu:0'):
   x = tf.convert_to_tensor(x, np.float32)
   y = tf.convert_to_tensor(y, np.float32)

and then, outside of the with statement:

model.fit(x=x, y=y)

Answer 5

I found a solution. I reduced the number of sample by

model.fit(X_train[0:3000], Y_train[0:3000], batch_size=128, epochs=5, validation_split=0.1)

Then, the error disappeared.

Good luck for everyone.

Answer 6

Reducing the sample size is not always the option because why would you have those many samples in the first place, therefore, I would recommend a few options:

1. Use a cloud VM (AWS, Azure or GCP) with higher specs and pay hourly and be done and dusted on this one

2. If you don't want to pay, and Ok to write extra code, then basically, you have to create your own custom generator to call flow_from_directory to load dataset in batches. Refer to this: https://www.askpython.com/python/examples/handling-large-datasets-machine-learning https://www.analyticsvidhya.com/blog/2020/08/image-augmentation-on-the-fly-using-keras-imagedatagenerator/

Answer 7

Might be silly answer, but I wasn't able to load 200MB of training data into RTX3090 (24GB RAM) and after fighting a bit with this problem the solution for me was:

sudo reboot now

afterwards everything started to work again.