Dear overflowing people,
after excessively and successfully using stackoverflow for years on the consuming end, I just came across an issue that I haven't found any proper answers for - hence:
Problem description:
In my current project I am running a GAN model in order to predict player moves in a chess game based on roughly 4000 games I've played. The base model is the same as the one used in the Pix2Pix paper and it has been working splendidly with a simple installation of tensorflow 2.5 (and the python chess module) onto python 3.8.5 and 3.8.10 respectively. After a strugglesome installation of CUDA and cudnn I finally managed to setup an environment with full GPU support (as training the model on my CPU would've taken an estimated 3 weeks).
However, without any changes to my model it wouldn't train anymore upon switching to GPU computing. My troubleshooting attempts point towards some issues with the prediction part of the model, presumably related to my model using too much memory for my GPU to handle (it's a GTX 960 with 4GB RAM). The windows explorer shows a steep increase in GPU RAM usage, quickly capping at close to 4GB upon running the model. My experience with computing on my CPU leads me to believe that the model actually requires around 5-6GB of RAM.
My troubleshooting so far:
My research on this issue led me to two advices: Decreasing the batch size and reducing the image resolution. Unfortunately, the way this model is designed, the batch size is already at 1 upon running it. With respect to image size/resolution it seems to me that I can't quite accommodate any changes here either, as Im using a one-hot encoded chess board - which conists of an 8x8 matrix (the board), multiplied by 13 (representing all unique chess pieces).
I also tried using the set_memory_growth command to True. However, this didn't change anything at all.
My specs:
- OS: Windows 10 64
- GPU: Geforce GTX 960 (4GB)
- Python GUI: Spyder 5
- Python: 3.8.5 / 3.8.10 in a conda environment
- Tensorflow: 2.5
- CUDA: 11.2.2
- cudnn: 8.1.1
My code:
I couldn't include all my code, hence only the GAN part.
from numpy import expand_dims
from numpy import zeros
from numpy import ones
from numpy import vstack
from numpy.random import randn
from numpy.random import randint
from tensorflow.keras.utils import plot_model
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Conv2D,Conv2DTranspose
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import concatenate
from tensorflow.keras.initializers import RandomNormal
from tensorflow.keras.layers import LeakyReLU
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.layers import Activation,Reshape
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dropout
from tensorflow.config import list_physical_devices
from tensorflow.config.experimental import set_memory_growth
from tensorflow.config.experimental import list_logical_devices
from tensorflow.config.experimental import VirtualDeviceConfiguration
from tensorflow.config.experimental import set_virtual_device_configuration
def define_discriminator():
init = RandomNormal(stddev=0.02)
in_src_image = Input(shape=image_shape) # image_shape = (8, 8, 13)
in_target_image = Input(shape=image_shape)
merged = concatenate([in_src_image, in_target_image])
d = Conv2D(64, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(merged)
d = LeakyReLU(alpha=0.2)(d)
d = Conv2D(128, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d)
d = BatchNormalization()(d)
d = LeakyReLU(alpha=0.2)(d)
d = Conv2D(256, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d)
d = BatchNormalization()(d)
d = LeakyReLU(alpha=0.2)(d)
d = Conv2D(512, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d)
d = BatchNormalization()(d)
d = LeakyReLU(alpha=0.2)(d)
d = Conv2D(512, (4,4), padding='same', kernel_initializer=init)(d)
d = BatchNormalization()(d)
d = LeakyReLU(alpha=0.2)(d)
d = Conv2D(1, (4,4), padding='same', kernel_initializer=init)(d)
patch_out = Activation('sigmoid')(d)
model = Model(inputs = [in_src_image, in_target_image], outputs = patch_out)
opt = Adam(lr=0.0002, beta_1=0.5)
model.compile(loss='binary_crossentropy', optimizer=opt, loss_weights=[0.5])
return model
def define_encoder_block(layer_in, n_filters, batchnorm=True):
init = RandomNormal(stddev=0.02)
g = Conv2D(n_filters, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(layer_in)
if batchnorm:
g = BatchNormalization()(g, training=True)
g = LeakyReLU(alpha=0.2)(g)
return g
def decoder_block(layer_in, skip_in, n_filters, dropout=True):
init = RandomNormal(stddev=0.02)
g = Conv2DTranspose(n_filters, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(layer_in)
g = BatchNormalization()(g, training=True)
if dropout:
g = Dropout(0.5)(g, training=True)
g = concatenate([g, skip_in])
g = Activation('relu')(g)
return g
def define_generator(image_shape=(8,8,13)):
init = RandomNormal(stddev=0.02)
in_image = Input(shape=image_shape)
e1 = define_encoder_block(in_image, 64, batchnorm=False)
e2 = define_encoder_block(e1, 128)
b = Conv2D(512, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(e2)
b = Activation('relu')(b)
d6 = decoder_block(b, e2, 128, dropout=False)
d7 = decoder_block(d6, e1, 64, dropout=False)
g = Conv2DTranspose(13, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d7)
out_image = Activation('softmax')(g)
model = Model(in_image, out_image)
return model
def define_gan(g_model, d_model, image_shape):
d_model.trainable = False
in_src = Input(shape=image_shape)
gen_out = g_model(in_src)
dis_out = d_model([in_src, gen_out])
model = Model(in_src, [dis_out, gen_out])
opt = Adam(lr=0.0002, beta_1=0.5)
model.compile(loss=['binary_crossentropy', 'mae'], optimizer=opt, loss_weights=[1,100])
return model
def generate_real_samples(dataset, n_samples, patch_shape):
trainA, trainB = dataset # dataset = [X, y]
ix = randint(0, trainA.shape[0], n_samples)
X1, X2 = trainA[ix], trainB[ix]
y = ones((n_samples, patch_shape, patch_shape, 1)) # output is always real
return [X1, X2], y
def generate_fake_samples(g_model, samples, patch_shape):
X = g_model.predict(samples)
y = zeros((len(X), patch_shape, patch_shape, 1)) # output is always fake
return X, y
def train(d_model, g_model, gan_model, dataset, n_epochs=100, n_batch=1):
n_patch = d_model.output_shape[1]
trainA, trainB = dataset
bat_per_epo = int(len(trainA) / n_batch)
n_steps = bat_per_epo * n_epochs # 15100 for test.txt (4 games, 151 moves) ~10,500,000 for games.txt
print('n_steps:', n_steps)
for i in range(n_steps):
[X_realA, X_realB], y_real = generate_real_samples(dataset, n_batch, n_patch)
X_fakeB, y_fake = generate_fake_samples(g_model, X_realA, n_patch)
d_loss1 = d_model.train_on_batch([X_realA, X_realB], y_real) # discrimininator loss on real data
d_loss2 = d_model.train_on_batch([X_realA, X_fakeB], y_fake) # discrimininator loss on fake data
g_loss, _, _ = gan_model.train_on_batch(X_realA, [y_real, X_realB])
print('>%d, d1[%.3f] d2[%.3f] g[%.3f]' % (i+1, d_loss1, d_loss2, g_loss))
image_shape = (8,8,13)
d_model = define_discriminator()
g_model = define_generator()
gan_model = define_gan(g_model, d_model, image_shape)
train(d_model, g_model, gan_model, [X,y])
Model.save(gan_model, 'models/gan_model')
import random
flatten = lambda l: [item for sublist in l for item in sublist]
instance = random.randint(1,len(X)-1) # len(X) = 18929 in test_big.txt
state = X[instance].reshape(1,8,8,13) # list of 8 matrices of 8x13 (zeros & ones; 13 different pieces)
action = gan_model.predict(state)[1] # list of 8 matrices of 8x13 (model output (tensors); the 13 numbers add to 100)
new_chess_dict = {}
for k, v in chess_dict.items():
new_chess_dict[tuple(v)] = k
def retranslate(action):
board = []
flatten_action = flatten(flatten(action))
for i in range(len(flatten_action)):
new_set = np.zeros((13,))
max_index = list(flatten_action[i]).index(max(flatten_action[i]))
new_set[max_index] = 1
board.append(new_set)
# len(board) = 64
for i in range(len(board)):
board[i] = new_chess_dict[tuple(board[i])]
board = np.array(board).reshape(8,8)
print(board)
retranslate(state)
retranslate(action)
My output:
The snippet in between the tensorflow gibberish shows that loss is only calculated at the beginning of the calculation and quickly changes to nan. I deleted some info code which shows that CUDA and cudnn were implemented successfully.
2021-06-01 19:21:59.858791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:21:59.860843: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-01 19:21:59.862149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:59.862221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:22:01.242343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-01 19:22:01.242369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-06-01 19:22:01.242375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-06-01 19:22:01.244898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2803 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2)
2021-06-01 19:22:02.043801: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-01 19:22:02.284522: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:21:54.076746: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.123386: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-06-01 19:21:58.188933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:58.189238: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.807361: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-01 19:21:58.807391: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-01 19:21:59.203448: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-06-01 19:21:59.257573: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-06-01 19:21:59.561624: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-06-01 19:21:59.837713: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-06-01 19:21:59.858693: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:21:59.858791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:21:59.860843: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-01 19:21:59.862149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:59.862221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:22:01.242343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-01 19:22:01.242369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-06-01 19:22:01.242375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-06-01 19:22:01.244898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2803 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2)
2021-06-01 19:22:02.043801: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-01 19:22:02.284522: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:22:03.425546: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
2021-06-01 19:21:54.076746: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.123386: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-06-01 19:21:58.188933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:58.189238: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.807361: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-01 19:21:58.807391: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-01 19:21:59.203448: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-06-01 19:21:59.257573: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-06-01 19:21:59.561624: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-06-01 19:21:59.837713: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-06-01 19:21:59.858693: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:21:59.858791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:21:59.860843: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-01 19:21:59.862149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:59.862221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:22:01.242343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-01 19:22:01.242369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-06-01 19:22:01.242375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-06-01 19:22:01.244898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2803 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2)
2021-06-01 19:22:02.043801: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-01 19:22:02.284522: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:22:03.425546: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
2021-06-01 19:22:05.146506: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-01 19:21:54.076746: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.123386: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-06-01 19:21:58.188933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:58.189238: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.807361: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-01 19:21:58.807391: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-01 19:21:59.203448: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-06-01 19:21:59.257573: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-06-01 19:21:59.561624: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-06-01 19:21:59.837713: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-06-01 19:21:59.858693: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:21:59.858791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:21:59.860843: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-01 19:21:59.862149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:59.862221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:22:01.242343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-01 19:22:01.242369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-06-01 19:22:01.242375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-06-01 19:22:01.244898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2803 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2)
2021-06-01 19:22:02.043801: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-01 19:22:02.284522: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:22:03.425546: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
2021-06-01 19:22:05.146506: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-01 19:22:06.043962: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
>1, d1[0.347] d2[0.347] g[nan]
>2, d1[0.346] d2[nan] g[nan]
>3, d1[nan] d2[nan] g[nan]
>4, d1[nan] d2[nan] g[nan]
>5, d1[nan] d2[nan] g[nan]
>6, d1[nan] d2[nan] g[nan]
>7, d1[nan] d2[nan] g[nan]
>8, d1[nan] d2[nan] g[nan]
>9, d1[nan] d2[nan] g[nan]
>10, d1[nan] d2[nan] g[nan]
2021-06-01 19:21:54.076746: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.123386: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-06-01 19:21:58.188933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:58.189238: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-06-01 19:21:58.807361: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-01 19:21:58.807391: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-01 19:21:59.203448: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-06-01 19:21:59.257573: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-06-01 19:21:59.561624: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-06-01 19:21:59.837713: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-06-01 19:21:59.858693: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:21:59.858791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:21:59.860843: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-01 19:21:59.862149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 960 computeCapability: 5.2
coreClock: 1.2785GHz coreCount: 8 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.46GiB/s
2021-06-01 19:21:59.862221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-06-01 19:22:01.242343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-01 19:22:01.242369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-06-01 19:22:01.242375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-06-01 19:22:01.244898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2803 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2)
2021-06-01 19:22:02.043801: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-06-01 19:22:02.284522: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-06-01 19:22:03.425546: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
2021-06-01 19:22:05.146506: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-06-01 19:22:06.043962: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-06-01 19:22:13.479273: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
INFO:tensorflow:Assets written to: models/gan_model\assets
[['.' '.' '.' '.' 'r' '.' '.' '.']
['.' '.' '.' '.' '.' '.' '.' '.']
['.' '.' '.' '.' '.' 'p' 'P' '.']
['p' 'p' '.' '.' '.' 'B' '.' '.']
['.' 'k' '.' '.' '.' 'P' 'P' '.']
['.' 'N' '.' '.' '.' '.' '.' '.']
['P' '.' 'K' '.' '.' '.' '.' '.']
['.' '.' '.' '.' '.' '.' '.' '.']]
Traceback (most recent call last):
File "C:\Users\EB\Desktop\ChessProject\edgarAI_GAN.py", line 398, in <module>
retranslate(action)
File "C:\Users\EB\Desktop\ChessProject\edgarAI_GAN.py", line 381, in retranslate
max_index = list(flatten_action[i]).index(max(flatten_action[i]))
ValueError: nan is not in list
