9

I run my code on tensorflow 2.1.0 Anaconda with CUDA Toolkit 10.1 CUDNN 7.6.0 (Windows 10) and it returns a issue

F .\tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device

My GPU : GT940MX Compute Capability 5.0

I already run the nvcc -V and it returns :

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:26_Pacific_Standard_Time_2019
Cuda compilation tools, release 10.1, V10.1.105

This is the full result :

2020-08-05 10:05:48.368012: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 10:06:00.488544: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-05 10:06:48.153611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce 940MX computeCapability: 5.0
coreClock: 0.8605GHz coreCount: 4 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-05 10:06:48.164731: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 10:06:48.245826: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 10:06:48.296245: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 10:06:48.338860: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 10:06:48.439393: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 10:06:48.489830: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 10:06:48.941872: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 10:06:48.946651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 10:06:48.951881: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-05 10:06:48.979077: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23d29b660d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-05 10:06:48.985680: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-05 10:06:48.990616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce 940MX computeCapability: 5.0
coreClock: 0.8605GHz coreCount: 4 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-05 10:06:49.003356: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 10:06:49.009869: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 10:06:49.014858: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 10:06:49.020699: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 10:06:49.028876: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 10:06:49.033607: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 10:06:49.039192: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 10:06:49.045288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 10:06:49.218497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-05 10:06:49.223536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-08-05 10:06:49.226857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-08-05 10:06:49.230413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1460 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0, compute capability: 5.0)
2020-08-05 10:06:49.244107: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23d301b8fa0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-05 10:06:49.250377: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce 940MX, Compute Capability 5.0
2020-08-05 10:06:49.446601: F .\tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device

What are the issues and how to fix it?

talonmies
  • 70,661
  • 34
  • 192
  • 269
MichaelJamesL
  • 131
  • 1
  • 6
  • The version of tensorflow you have has been built without support for your GPU – talonmies Aug 05 '20 at 03:42
  • 1
    @talonmies I have been meet Tensorflow requirements(Cuda compute capability >3.5) – MichaelJamesL Aug 05 '20 at 09:03
  • I'm facing exactly the same problem. @talonmies please provide info if you can which tensorflow version will be compatible because on the tensorflow website it was said that version 2.3 will be ok with CUDA 10.1 and cuDNN 7.6? – Martin Aug 05 '20 at 09:53
  • It isn't whether a particular release of Tensorflow can theoretically support your GPU. It is whether the people than built the binary version you have installed chose to compile in support for your GPU. That question needs to be posed to the people who built the binary you have. – talonmies Aug 05 '20 at 10:12
  • @talonmies Can i run the tensorflow by build it from source with bazel? – MichaelJamesL Aug 05 '20 at 11:41
  • I have no idea, sorry. I can tell you exactly what the problem is at the GPU level, but I know nothing (or really care) about Tensorflow, I'm afraid – talonmies Aug 05 '20 at 12:42
  • @talonmies no problem, thank you – MichaelJamesL Aug 05 '20 at 13:22
  • I tried both: 1.GTX 960M+ cuDNN7.4(for CUDA10.0)+CUDA10.1+tensorflow2.3 2.GTX 960M+ cuDNN7.6(for CUDA10.1)+CUDA10.1+tensorflow2.3 the same issue remains. – Max xu Aug 14 '20 at 18:43

4 Answers4

4

Looks like this is an issue with Python 3.8 and Tensorflow 2.3. I tried the tensorflow 2.3.0 with python 3.7, but it returns an error with python 3.7 because python38.dll (I don't remember exactly the error and i already delete the env), anyway i used python 3.7 on anaconda env and installed tensorflow 2.1.0 with pip and it works.

I posted the question too in github and this question is answered in github https://github.com/tensorflow/tensorflow/issues/42052

MichaelJamesL
  • 131
  • 1
  • 6
  • 1
    Thanks for the tip! The following combination seemed to have worked for me... GPU: GeForce GTX 750 Ti, python 3.7.8, tf version 2.1.1, cuda-v7.6.5.32 – Jason Turan Oct 03 '20 at 03:24
3

As per the screenshot below, Tensorflow Versions 2.1, 2.2 and 2.3 work with cuDNN version of 7.4 but cuDNN version of your GPU is 7.6.

enter image description here

That might be, most probably, the reason for the error.

Solution is to downgrade cuDNN Version of your GPU.

Existing Version of cuDNN can be uninstalled through the Windows Control Panel by using the Programs and Features widget.

New Version of cuDNN can be installed as shown in this NVIDIA Installation Guide.

Also, please refer this Github Issue to know more on how to downgrade cuDNN Version.

The above screenshot has been taken from this Tensorflow Documentation.

  • But there is no cuDNN 7.4 that support CUDA 10.1 @Tensorflow Support – MichaelJamesL Aug 08 '20 at 06:14
  • You can try downgrading cuDNN Version to 7.4. –  Aug 08 '20 at 14:35
  • I tried the cuDNN 7.4 for **CUDA 10.1 (because there is no cuDNN 7.4 for CUDA 10.1)** and it returns the **same** issue `F .\tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device (D:\Tensorflow\anaconda) PS D:\Tensorflow\TensorFlow-2.x-YOLOv3>` – MichaelJamesL Aug 08 '20 at 17:09
  • I tried the cuDNN 7.4 for CUDA 10.1 (because there is no cuDNN 7.4 for CUDA 10.1) ==> This sentence is bit confusing. Can you please rephrase it. Thanks! –  Aug 09 '20 at 01:59
  • Sorry i mean : I tried the cuDNN 7.4 for CUDA 10.0 (because there is no cuDNN 7.4 for CUDA 10.1) and it returns the same issue @Tensorflow Support – MichaelJamesL Aug 11 '20 at 05:24
  • 2023 update: tensorflow-gpu is a [deprecated library](https://pypi.org/project/tensorflow-gpu/). Use regular tensorflow instead – tandav Apr 04 '23 at 07:54
0

I have the same problem, my cuDNN is 8.0.2. As you say, there is no cuDNN 7.4 for CUDA 10.1. So, I tried the cuDNN 7.5 for CUDA 10.1 and it works!!!! Hope my experience can help someone else. :) enter image description here

Amoyensis
  • 163
  • 5
0

Seem like some cuDNN only support by some specific versions of tensorflow.

As a Window user this is how I do:

  1. Check which version that which Tensorflow and CUDA version combinations are compatible (you can click on other OS on the left)
  2. As Rock Jefferson comment you can use cuDNN 7.5 for CUDA 10.1. It work for me. Download here

Try it. Hope it useful for U guy.

BK-DDT Lê Minh Quang
  • 1
  • I was having this problem with CUDA 10.1 and cuDNN 7.6.5 (the latest version listed to be compatible with 10.1). I tried downgrading to cuDNN 7.5.1 but it had the same problem. Same for 7.5.0. – yoyo Oct 30 '20 at 05:06