Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

[skyuuka]

I got the following error when running python main_unset.py --training

2019-09-12 21:17:30.594167: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2019-09-12 21:17:31.140393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1a:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.324298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:1b:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.508353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 2 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:3d:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.691047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 3 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:3e:00.0
totalMemory: 10.73GiB freeMemory: 10.57GiB
2019-09-12 21:17:31.691543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1, 2, 3
2019-09-12 21:17:33.385536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-12 21:17:33.385595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 2 3 
2019-09-12 21:17:33.385603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N N N N 
2019-09-12 21:17:33.385608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   N N N N 
2019-09-12 21:17:33.385613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 2:   N N N N 
2019-09-12 21:17:33.385618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 3:   N N N N 
2019-09-12 21:17:33.386348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10439 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:1a:00.0, compute capability: 7.5)
2019-09-12 21:17:33.589232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10439 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:1b:00.0, compute capability: 7.5)
2019-09-12 21:17:33.799606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 10439 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:3d:00.0, compute capability: 7.5)
2019-09-12 21:17:33.995857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10439 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:3e:00.0, compute capability: 7.5)
2019-09-12 21:18:10.689782: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Fatal Python error: Segmentation fault

Thread 0x00007f1e85266740 (most recent call first):
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350 in _call_tf_sessionrun
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1263 in _run_fn
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1278 in _do_call
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1272 in _do_run
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1100 in _run
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 877 in run
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4986 in _run_using_default_session
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2241 in run
  File "/home/skyuuka/workspace/FewShot_GAN-Unet3D/unet3D/model_unet.py", line 213 in train
  File "main_unet.py", line 62 in main
  File "/home/skyuuka/anaconda2/envs/FewShot_GAN-Unet3D/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 125 in run
  File "main_unet.py", line 69 in <module>

[little1TouwaErio]

@ @skyuuka Hello, did you successfully run the original code？

[skyuuka]

@ @skyuuka Hello, did you successfully run the original code？

Yes. I tried newer version of TF > 1.7.0, and it works.

[little1TouwaErio]

我这边可以加你QQ 请教一下吗 非常感谢啊 发自我的iPhone

…

------------------ Original ------------------ From: Lin Chen <[email protected]&gt; Date: Tue,Oct 15,2019 0:53 PM To: arnab39/FewShot_GAN-Unet3D <[email protected]&gt; Cc: Lililicg <[email protected]&gt;, Comment <[email protected]&gt; Subject: Re: [arnab39/FewShot_GAN-Unet3D] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR (#12)

[little1TouwaErio]

Hello, my registration on iseg&nbsp; website has not passed. Could you please share the dataset with me？

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人:&nbsp;"Lin Chen"<[email protected]&gt;; 发送时间:&nbsp;2019年10月15日(星期二) 中午12:53 收件人:&nbsp;"arnab39/FewShot_GAN-Unet3D"<[email protected]&gt;; 抄送:&nbsp;"416580793"<[email protected]&gt;;"Comment"<[email protected]&gt;; 主题:&nbsp;Re: [arnab39/FewShot_GAN-Unet3D] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR (#12) @ @skyuuka Hello, did you successfully run the original code？ Yes. I tried newer version of TF &gt; 1.7.0, and it works. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[hkthirano]

If the version of tensorflow is 1.7.0 and the version of cudnn is 7.0.5, the program run.
However, If the version of cudnn is 7.3.0, the program does not run.