[Tensorflow] . failed to create cublas handle: CUBLAS_STATUS_INTERNAL_ERROR

Tensorflow 2에서 자꾸 아래와 같은 오류가 나서 고생 했습니다.

 

2019-12-30 01:42:43.383561: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_100.dll'; dlerror: cublas64_100.dll not found 
2019-12-30 01:42:43.383761: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_INTERNAL_ERROR 
2019-12-30 01:42:43.384698: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_INTERNAL_ERROR 
2019-12-30 01:42:43.385885: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_INTERNAL_ERROR 
2019-12-30 01:42:43.386065: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support 
2019-12-30 01:42:43.386224: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Blas GEMV launch failed:  m=1, n=4 
 [[{{node sequential/dense/MatMul}}]]





[Op:__inference_distributed_function_418]



Function call stack: 
distributed_function

 

이게 처음에 되던게 어느 순간 안되면 멘붕이 일어납니다.

 

아니 난 아무것도 안만졌는데 왜 아깐되고 지금은 안되는 것인가...

 

알고 봤더니 tensorflow GPU 메모리 문제였습니다.

 

간단하게

import os
os.environ["CUDA_VISIBLE_DEVICES"] = '1'

이 코드로 해결했습니다.

 

GPU '1' 만 지정하도록 강제하는 방법입니다.

 

mickael-k.tistory.com/32

windstop.tistory.com/159