RuntimeError：CUDA error: CUBLAS_STATUS_EXECUTION_FAILED

[hxzwqw]

COMMAND: python main.py --encoder cnn --decoder rnn --encoder-dropout 0.05 --decoder-dropout 0.2

Namespace(batch_size=1, cuda=True, decoder='rnn', decoder_dropout=0.2, decoder_hidden=256, dims=6, dynamic_graph=True, edge_types=4, encoder='cnn', encoder_dropout=0.05, encoder_hidden=256, epochs=500, factor=True, gamma=0.5, hard=True, load_folder='', lr=0.0005, lr_decay=200, no_cuda=False, no_factor=False, num_residues=77, number_exp=56, number_expstart=0, prediction_steps=1, prior=True, save_folder='logs', seed=42, skip_first=True, temp=0.5, timesteps=50, var=5e-05)
Testing with dynamically re-computed graph.
Using factor graph CNN encoder.
Using learned recurrent interaction net decoder.
Using prior
[0.91 0.03 0.03 0.03]
Start Training...
Traceback (most recent call last):
File "main.py", line 419, in
epoch, best_val_loss)
File "main.py", line 209, in train
logits = encoder(data, rel_rec, rel_send)
File "/home/hxz/anaconda3/envs/netw_2.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/media/hxz/Mdel/NRI-MD/NRI-MD/modules.py", line 213, in forward
edges = self.node2edge_temporal(inputs, rel_rec, rel_send)
File "/media/hxz/Mdel/NRI-MD/NRI-MD/modules.py", line 182, in node2edge_temporal
receivers = torch.matmul(rel_rec, x)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

Description:
I have tried the default pdb file and even the default setting(without cnn or rnn),the error always occur.
I have also tried to follow the error report and tried to debug, it seems endless to track.
Thanks if someone or the authur can give me some advice.

[DivingWhale]

I met the same situation. Did you solve it?

[UR-Free]

It seems like a problem with cuda
I solve it by turning off the torch.cuda, and just let it calculate on cpu