CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasGemmEx

[qxzha]

when I run the sh script, I got an error:

./demo/run_alien_cobble.sh

---

Traceback (most recent call last):
File "/home/zqx/text2mesh/main.py", line 499, in
run_branched(args)
File "/home/zqx/text2mesh/main.py", line 140, in run_branched
encoded_text = clip_model.encode_text(prompt_token)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 348, in encode_text
x = self.transformer(x)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 203, in forward
return self.resblocks(x)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 190, in forward
x = x + self.attention(self.ln_1(x))
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 187, in attention
return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0]
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1153, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/functional.py", line 5066, in multi_head_attention_forward
q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/functional.py", line 4745, in _in_projection_packed
return linear(q, w, b).chunk(3, dim=-1)
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)

---

Anyone knows how to fix it, please tell me.