RuntimeError: "ctc_loss_cuda" not implemented for 'BFloat16' #2764
Description
File "/workspace/svit_vsr/model/backbone/ctc.py", line 67, in forward
loss = self.ctc_loss(ys_hat, ys_pad, hlens, ys_lens)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/loss.py", line 1980, in forward
return F.ctc_loss(
^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/functional.py", line 3069, in ctc_loss
return torch.ctc_loss(
^^^^^^^^^^^^^^^
RuntimeError: "ctc_loss_cuda" not implemented for 'BFloat16'
this error when I train whisper large v3 with deepspeed as the examples wenetspeech ds config.
wish for help, thanks!
torch==2.5.1+cu124
deepspeed==0.16.1