torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device

[sasikaran20]

Running on gpu NVIDIA TITAN XP NVIDIA-SMI 535.161.08 Driver Version: 535.161.08 CUDA Version: 12.2

faced the following error GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
/home/sasikarank/test/env/lib/python3.10/site-packages/torch/cuda/init.py:283: UserWarning:
Found GPU0 NVIDIA TITAN Xp which is of cuda capability 6.1.
Minimum and Maximum cuda capability supported by this version of PyTorch is
(7.0) - (12.0)

warnings.warn(
/home/sasikarank/test/env/lib/python3.10/site-packages/torch/cuda/init.py:304: UserWarning:
Please install PyTorch with a following CUDA
configurations: 12.6 following instructions at
https://pytorch.org/get-started/locally/

warnings.warn(matched_cuda_warn.format(matched_arches))
/home/sasikarank/test/env/lib/python3.10/site-packages/torch/cuda/init.py:326: UserWarning:
NVIDIA TITAN Xp with CUDA capability sm_61 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_70 sm_75 sm_80 sm_86 sm_90 sm_100 sm_120.
If you want to use the NVIDIA TITAN Xp GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/home/sasikarank/test/env/lib/python3.10/site-packages/torch/utils/data/dataloader.py:627: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(

| Name | Type | Params | Mode

0 | pre_processor | PreProcessor | 0 | train
1 | post_processor | PostProcessor | 0 | train
2 | evaluator | Evaluator | 0 | train
3 | model | EfficientAdModel | 8.1 M | train

8.1 M Trainable params
0 Non-trainable params
8.1 M Total params
32.235 Total estimated model params size (MB)
48 Modules in train mode
7 Modules in eval mode
Traceback (most recent call last):
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 49, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 598, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1011, in _run
results = self._run_stage()
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1055, in _run_stage
self.fit_loop.run()
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 208, in run
self.setup_data()
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 265, in setup_data
length = len(dl) if has_len_all_ranks(dl, trainer.strategy, allow_zero_length) else float("inf")
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/utilities/data.py", line 105, in has_len_all_ranks
if total_length == 0:
torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/sasikarank/test/train.py", line 268, in
main()
File "/home/sasikarank/test/train.py", line 240, in main
train(
File "/home/sasikarank/test/train.py", line 159, in train
engine.fit(model=model, datamodule= datamodule)
File "/home/sasikarank/test/env/lib/python3.10/site-packages/anomalib/engine/engine.py", line 416, in fit
self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 560, in fit
call._call_and_handle_interrupt(
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 70, in _call_and_handle_interrupt
trainer._teardown()
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1034, in _teardown
self.strategy.teardown()
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 536, in teardown
self.lightning_module.cpu()
File "/home/sasikarank/test/env/lib/python3.10/site-packages/lightning/fabric/utilities/device_dtype_mixin.py", line 82, in cpu
return super().cpu()
File "/home/sasikarank/test/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1150, in cpu
return self._apply(lambda t: t.cpu())
File "/home/sasikarank/test/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply
module._apply(fn)
File "/home/sasikarank/test/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 928, in _apply
module._apply(fn)
File "/home/sasikarank/test/env/lib/python3.10/site-packages/torchmetrics/metric.py", line 907, in _apply
_dummy_tensor = fn(torch.zeros(1, device=self.device))
torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[github-actions[bot]]

Thanks for submitting this issue! It has been added to our triage queue. A maintainer will review it shortly.

[rajeshgangireddy]

Hi @sasikaran20
This looks more like a torch + cuda issue on older GPUs than something specific to Anomalib.
From what I see here (and from the trace log you pasted) , latest torch + cuda might not be supported on this GPU.

Could you try installing torch with a CUDA <12, like so :
pip install "anomalib[cu118]"

if this doesn't work, then try:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

[knefenk]

Hi sir, just wanted to make sure that when i install with the command : uv pip install "anomalib[full,cu124]", It wont necessary install torch with cuda support by deafult? This happens for my case as GPU was not detected after installation when I try to train a Dinomaly model