no kernel image is available error #848

Ishihara-Masabumi · 2023-01-13T01:02:21Z

Ishihara-Masabumi
Jan 13, 2023

When I tried to run using the following command line,

python3 tools/train.py --config anomalib/models/padim/config.yaml

the following error saying "no kernel image is available" occurrd.

2023-01-13 00:58:31,025 - pytorch_lightning.callbacks.model_summary - INFO - 
  | Name                  | Type                     | Params
-------------------------------------------------------------------
0 | image_threshold       | AnomalyScoreThreshold    | 0     
1 | pixel_threshold       | AnomalyScoreThreshold    | 0     
2 | model                 | PadimModel               | 2.8 M 
3 | image_metrics         | AnomalibMetricCollection | 0     
4 | pixel_metrics         | AnomalibMetricCollection | 0     
5 | normalization_metrics | MinMax                   | 0     
-------------------------------------------------------------------
2.8 M     Trainable params
0         Non-trainable params
2.8 M     Total params
11.131    Total estimated model params size (MB)
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
    results = self._run_stage()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1323, in _run_stage
    return self._run_train()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1353, in _run_train
    self.fit_loop.run()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 199, in run
    self.on_run_start(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 207, in on_run_start
    self.trainer.reset_train_dataloader(self.trainer.lightning_module)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1891, in reset_train_dataloader
    if has_len_all_ranks(self.train_dataloader, self.strategy, module)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/data.py", line 126, in has_len_all_ranks
    if total_length == 0:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tools/train.py", line 75, in <module>
    train()
  File "tools/train.py", line 61, in train
    trainer.fit(model=model, datamodule=datamodule)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
    self._call_and_handle_interrupt(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 738, in _call_and_handle_interrupt
    self._teardown()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1300, in _teardown
    self.strategy.teardown()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/strategies/single_device.py", line 96, in teardown
    self.lightning_module.cpu()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pytorch_lightning/core/mixins/device_dtype_mixin.py", line 147, in cpu
    return super().cpu()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 738, in cpu
    return self._apply(lambda t: t.cpu())
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 579, in _apply
    module._apply(fn)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torchmetrics/metric.py", line 650, in _apply
    self._device = fn(torch.zeros(1, device=self.device)).device
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Please let me know how to fix it !

Answered by samet-akcay

Jan 18, 2023

As I mentioned previously, there is a problem with your torch and cuda compatibility for your GPU card. Your have '1.12.0+cu102' as the pytorch version. It is probably not compatible with your GPU architecture. If you install pytorch using the following, you could potentially fix your problem:

pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

I'll be converting this to a discussion since it is not an anomalib issue, but feel free to continue from the discussions.

View full answer

samet-akcay · 2023-01-13T12:29:10Z

samet-akcay
Jan 13, 2023
Maintainer

it's probably an issue with your torch installation. Can you check the following questions?

Do you have a GPU on your machine?
If yes, do you have a cuda installed on your system? If so, what's your cuda version?
What's your torch version?

0 replies

Ishihara-Masabumi · 2023-01-16T00:08:33Z

Ishihara-Masabumi
Jan 16, 2023
Author

I answer the questions above.

・Yes.
・Yes, 11.6.
・1.12.0.

That's it.

0 replies

samet-akcay · 2023-01-16T06:13:36Z

samet-akcay
Jan 16, 2023
Maintainer

Thanks, can you also tell me what the outputs of these are ?

>>> import torch
>>> torch.__version__
>>> torch.cuda.is_available()

For example, mine are as follows:

>>> import torch
>>> torch.__version__
'1.12.0+cu102'
>>> torch.cuda.is_available()
True

0 replies

Ishihara-Masabumi · 2023-01-16T06:30:51Z

Ishihara-Masabumi
Jan 16, 2023
Author

My output is as follows:

$ python3
Python 3.8.10 (default, Nov 14 2022, 12:59:47) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.12.0+cu102'
>>> torch.cuda.is_available()
True
>>>

$ nvidia-smi
Mon Jan 16 06:31:10 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.08    Driver Version: 510.73.08    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A10G         On   | 00000000:00:1E.0 Off |                    0 |
|  0%   21C    P8    15W / 300W |      0MiB / 23028MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

0 replies

samet-akcay · 2023-01-16T08:55:02Z

samet-akcay
Jan 16, 2023
Maintainer

This is weird. Despite things are looking normal, I still think that it is related to your cuda and torch installation. What's the output of the following in your terminal?

>>> import torch
>>> torch.rand((32, 3, 256, 256)).cuda()

0 replies

Ishihara-Masabumi · 2023-01-18T07:54:09Z

Ishihara-Masabumi
Jan 18, 2023
Author

The results of the above python command is as follwos:

$ python3
Python 3.8.10 (default, Nov 14 2022, 12:59:47) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand((32, 3, 256, 256)).cuda()
/home/ubuntu/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:146: UserWarning: 
NVIDIA A10G with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA A10G GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor.py", line 338, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 481, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 447, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 270, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 302, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 302, in <listcomp>
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 304, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in self])
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 304, in <listcomp>
    return torch.stack([get_summarized_data(x) for x in self])
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 302, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 302, in <listcomp>
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/home/ubuntu/.local/lib/python3.8/site-packages/torch/_tensor_str.py", line 295, in get_summarized_data
    return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
>>>

0 replies

samet-akcay · 2023-01-18T16:55:10Z

samet-akcay
Jan 18, 2023
Maintainer

As I mentioned previously, there is a problem with your torch and cuda compatibility for your GPU card. Your have '1.12.0+cu102' as the pytorch version. It is probably not compatible with your GPU architecture. If you install pytorch using the following, you could potentially fix your problem:

pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

I'll be converting this to a discussion since it is not an anomalib issue, but feel free to continue from the discussions.

1 reply

Ishihara-Masabumi Jan 19, 2023
Author

I installed as your instruction above.
Then, I did it.
Thanks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no kernel image is available error #848

{{title}}

Replies: 7 comments 1 reply

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

no kernel image is available error #848

Ishihara-Masabumi Jan 13, 2023

Replies: 7 comments · 1 reply

samet-akcay Jan 13, 2023 Maintainer

Ishihara-Masabumi Jan 16, 2023 Author

samet-akcay Jan 16, 2023 Maintainer

Ishihara-Masabumi Jan 16, 2023 Author

samet-akcay Jan 16, 2023 Maintainer

Ishihara-Masabumi Jan 18, 2023 Author

samet-akcay Jan 18, 2023 Maintainer

Ishihara-Masabumi Jan 19, 2023 Author

Ishihara-Masabumi
Jan 13, 2023

Replies: 7 comments 1 reply

samet-akcay
Jan 13, 2023
Maintainer

Ishihara-Masabumi
Jan 16, 2023
Author

samet-akcay
Jan 16, 2023
Maintainer

Ishihara-Masabumi
Jan 16, 2023
Author

samet-akcay
Jan 16, 2023
Maintainer

Ishihara-Masabumi
Jan 18, 2023
Author

samet-akcay
Jan 18, 2023
Maintainer

Ishihara-Masabumi Jan 19, 2023
Author