pytorch-quantization ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

[yamashitadesu]

I installed pytorch-quantization, but I can not load the package.

Terminal:

# pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com pytorch-quantization
Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com
Collecting pytorch-quantization
  Downloading https://pypi.nvidia.com/pytorch-quantization/pytorch_quantization-2.2.1-cp38-cp38-linux_x86_64.whl (2.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 5.7 MB/s eta 0:00:00
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from pytorch-quantization) (1.24.2)
Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from pytorch-quantization) (2.1.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from pytorch-quantization) (1.10.1)
Collecting sphinx-glpi-theme (from pytorch-quantization)
  Downloading sphinx_glpi_theme-0.6-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting prettytable (from pytorch-quantization)
  Downloading prettytable-3.11.0-py3-none-any.whl.metadata (30 kB)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.8/dist-packages (from pytorch-quantization) (6.0.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.8/dist-packages (from prettytable->pytorch-quantization) (0.2.13)
Downloading prettytable-3.11.0-py3-none-any.whl (28 kB)
Downloading sphinx_glpi_theme-0.6-py2.py3-none-any.whl (4.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 57.5 MB/s eta 0:00:00
Installing collected packages: sphinx-glpi-theme, prettytable, pytorch-quantization
Successfully installed prettytable-3.11.0 pytorch-quantization-2.2.1 sphinx-glpi-theme-0.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[notice] A new release of pip is available: 24.3.1 -> 25.0
[notice] To update, run: python3 -m pip install --upgrade pip

# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jun_13_19:16:58_PDT_2023
Cuda compilation tools, release 12.2, V12.2.91
Build cuda_12.2.r12.2/compiler.32965470_0

# find /usr/local/cuda -name libcudart.so

# export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

# find /usr/local/cuda -name libcudart.so

# find /usr/local/cuda-12.2 -name libcudart.so
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudart.so

# export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64:$LD_LIBRARY_PATH

# echo $LD_LIBRARY_PATH
/usr/local/cuda-12.2/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:

Python code:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data as data
import torchvision.transforms as transforms
import torchvision.datasets as datasets
import torch_tensorrt

from torch.utils.tensorboard import SummaryWriter

import pytorch_quantization
from pytorch_quantization import nn as quant_nn
from pytorch_quantization import quant_modules
from pytorch_quantization.tensor_quant import QuantDescriptor
from pytorch_quantization import calib
from tqdm import tqdm

print(pytorch_quantization.__version__)

import os
import sys
sys.path.insert(0, "../examples/int8/training/vgg16")
from vgg16 import vgg16

The encountered error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[7], line 12
      8 import torch_tensorrt
     10 from torch.utils.tensorboard import SummaryWriter
---> 12 import pytorch_quantization
     13 from pytorch_quantization import nn as quant_nn
     14 from pytorch_quantization import quant_modules

File /usr/local/lib/python3.8/dist-packages/pytorch_quantization/__init__.py:20
     18 from absl import logging
     19 from .version import __version__
---> 20 from .quant_modules import *
     22 logging.use_absl_handler()

File /usr/local/lib/python3.8/dist-packages/pytorch_quantization/quant_modules.py:23
     20 from contextlib import contextmanager
     22 import torch
---> 23 from pytorch_quantization import nn as quant_nn
     25 __all__ = ['initialize', 'deactivate', 'enable_onnx_export']
     27 # Definition of the named tuple that is used to store mapping of the quantized modules

File /usr/local/lib/python3.8/dist-packages/pytorch_quantization/nn/__init__.py:19
      1 #
      2 # SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
      3 # SPDX-License-Identifier: Apache-2.0
   (...)
     15 # limitations under the License.
     16 #
---> 19 from pytorch_quantization.nn.modules.tensor_quantizer import *
     20 from pytorch_quantization.nn.modules.quant_conv import *
     21 from pytorch_quantization.nn.modules.quant_linear import *

File /usr/local/lib/python3.8/dist-packages/pytorch_quantization/nn/modules/tensor_quantizer.py:24
     21 import torch
     22 from torch import nn
---> 24 from pytorch_quantization.tensor_quant import QuantDescriptor, tensor_quant, fake_tensor_quant, scaled_e4m3
     25 from pytorch_quantization.nn.modules.clip import Clip
     27 from pytorch_quantization import calib

File /usr/local/lib/python3.8/dist-packages/pytorch_quantization/tensor_quant.py:28
     25 import torch._C._onnx as _C_onnx
     26 from torch.autograd import Function
---> 28 from pytorch_quantization import cuda_ext
     30 from torch.onnx import symbolic_helper
     33 class ScaledQuantDescriptor():

ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

Versions:
Python 3.8.10
PyTorch 2.4.1+cu121

How can I solve this error?

[kevinch-nv]

pytorch-quantization is a deprecated package (from the error message, it's looking for CUDA 11 libraries), we recommend using https://github.com/NVIDIA/TensorRT-Model-Optimizer to optimize your models now.

[yamashitadesu]

@kevinch-nv
Thank you for your response and for introducing TensorRT-Model-Optimizer. I appreciate the detailed information.

I have one question. I am interested in applying pruning and quantization to computer vision models, including ResNet, rather than LLMs. The documentation primarily discusses LLM-related optimizations. Is it also possible to use Model Optimizer for computer vision models?

I would appreciate any clarification on this.

Thank you.

[kevinch-nv]

You can look at this example: https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/examples/onnx_ptq/README.md

I would recommended following up in the Model-Opt repo for more information on your specific usecase