TRT-LLM loading mechanism tool #3398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

apbose wants to merge 1 commit into main from nccl_ops_trt_llm_installation

Collaborator

apbose commented Feb 14, 2025 •

edited

Loading

TRT-LLM download utility

apbose self-assigned this

facebook-github-bot added the cla signed label

apbose marked this pull request as draft

February 14, 2025 17:58

github-actions bot added component: conversion component: api [Python] component: dynamo labels

github-actions bot requested a review from peri044

February 14, 2025 17:58

apbose force-pushed the nccl_ops_trt_llm_installation branch from 57dbb3f to 3e38e87 Compare

February 25, 2025 14:34

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/conversion/converter_utils.py Outdated

    
                              f"Ensure the path is correct and the library is compatible",

                              exc_info=e_os_error,

                      else:

                          py_version = f"cp{sys.version_info.major}{sys.version_info.minor}"

Collaborator

narendasan Feb 25, 2025

Why do we restrict to cp310 and cp312, It shouldnt matter if we are pulling the whl and unzipping ourselves

Collaborator Author

apbose Feb 27, 2025

https://pypi.nvidia.com/tensorrt-llm/ In this since I see the tags for only cp310 and cp312 I added the check

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/conversion/converter_utils.py Outdated Show resolved Hide resolved

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/conversion/converter_utils.py Show resolved Hide resolved

github-actions bot added the component: tests label

github-actions bot requested changes

View reviewed changes

github-actions bot left a comment

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:03:00.014038+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:03:24.885031+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

apbose force-pushed the nccl_ops_trt_llm_installation branch from 9ba407b to 5f3fdac Compare

February 27, 2025 20:05

apbose marked this pull request as ready for review

February 27, 2025 20:05

github-actions bot requested changes

View reviewed changes

github-actions bot left a comment

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:05:38.023287+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:06:02.662188+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

github-actions bot requested changes

View reviewed changes

github-actions bot left a comment

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:05:54.405311+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-02-27 20:06:21.454993+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

apbose force-pushed the nccl_ops_trt_llm_installation branch from 5f3fdac to b66350e Compare

April 15, 2025 19:57

github-actions bot requested changes

View reviewed changes

github-actions bot left a comment

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 19:58:05.267724+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 19:58:36.145897+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

apbose changed the title ~~change in TRT-LLM loading mechanism and exposing aot_joint_export in _compiler.py~~ TRT-LLM loading mechanism tool

github-actions bot requested changes

View reviewed changes

github-actions bot left a comment

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 21:00:13.719714+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/conversion/test_nccl_ops.py	2025-04-15 21:00:40.093669+00:00
@@ -22,11 +22,11 @@

from .harness import DispatchTestCase


class TestGatherNcclOpsConverter(DispatchTestCase):
-    @parameterized.expand([(8)])
+    @parameterized.expand([8])
    def test_nccl_ops(self, linear_layer_dim):
        class DistributedGatherModel(nn.Module):
            def __init__(self, input_dim):
                super().__init__()
                self.fc = torch.nn.Linear(input_dim, input_dim)

github-actions bot removed the component: tests label

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from 6e893ed to 77f2145 Compare

April 18, 2025 00:51

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/_compiler.py Outdated Show resolved Hide resolved

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

apbose force-pushed the nccl_ops_trt_llm_installation branch from f30acb7 to 9c238ae Compare

April 29, 2025 21:33

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from 89d621d to 27aa2f2 Compare

May 2, 2025 02:36

apbose commented

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated Show resolved Hide resolved

apbose commented

View reviewed changes

tests/py/dynamo/distributed/test_nccl_ops.py Show resolved Hide resolved

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Show resolved Hide resolved

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from dbfd7ee to 15d681a Compare

July 4, 2025 00:40

narendasan reviewed

View reviewed changes

Collaborator

narendasan left a comment

Seems like its close are there any tests for the downloader? like verifying the correct file is downloaded and available?

Collaborator Author

apbose commented Jul 7, 2025

The test tests/py/dynamo/distributed/test_nccl_ops.py should verify it. The USE_TRTLLM_PLUGINS=1 pytest distributed/test_nccl_ops.py is the command. It is not part of CI though

apbose force-pushed the nccl_ops_trt_llm_installation branch from c8b8337 to 340182b Compare

July 7, 2025 21:20

apbose force-pushed the nccl_ops_trt_llm_installation branch 4 times, most recently from 1502865 to 7c99c11 Compare

July 24, 2025 21:51

narendasan reviewed

View reviewed changes

.github/scripts/install-torch-tensorrt.sh Outdated

    
                  install_cuda_aarch64

              fi

              if [[ "$(uname -s)" == "Linux" && "$(uname -m)" == "x86_64" ]]; then

Collaborator

narendasan Jul 28, 2025

Can this just be called in the test script instead installing it for all tests?

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated

    
                          "TensorRT-LLM plugins for NCCL backend are not supported on Windows"

                      )

                      return False

                  if "aarch64" in platform:

Collaborator

narendasan Jul 28, 2025

What about SBSA?

Collaborator Author

apbose Jul 31, 2025

aarch64 should take care of it, right? I should change the log message to maybe "TensorRT-LLM plugins for NCCL backend are not supported on Jetson and sbsa devices (aarch64)"?

apbose force-pushed the nccl_ops_trt_llm_installation branch 3 times, most recently from 6550195 to 76f7068 Compare

August 7, 2025 18:53

narendasan reviewed

View reviewed changes

py/torch_tensorrt/dynamo/utils.py Outdated

    
                      )

                      return False

                  if torch.cuda.is_available():

                      device_name = torch.cuda.get_device_name().lower()

Collaborator

narendasan Aug 7, 2025

Did you test this, I was recommending using this: https://docs.python.org/ro/3/library/platform.html#platform.release

We use it here:

TensorRT/pyproject.toml

Line 54 in e760c3c

    
           "torch>=2.9.0.dev,<2.10.0; platform_machine != 'aarch64' or (platform_machine == 'aarch64' and 'tegra' not in platform_release)",

Collaborator Author

apbose Aug 8, 2025

ok changed to the above

narendasan reviewed

View reviewed changes

.github/workflows/build-test-linux-x86_64.yml Show resolved Hide resolved

narendasan reviewed

View reviewed changes

Collaborator

narendasan left a comment

add the test setup for sbsa

apbose force-pushed the nccl_ops_trt_llm_installation branch 4 times, most recently from 8ea4010 to 8e59794 Compare

August 13, 2025 04:53

apbose force-pushed the nccl_ops_trt_llm_installation branch 2 times, most recently from d656160 to 86d39ad Compare

August 19, 2025 06:00

apbose force-pushed the nccl_ops_trt_llm_installation branch from 86d39ad to 2f6ff54 Compare

September 19, 2025 20:11


          TensorRT-LLM import fix and aot_joint_export specify as explicit sett…

68b3346

…ing in dynamo.compile

TRT-LLM installation utilities and adding test cases

adding the option in _compiler.py

changes in the TRT-LLM loading tool- removing install_wget, install_unzip, install_mpi

Further changes in error logging of the TRT-LLM installation tool

moving the load_tensorrt_llm to dynamo/utils.py

correcting misprint for TRT LLM load

Using python lib for download to make it platform agnostic

dll file path update for windows

correcting the non critical lint error

Including version in versions.txt

linting error fixes and rebase fix

removing Platform enum from converter_utils.py

Addressing review comments- tmp dir for wheel download and wheel extraction, variable for py_version

checks for windows where NCCL backend is not supported

adding checks for windows and jetson devices

Keeping the extracted and deleting download file, restructuring test

modifying the error warning of missing libmpi libs

removing the redundant initializations

adding tests in CI

correcting the skip test condition

install MPI libs for linux x86

adding SBSA to the supported platform of TRT-LLM libs and installing MPI libs for the distributed tests

Using python package for platform detection

using python platform

apbose force-pushed the nccl_ops_trt_llm_installation branch from 2f6ff54 to 68b3346 Compare

September 19, 2025 20:23

Collaborator Author

apbose commented Sep 22, 2025

Closing in favor of #3829

apbose closed this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] component: build system component: conversion component: dynamo component: tests