Replies: 3 comments 10 replies
-
|
Hello - thank you for the question. I am able to reproduce the error in the TorchScript path on the latest Using Latest NightlyCompile Modelimport torch
import torch_tensorrt
...
# Model Definition
...
trt_optimized_module = torch.compile(model,
backend="tensorrt",
dynamic=False,
options={"truncate_long_and_double": True,
"enabled_precisions": {torch.half}})
out = trt_optimized_module(torch.randn((1, 3, 224, 224)).cuda())Regarding TorchScript, it seems that this model also cannot be traced with JIT (only scripted), since |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Thank you. I installed the following versions: And now I get this error when running These are the available backends: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the follow-up - could you verify that |
Beta Was this translation helpful? Give feedback.
-
|
Yes it does. import torch_tensorrt
import torchvision
import torch
model = torchvision.models.detection.maskrcnn_resnet50_fpn_v2(weights_backbone=None)
model.cuda()
model.eval()
input_shape = (3, 224, 224)
x = torch.randn(1, *input_shape, device='cuda')
out = model(x)
# Define inputs
inputs = [
torch_tensorrt.Input(
min_shape=[1, *input_shape],
opt_shape=[1, *input_shape],
max_shape=[1, *input_shape],
dtype=x.dtype,
)
]
# Compile to TensorRT
trt_optimized_module = torch.compile(model,
backend="tensorrt",
dynamic=False,
options={"truncate_long_and_double": True,
"enabled_precisions": {torch.half}}) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the follow-up - based on the version of Torch-TRT you are using ( |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I have to keep to versions that are available through NGC, so I can't install nightly builds. When I run with |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Thank you for the follow-up. This error message looks familiar and I believe we addressed a similar issue in a recent PR. Could you share the output of |
Beta Was this translation helpful? Give feedback.
-
|
Sure, the version of |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply - I tested out the sample code on the latest Additionally, we are working on the converters for two currently-unsupported operators for this model, |
Beta Was this translation helpful? Give feedback.
-
|
So I'm waiting for either a NGC version that would contain the fix, or for the converter? |
Beta Was this translation helpful? Give feedback.
-
|
The converter is not needed for the model to work; it would just improve performance by reducing segmentation. The NGC container with the fix (or using the |
Beta Was this translation helpful? Give feedback.
-
|
Hello. I'm encountering a similar problem. I'm using the latest Nvidia container (nvcr.io/nvidia/pytorch:24.04-py3), and deploying the same code as above. The compiling part works fine, but the scripting for the TensorScript model fails, with the following error: I'm using this modified code (the first script, before the compiling, works. The second fail): Scripting with example_inputs or with tracing did not work as well. I've also installed MonkeyType in the container. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Hi - thanks for the question. The issue here is the usage of |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Here is a minimal example reproducing the issue:
I get this error:
How do I make this conversion?
Beta Was this translation helpful? Give feedback.
All reactions