Skip to content

[NvTensorRTRTX EP] Enable automatic selection of NvTensorRTRTX EP for PREFER_GPU policy #24689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

nieubank
Copy link

@nieubank nieubank commented May 8, 2025

Description

Enables automatic selection of NvTensorRTRTX EP for PREFER_GPU policy similar to #24629

Sample code from unit test:

  auto env = Ort::Env();

  env.RegisterExecutionProviderLibrary(kNvTensorRTRTXExecutionProvider, ORT_TSTR("onnxruntime_providers_nv_tensorrt_rtx.dll"));

  Ort::SessionOptions so;
  so.SetEpSelectionPolicy(OrtExecutionProviderDevicePolicy_PREFER_GPU);
  Ort::Session session_object(env, model_name_ctx.c_str(), so);
  EXPECT_TRUE(SessionHasEp(session_object, kNvTensorRTRTXExecutionProvider));

  env.UnregisterExecutionProviderLibrary(kNvTensorRTRTXExecutionProvider);
Note: Google Test filter = *NvExecutionProviderTest*AutoEp_PreferGpu*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from NvExecutionProviderTest
[ RUN      ] NvExecutionProviderTest.AutoEp_PreferGpu
[       OK ] NvExecutionProviderTest.AutoEp_PreferGpu (1136 ms)
[----------] 1 test from NvExecutionProviderTest (1137 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1138 ms total)
[  PASSED  ] 1 test.

Motivation and Context

A recent feature allows ORT to automatically select an EP according to policies set by the user (e.g., prefer npu or prefer gpu). This PR allows NvTensorRTRTX EP to be potentially selected when the user sets the PREFER_GPU policy.

@adrianlizarraga adrianlizarraga requested a review from skottmckay May 8, 2025 16:28
@nieubank nieubank marked this pull request as ready for review May 8, 2025 16:59
adrianlizarraga
adrianlizarraga previously approved these changes May 8, 2025
@nieubank nieubank force-pushed the nieubank/nvrtx-support-autoep branch from 4edbfe6 to 4d84acb Compare May 21, 2025 17:59
@nieubank
Copy link
Author

@adrianlizarraga - I've updated the PR on the latest main with the change we took for Windows ML to fix multi-device path. When you get a chance, please review and let me know if there is anything else that needs to be done to unblock merging. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants