chore: Upgrade to CUDA 12.1 #2020

gs-olive · 2023-06-14T00:46:19Z

Description

Upgrade CUDA version across CI, dependencies, and build systems
Upgrade versions for Torch, TensorRT, and other dependencies in accordance with new CUDA version 12.1
Upgrade Torch nightly date to latest available

Fixes #2010

Type of change

Chore: version + dependency upgrades

Checklist:

[ x ] My code follows the style guidelines of this project (You can use the linters)
[ x ] I have performed a self-review of my own code
[ x ] I have commented my code, particularly in hard-to-understand areas and hacks
[ x ] I have made corresponding changes to the documentation
[ x ] I have added tests to verify my fix or my feature
[ x ] New and existing unit tests pass locally with my changes
[ x ] I have added the relevant labels to my PR in so that relevant reviewers are notified

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive · 2023-06-15T04:41:51Z

Seems to be failing CI due to a strange interaction between timm, torch, and tensorrt imports. This is a minimal Segfaulting example:

# Does not segfault if we remove the tensorrt import
import tensorrt

import torch
import timm

model = (timm.create_model("efficientnet_b0", pretrained=True).eval().to("cuda"))
input_ = torch.randn((1, 3, 224, 224)).to("cuda")

# Segfaults, with the current CI machine configuration
model(input_)

Note that the versions of libnvinfer are for CUDA 12.0, since there aren't 12.1 releases.

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

HolyWu · 2023-06-18T15:19:25Z

Seems to be failing CI due to a strange interaction between timm, torch, and tensorrt imports.

The segfault should be gone if you move import tensorrt below import torch, which means importing torch first and importing tensorrt later.

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive · 2023-06-28T22:24:42Z

.circleci/config.yml


  install-cudnn:
-    description: "Install CUDNN 8.8.0"
+    description: "Install CUDNN 8.8.1"


Based on feedback and discussion with @HolyWu and @narendasan, the CuDNN was only upgraded in the patch version to avoid issues with PyTorch and CuDNN 8.9, while preserving the libtorch.so version installed by Torch. A subsequent upgrade to the newest CuDNN is to come when PyTorch support is more robust.

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

narendasan

LGTM

- Upgrade CUDA version across CI, dependencies, and build systems - Upgrade versions for Torch, CuDNN, TensorRT, and other dependencies in accordance with new CUDA version 12.1 - Upgrade Torch nightly date to latest available

This reverts commit 5e5a6d9.

This reverts commit c9145a6. - Also upgrades to Jun 19 nightly and changes CI WORKSPACE

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive self-assigned this Jun 14, 2023

facebook-github-bot added cla signed fx labels Jun 14, 2023

github-actions bot added component: api [Python] Issues re: Python API component: build system Issues re: Build system component: fx labels Jun 14, 2023

github-actions bot requested a review from wushirong June 14, 2023 00:46

github-actions bot approved these changes Jun 14, 2023

View reviewed changes

gs-olive removed the request for review from wushirong June 14, 2023 00:47

gs-olive force-pushed the cuda_121_upgrade branch from 23038f2 to 59a2687 Compare June 14, 2023 01:03

github-actions bot approved these changes Jun 14, 2023

View reviewed changes

gs-olive force-pushed the cuda_121_upgrade branch from 59a2687 to 7338f9e Compare June 14, 2023 01:07

github-actions bot approved these changes Jun 14, 2023

View reviewed changes

gs-olive force-pushed the cuda_121_upgrade branch from 7338f9e to 5f546c5 Compare June 14, 2023 01:14

github-actions bot approved these changes Jun 14, 2023

View reviewed changes

gs-olive force-pushed the cuda_121_upgrade branch from 5f546c5 to f0f1df3 Compare June 14, 2023 16:28

github-actions bot added the component: tests Issues re: Tests label Jun 14, 2023

github-actions bot approved these changes Jun 14, 2023

View reviewed changes

github-actions bot approved these changes Jun 15, 2023

View reviewed changes

gs-olive force-pushed the cuda_121_upgrade branch from 51a737e to 592f97a Compare June 15, 2023 05:15

github-actions bot approved these changes Jun 15, 2023

View reviewed changes

gs-olive requested a review from narendasan June 27, 2023 20:21

github-actions bot approved these changes Jun 27, 2023

View reviewed changes

github-actions bot approved these changes Jun 28, 2023

View reviewed changes

gs-olive force-pushed the cuda_121_upgrade branch from ad0ef97 to 89fbb3e Compare June 28, 2023 20:59

github-actions bot approved these changes Jun 28, 2023

View reviewed changes

gs-olive changed the title ~~chore: Upgrade to CUDA 12.1 and CuDNN 8.9~~ chore: Upgrade to CUDA 12.1 Jun 28, 2023

gs-olive commented Jun 28, 2023

View reviewed changes

gs-olive force-pushed the cuda_121_upgrade branch from 89fbb3e to b230c86 Compare June 29, 2023 02:09

github-actions bot approved these changes Jun 29, 2023

View reviewed changes

gs-olive force-pushed the cuda_121_upgrade branch from b230c86 to a871d62 Compare June 29, 2023 02:12

github-actions bot approved these changes Jun 29, 2023

View reviewed changes

narendasan approved these changes Jun 30, 2023

View reviewed changes

gs-olive added 6 commits June 30, 2023 12:09

chore: Upgrade to CUDA 12.1 and CuDNN 8.9

e9e50ad

- Upgrade CUDA version across CI, dependencies, and build systems - Upgrade versions for Torch, CuDNN, TensorRT, and other dependencies in accordance with new CUDA version 12.1 - Upgrade Torch nightly date to latest available

Downgrade to CuDNN 8.9.0

27651e8

Revert "Downgrade to CuDNN 8.9.0"

9d33ee7

This reverts commit 5e5a6d9.

with_pypi_cudnn_experiment

0e5e62e

Revert "with_pypi_cudnn_experiment"

16f5d0d

This reverts commit c9145a6. - Also upgrades to Jun 19 nightly and changes CI WORKSPACE

Roll back to cuDNN 8.8

e6cc509

gs-olive force-pushed the cuda_121_upgrade branch from a871d62 to e6cc509 Compare June 30, 2023 19:09

github-actions bot approved these changes Jun 30, 2023

View reviewed changes

gs-olive merged commit 6ceaed8 into main Jul 3, 2023
4 checks passed

gs-olive deleted the cuda_121_upgrade branch July 3, 2023 19:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Upgrade to CUDA 12.1 #2020

chore: Upgrade to CUDA 12.1 #2020

gs-olive commented Jun 14, 2023 •

edited

Loading

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

gs-olive commented Jun 15, 2023 •

edited

Loading

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

HolyWu commented Jun 18, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

gs-olive Jun 28, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

narendasan left a comment

github-actions bot left a comment

github-actions bot left a comment

chore: Upgrade to CUDA 12.1 #2020

chore: Upgrade to CUDA 12.1 #2020

Conversation

gs-olive commented Jun 14, 2023 • edited Loading

Description

Type of change

Checklist:

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive commented Jun 15, 2023 • edited Loading

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

HolyWu commented Jun 18, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive Jun 28, 2023

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

narendasan left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive commented Jun 14, 2023 •

edited

Loading

gs-olive commented Jun 15, 2023 •

edited

Loading