Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect cuda-nvcc version #102

Open
ymtoo opened this issue Aug 4, 2023 · 3 comments
Open

Incorrect cuda-nvcc version #102

ymtoo opened this issue Aug 4, 2023 · 3 comments

Comments

@ymtoo
Copy link

ymtoo commented Aug 4, 2023

To install cuda-toolkit 11.7.1:

julia> CondaPkg.add("cuda-toolkit"; version="11.7.1", channel = "nvidia/label/cuda-11.7.1")

(test-condapkg) pkg> conda st
CondaPkg Status /home/ymtoo/Projects/tmp/test-condapkg/CondaPkg.toml
Environment
  /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env
Packages
  cuda-toolkit v11.7.1 (11.7.1, channel=nvidia/label/cuda-11.7.1)

shell> .CondaPkg/env/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0
The installed packages: 
$ conda list
# packages in environment at /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
binutils                  2.40                 hdd6e379_0    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
binutils_linux-64         2.40                 hbdbef99_1    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-compiler                1.6.0                hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cuda-cccl                 12.0.90              ha770c72_1    conda-forge
cuda-cccl-impl            2.0.1                ha770c72_0    conda-forge
cuda-cccl_linux-64        12.0.90              ha770c72_1    conda-forge
cuda-command-line-tools   11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-compiler             12.0.0               h9bef630_0    conda-forge
cuda-cudart               12.0.107             h59595ed_6    conda-forge
cuda-cudart-dev           12.0.107             h59595ed_6    conda-forge
cuda-cudart-dev_linux-64  12.0.107             h59595ed_6    conda-forge
cuda-cudart-static        12.0.107             h59595ed_6    conda-forge
cuda-cudart-static_linux-64 12.0.107             h59595ed_6    conda-forge
cuda-cudart_linux-64      12.0.107             h59595ed_6    conda-forge
cuda-cuobjdump            12.0.76              h59595ed_0    conda-forge
cuda-cupti                12.0.90              h59595ed_1    conda-forge
cuda-cuxxfilt             12.0.76              h59595ed_0    conda-forge
cuda-documentation        11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-driver-dev           12.0.107             h59595ed_6    conda-forge
cuda-driver-dev_linux-64  12.0.107             h59595ed_6    conda-forge
cuda-gdb                  12.0.90              hd47b8d6_0    conda-forge
cuda-libraries            12.0.0               ha770c72_1    conda-forge
cuda-libraries-dev        12.0.0               ha770c72_1    conda-forge
cuda-memcheck             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nsight               12.0.78              ha770c72_0    conda-forge
cuda-nsight-compute       11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-nvcc                 12.0.76             hba56722_10    conda-forge
cuda-nvcc-dev_linux-64    12.0.76              ha770c72_0    conda-forge
cuda-nvcc-impl            12.0.76              h59595ed_0    conda-forge
cuda-nvcc-tools           12.0.76              h59595ed_0    conda-forge
cuda-nvcc_linux-64        12.0.76             h8a487aa_10    conda-forge
cuda-nvdisasm             12.0.76              h59595ed_0    conda-forge
cuda-nvml-dev             12.0.76              hcb278e6_0    conda-forge
cuda-nvprof               12.0.90              h59595ed_0    conda-forge
cuda-nvprune              12.0.76              h59595ed_0    conda-forge
cuda-nvrtc                12.0.76              h59595ed_1    conda-forge
cuda-nvrtc-dev            12.0.76              h59595ed_1    conda-forge
cuda-nvtx                 12.0.76              hcb278e6_0    conda-forge
cuda-nvvp                 12.0.90              h59595ed_0    conda-forge
cuda-opencl               12.0.76              h59595ed_0    conda-forge
cuda-opencl-dev           12.0.76              ha770c72_0    conda-forge
cuda-profiler-api         12.0.76              ha770c72_0    conda-forge
cuda-sanitizer-api        12.0.90              h59595ed_0    conda-forge
cuda-toolkit              11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-tools                11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-version              12.0                 hffde075_2    conda-forge
cuda-visual-tools         11.7.1                        0    nvidia/label/cuda-11.7.1
cxx-compiler              1.6.0                h00ab1b0_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
gcc                       12.3.0               h8d2909c_1    conda-forge
gcc_impl_linux-64         12.3.0               he2b93b0_0    conda-forge
gcc_linux-64              12.3.0               h76fc315_1    conda-forge
gds-tools                 1.5.0.59             hcb278e6_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gxx                       12.3.0               h8d2909c_1    conda-forge
gxx_impl_linux-64         12.3.0               he2b93b0_0    conda-forge
gxx_linux-64              12.3.0               h8a814eb_1    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.19.3               h08a2579_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libcublas                 12.0.1.189           hcb278e6_2    conda-forge
libcublas-dev             12.0.1.189           hcb278e6_2    conda-forge
libcufft                  11.0.0.21            hcb278e6_1    conda-forge
libcufft-dev              11.0.0.21            hcb278e6_1    conda-forge
libcufile                 1.5.0.59             hcb278e6_0    conda-forge
libcufile-dev             1.5.0.59             hcb278e6_0    conda-forge
libcurand                 10.3.1.50            hcb278e6_0    conda-forge
libcurand-dev             10.3.1.50            hcb278e6_0    conda-forge
libcusolver               11.4.2.57            hcb278e6_1    conda-forge
libcusolver-dev           11.4.2.57            hcb278e6_1    conda-forge
libcusparse               12.0.0.76            hcb278e6_1    conda-forge
libcusparse-dev           12.0.0.76            hcb278e6_1    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-devel_linux-64     12.3.0               h8bca6fd_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libglib                   2.76.4               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libnpp                    12.0.0.30            h59595ed_0    conda-forge
libnpp-dev                12.0.0.30            h59595ed_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libnvjitlink              12.0.76              hcb278e6_1    conda-forge
libnvjitlink-dev          12.0.76              hcb278e6_1    conda-forge
libnvjpeg                 12.0.0.28            hcb278e6_0    conda-forge
libnvjpeg-dev             12.0.0.28            ha770c72_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsanitizer              12.3.0               h0f45ef3_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libstdcxx-devel_linux-64  12.3.0               h8bca6fd_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxkbcommon              1.5.0                h5d7e998_3    conda-forge
libxkbfile                1.1.0                h166bdaf_1    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nsight-compute            2022.4.0.15          h1166182_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
ocl-icd                   2.3.1                h7f98852_0    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
wayland                   1.22.0               h8c25dac_1    conda-forge
xcb-util                  0.4.0                hd590300_1    conda-forge
xcb-util-image            0.4.0                h8ee46fc_1    conda-forge
xcb-util-keysyms          0.4.0                h8ee46fc_1    conda-forge
xcb-util-renderutil       0.3.9                hd590300_1    conda-forge
xcb-util-wm               0.4.1                h8ee46fc_1    conda-forge
xkeyboard-config          2.39                 hd590300_0    conda-forge
xorg-compositeproto       0.4.2             h7f98852_1001    conda-forge
xorg-damageproto          1.2.1             h7f98852_1002    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxcomposite        0.4.6                h0b41bf4_1    conda-forge
xorg-libxdamage           1.1.5                h7f98852_1    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrandr            1.5.2                h7f98852_1    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-libxtst              1.2.3             h7f98852_1002    conda-forge
xorg-randrproto           1.5.0             h7f98852_1001    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-util-macros          1.19.3               h7f98852_0    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

By adding channels = ["anaconda"] to the CondaPkg.toml, the correct cuda-nvcc version is installed.

julia> conda up

(test-condapkg) pkg> conda st
CondaPkg Status /home/ymtoo/Projects/tmp/test-condapkg/CondaPkg.toml
Environment
  /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env
Packages
  cuda-toolkit v11.7.1 (11.7.1, channel=nvidia/label/cuda-11.7.1)
Channels
  anaconda

shell> .CondaPkg/env/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0
The installed packages: 
$ conda list
# packages in environment at /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env:
#
# Name                    Version                   Build  Channel
cuda-cccl                 11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-command-line-tools   11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-compiler             11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-cudart               11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-cudart-dev           11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-cuobjdump            11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-cupti                11.7.101                      0    nvidia/label/cuda-11.7.1
cuda-cuxxfilt             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-documentation        11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-driver-dev           11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-gdb                  11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-libraries            11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-libraries-dev        11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-memcheck             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nsight               11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nsight-compute       11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-nvcc                 11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-nvdisasm             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvml-dev             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvprof               11.7.101                      0    nvidia/label/cuda-11.7.1
cuda-nvprune              11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvrtc                11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-nvrtc-dev            11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-nvtx                 11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvvp                 11.7.101                      0    nvidia/label/cuda-11.7.1
cuda-sanitizer-api        11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-toolkit              11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-tools                11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-visual-tools         11.7.1                        0    nvidia/label/cuda-11.7.1
gds-tools                 1.3.1.18                      0    nvidia/label/cuda-11.7.1
libcublas                 11.10.3.66                    0    nvidia/label/cuda-11.7.1
libcublas-dev             11.10.3.66                    0    nvidia/label/cuda-11.7.1
libcufft                  10.7.2.91                     0    nvidia/label/cuda-11.7.1
libcufft-dev              10.7.2.91                     0    nvidia/label/cuda-11.7.1
libcufile                 1.3.1.18                      0    nvidia/label/cuda-11.7.1
libcufile-dev             1.3.1.18                      0    nvidia/label/cuda-11.7.1
libcurand                 10.2.10.91                    0    nvidia/label/cuda-11.7.1
libcurand-dev             10.2.10.91                    0    nvidia/label/cuda-11.7.1
libcusolver               11.4.0.1                      0    nvidia/label/cuda-11.7.1
libcusolver-dev           11.4.0.1                      0    nvidia/label/cuda-11.7.1
libcusparse               11.7.4.91                     0    nvidia/label/cuda-11.7.1
libcusparse-dev           11.7.4.91                     0    nvidia/label/cuda-11.7.1
libnpp                    11.7.4.75                     0    nvidia/label/cuda-11.7.1
libnpp-dev                11.7.4.75                     0    nvidia/label/cuda-11.7.1
libnvjpeg                 11.8.0.2                      0    nvidia/label/cuda-11.7.1
libnvjpeg-dev             11.8.0.2                      0    nvidia/label/cuda-11.7.1
nsight-compute            2022.2.1.3                    0    nvidia/label/cuda-11.7.1

Julia and the package version:

julia> versioninfo()
Julia Version 1.9.2
Commit e4ee485e909 (2023-07-05 09:39 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 8 virtual cores

(test-condapkg) pkg> st
Status `~/Projects/tmp/test-condapkg/Project.toml`
  [992eb4ea] CondaPkg v0.2.18

(test-condapkg) pkg> st --manifest
Status `~/Projects/tmp/test-condapkg/Manifest.toml`
  [992eb4ea] CondaPkg v0.2.18
  [692b3bcd] JLLWrappers v1.4.1
  [0f8b85d8] JSON3 v1.13.2
  [0b3b1443] MicroMamba v0.1.14
  [69de0a69] Parsers v2.7.2
  [fa939f87] Pidfile v1.3.0
  [aea7be01] PrecompileTools v1.1.2
  [21216c6a] Preferences v1.4.0
  [6c6a2e73] Scratch v1.2.0
  [856f2bd8] StructTypes v1.10.0
  [f8abcde7] micromamba_jll v1.4.9+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.3
  [76f85450] LibGit2
  [8f399da3] Libdl
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.9.2
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [deac9b47] LibCURL_jll v7.84.0+0
  [29816b5a] LibSSH2_jll v1.10.2+0
  [c8ffd9c3] MbedTLS_jll v2.28.2+0
  [14a3606d] MozillaCACerts_jll v2022.10.11
  [83775a58] Zlib_jll v1.2.13+0
  [8e850ede] nghttp2_jll v1.48.0+0
  [3f19e933] p7zip_jll v17.4.0+0
@cjdoris
Copy link
Collaborator

cjdoris commented Aug 5, 2023

What about channels = ["nvidia"] or channels = ["nvidia/label/cuda-11.7.1"]?

@cjdoris
Copy link
Collaborator

cjdoris commented Aug 5, 2023

The underlying issue is that CondaPkg passes -c conda-forge if you don't specify any channels, and so it's finding a bunch of compatible packages from the conda-forge channel with higher version numbers, and therefore installing those.

Currently CondaPkg specifies --no-channel-priority meaning that all channels are given equal priority. A future CondaPkg could specify a priority (e.g. put conda-forge at the bottom) which could mean that nvidia has higher priority than conda-forge.

@ymtoo
Copy link
Author

ymtoo commented Aug 6, 2023

channels = ["nvidia/label/cuda-11.7.1"] yields the correct cuda-nvcc version. For the following CondaPkg.toml

channels = ["nvidia/label/cuda-11.7.1", "conda-forge"]

[deps.cuda-toolkit]
channel = "nvidia/label/cuda-11.7.1"
version = "11.7.1"

conda up installs the wrong cuda-nvcc. This prevents the installation of other packages from conda-forge.

To clarify, CondaPkg installs the correct cuda-toolkit version from "nvidia/label/cuda-11.7.1" but it installs the dependencies, e.g., cuda-nvcc, with higher version numbers from "conda-forge". Am I right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cjdoris @ymtoo