[gpu] strict driver and cuda version assignment #1275

cjac · 2024-12-13T06:44:34Z

Resolves Issues

[gpu] versions installed by gpu/install_gpu_driver.sh do not match requested versions #1268

gpu/install_gpu_driver.sh

Driver version defaults to version in driver .run file if specified
CUDA version defaults to version in cuda .run file if specified
exclusively using .run file installation method for cuda and driver installation
- Installing non-open driver from .run file on rocky8
build nccl from source, since that is the only mechanism which supports all Dataproc OSs
wrap expensive functions in completion checks to reduce re-run time when testing manually
cache build results in GCS
waiting on apt lock when it exists
only install build dependencies if build is necessary
fix problem with ops agent not installing ; using venv
Installing gcc-12 on ubuntu22 to fix kernel driver FTBFS
mark task completion by creating a file rather than setting a variable
added functions to check and report secure-boot and os version details
if the correct metadata attributes are specified, pytorch will be installed, but not by default

gpu/manual-test-runner.sh

order commands correctly
point to origin rather than staging repo

gpu/test_gpu.py

clearer test skipping logic

cjac · 2024-12-13T06:44:44Z

/gcbrun

cjac · 2024-12-13T07:50:18Z

/gcbrun

cjac · 2024-12-14T01:16:00Z

/gcbrun

cjac · 2024-12-14T01:48:37Z

/gcbrun

cjac · 2024-12-14T03:27:41Z

Tests took 43 minutes to run this time. I think it's because the binary driver build cache was invalidated. Let's see if this one takes more like 14 minutes.

cjac · 2024-12-14T03:33:22Z

the intention is for this PR to resolve issue #1268

cjac · 2024-12-14T03:39:33Z

cloudbuild/Dockerfile

@@ -9,16 +9,23 @@ COPY --chown=ia-tests:ia-tests . /init-actions

 # Install Bazel:
 # https://docs.bazel.build/versions/master/install-ubuntu.html
-ENV bazel_kr_path=/usr/share/keyrings/bazel-keyring.gpg
+ENV bazel_kr_path=/usr/share/keyrings/bazel-keyring.gpg \
+    bazel_version=7.4.0 \


Pinning to 7.4.0 is necessary since the release of and default set to version 8

cjac · 2024-12-14T03:39:56Z

cloudbuild/Dockerfile

-RUN /usr/bin/curl https://bazel.build/bazel-release.pub.gpg | \
-    gpg --dearmor -o "${bazel_kr_path}"
-RUN echo "deb [arch=amd64 signed-by=${bazel_kr_path}] http://storage.googleapis.com/bazel-apt stable jdk1.8" | \
+RUN /usr/bin/curl -s https://bazel.build/bazel-release.pub.gpg | \


reduce noise in docker build with the -s (silent) option

cjac · 2024-12-14T03:41:03Z

cloudbuild/Dockerfile

    apt-get clean

+# Set bazel-${bazel_version} as the default bazel alternative in this container
+RUN update-alternatives --install /usr/bin/bazel bazel /usr/bin/bazel-${bazel_version} 1 && \


The file /usr/bin/bazel does not exist in the bazel-${version} package ; set up an alternative and use it

cjac · 2024-12-14T03:41:45Z

cloudbuild/create-key-pair.sh

@@ -0,0 +1,135 @@
+#!/bin/bash


this file copied from https://github.com/GoogleCloudDataproc/custom-images/tree/2024.11/examples/secure-boot

cjac · 2024-12-14T03:42:10Z

cloudbuild/presubmit.sh

@@ -70,6 +70,7 @@ determine_tests_to_run() {
    changed_dir="${changed_dir%%/*}/"
    # Run all tests if common directories modified
    if [[ ${changed_dir} =~ ^(integration_tests|util|cloudbuild)/$ ]]; then
+      continue


must remove this before merging

cjac · 2024-12-14T03:46:06Z

slightly longer than the 14 minutes I predicted, this run completed in 16:29

cjac · 2024-12-14T04:12:40Z

gpu/test_gpu.py

+    ("STANDARD", ["m", "w-0", "w-1"], GPU_T4, GPU_T4, "11.8", 'rocky', '2.0'),
+    ("STANDARD", ["m", "w-0", "w-1"], GPU_T4, GPU_T4, "12.4", 'rocky', '2.1'),
+    ("STANDARD", ["m", "w-0", "w-1"], GPU_T4, GPU_T4, "12.0", 'rocky', '2.2'),
+    ("KERBEROS", ["m", "w-0", "w-1"], GPU_T4, GPU_T4, "12.6", 'rocky', '2.2'),


We now exercise Kerberos in this test suite. True, it's only tested on 2.2-rocky9 with CUDA 12.6, but we could change any of the STANDARD definitions above to KERBEROS to improve that set.

cjac · 2024-12-14T04:15:30Z

/gcbrun

cjac · 2024-12-14T04:43:21Z

/gcbrun

cjac · 2024-12-14T07:17:10Z

/gcbrun

cjac · 2024-12-14T19:55:39Z

/gcbrun

cjac · 2024-12-14T20:27:45Z

/gcbrun

cjac · 2024-12-14T21:38:15Z

/gcbrun

cjac · 2024-12-14T21:57:14Z

/gcbrun

cjac · 2024-12-14T23:19:51Z

/gcbrun

cjac · 2024-12-15T01:32:16Z

/gcbrun

cjac · 2024-12-15T03:17:59Z

/gcbrun

cjac · 2024-12-15T03:52:04Z

/gcbrun

cjac · 2024-12-15T04:11:13Z

okay, taking a little break...

* changing layout of gcs package folder * install_pytorch function created and called when cuDNN is being installed

…tadata check within the function ; create new function harden_sshd_config and call it

…ber for cuda 11.1.1 url generation

…he platforms call it

cjac · 2025-01-22T08:31:11Z

/gcbrun

cjac · 2025-01-22T09:06:29Z

okay. it's a lot, but it all looks okay and has been thoroughly and repeatedly tested. Nota Bene that 'continue' in presubmit.sh ; I'll remove that once I receive an LGTM.

…ore closely

cjac · 2025-01-23T00:32:00Z

/gcbrun

… the templates/ directory from GoogleCloudDataproc#1282

cjac · 2025-01-23T01:21:20Z

/gcbrun

cjac · 2025-01-23T02:34:49Z

/gcbrun

cjac · 2025-01-23T04:57:52Z

/gcbrun

cjac · 2025-01-23T14:50:27Z

/gcbrun

cjac · 2025-01-23T15:56:23Z

/gcbrun

…able spark rapids on images >= 2.1

cjac · 2025-01-23T19:30:38Z

/gcbrun

cjac self-assigned this Dec 13, 2024

cjac requested review from Deependra-Patel, prince-cs and kuldeepkk-dev December 13, 2024 18:09

cjac commented Dec 14, 2024

View reviewed changes

cjac requested a review from jayadeep-jayaraman December 15, 2024 20:21

cjac added 11 commits January 22, 2025 00:19

* adding version 12.6 to the support matrix

0c420b7

* changing layout of gcs package folder * install_pytorch function created and called when cuDNN is being installed

incorrect version check removed

f69d071

only install pytorch if include-pytorch metadata set to true

73ffce5

since call to install_pytorch is protected by metadata check, skip me…

521df62

…tadata check within the function ; create new function harden_sshd_config and call it

increasing timeout and machine shape to reduce no-cache build time

c0b60b2

skip full test run due to edits to integration_tests directory

30c97c4

ubuntu18 does not know about kex-gss ; use correct driver version num…

84b1fb9

…ber for cuda 11.1.1 url generation

on rocky9 sshd service is called sshd instead of ssh as the rest of t…

11cbe95

…he platforms call it

kex-gss is new in debian11

56fe50c

all rocky call it sshd it seems

b1cd1d0

cudnn no longer available on debian10

ca94393

cjac force-pushed the gpu-20241212 branch from 906baea to ca94393 Compare January 22, 2025 08:19

cjac linked an issue Jan 22, 2025 that may be closed by this pull request

[gpu] versions installed by gpu/install_gpu_driver.sh do not match requested versions #1268

Open

compared with GoogleCloudDataproc#1282 ; this change matches parity m…

1d2166c

…ore closely

cjac added a commit to LLC-Technologies-Collier/initialization-actions that referenced this pull request Jan 23, 2025

reviewed GoogleCloudDataproc#1275 and brought closer to parity

576bbb6

slightly better variable declaration ordering ; it is better still in…

50142f6

… the templates/ directory from GoogleCloudDataproc#1282

install spark rapids

6363203

cache the results of nvidia-smi --query-gpu

dba00df

reduce development time

96a8d6d

exercising more CUDA variants ; testing whether tests fail on long runs

11f099c

try to reduce concurrent builds ; extend build time further ; only en…

8ae2c0a

…able spark rapids on images >= 2.1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[gpu] strict driver and cuda version assignment #1275

[gpu] strict driver and cuda version assignment #1275

cjac commented Dec 13, 2024 •

edited

Loading

cjac commented Dec 13, 2024

cjac commented Dec 13, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac Dec 14, 2024

cjac Dec 14, 2024

cjac Dec 14, 2024

cjac Dec 14, 2024

cjac Dec 14, 2024

cjac commented Dec 14, 2024

cjac Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 15, 2024

cjac commented Dec 15, 2024

cjac commented Dec 15, 2024

cjac commented Dec 15, 2024

cjac commented Jan 22, 2025

cjac commented Jan 22, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

[gpu] strict driver and cuda version assignment #1275

Are you sure you want to change the base?

[gpu] strict driver and cuda version assignment #1275

Conversation

cjac commented Dec 13, 2024 • edited Loading

cjac commented Dec 13, 2024

cjac commented Dec 13, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac Dec 14, 2024

Choose a reason for hiding this comment

cjac Dec 14, 2024

Choose a reason for hiding this comment

cjac Dec 14, 2024

Choose a reason for hiding this comment

cjac Dec 14, 2024

Choose a reason for hiding this comment

cjac Dec 14, 2024

Choose a reason for hiding this comment

cjac commented Dec 14, 2024

cjac Dec 14, 2024

Choose a reason for hiding this comment

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 14, 2024

cjac commented Dec 15, 2024

cjac commented Dec 15, 2024

cjac commented Dec 15, 2024

cjac commented Dec 15, 2024

cjac commented Jan 22, 2025

cjac commented Jan 22, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Jan 23, 2025

cjac commented Dec 13, 2024 •

edited

Loading