Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CI jobs #10

Merged
merged 80 commits into from
Nov 17, 2023
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
76d4540
add initial check
brandon-b-miller Oct 31, 2023
71f31dc
move pr.yaml
brandon-b-miller Oct 31, 2023
cadc4bc
add empty check_style.sh
brandon-b-miller Oct 31, 2023
3f46b26
basic check_style.sh that does nothing
brandon-b-miller Oct 31, 2023
2dbd68f
add +x
brandon-b-miller Oct 31, 2023
be8c202
try adding a wheel-build job
brandon-b-miller Oct 31, 2023
695a133
fix spacing
brandon-b-miller Oct 31, 2023
cea436a
add runs-on
brandon-b-miller Oct 31, 2023
bc7116f
install ctk
brandon-b-miller Oct 31, 2023
351fd7e
add -q for now
brandon-b-miller Oct 31, 2023
b18b582
switch to apt-get
brandon-b-miller Oct 31, 2023
c3ebe4c
minor fixes
brandon-b-miller Oct 31, 2023
cb21a8e
update package name
brandon-b-miller Oct 31, 2023
7e67976
add install options
brandon-b-miller Oct 31, 2023
487341d
expanding build_wheel.sh
brandon-b-miller Oct 31, 2023
e58540a
add tests and logging
brandon-b-miller Oct 31, 2023
d40ce2e
more test build steps
brandon-b-miller Oct 31, 2023
cefb8f6
fix path
brandon-b-miller Oct 31, 2023
29b319d
Apply suggestions from code review
brandon-b-miller Nov 2, 2023
b4ef9f6
fix style
brandon-b-miller Nov 2, 2023
a3fef15
add wheel-build to needs
brandon-b-miller Nov 2, 2023
5a36b01
apt-get -> yum due to switching to centos
brandon-b-miller Nov 2, 2023
37ce733
fix centos cuda install
brandon-b-miller Nov 2, 2023
cca7db5
set c++11
brandon-b-miller Nov 3, 2023
abeae67
remove testing phase for now, will test on downstream jobs instead later
brandon-b-miller Nov 3, 2023
8af9c15
try adding a matrix
brandon-b-miller Nov 3, 2023
510db4a
Update .github/workflows/pr.yaml
brandon-b-miller Nov 6, 2023
1335b56
install python via conda
brandon-b-miller Nov 6, 2023
cb57054
missed endquote
brandon-b-miller Nov 6, 2023
25bd2b7
try adding test steps in separate jobs
brandon-b-miller Nov 6, 2023
1669c92
fix syntax
brandon-b-miller Nov 6, 2023
42b0c5b
switch to * for now
brandon-b-miller Nov 6, 2023
40b8d91
formatting
brandon-b-miller Nov 6, 2023
f84ba0b
add missing job
brandon-b-miller Nov 6, 2023
946509c
try uploading a directory
brandon-b-miller Nov 6, 2023
a6e76c7
update paths
brandon-b-miller Nov 6, 2023
f59a41a
drop 3.12 for now
brandon-b-miller Nov 6, 2023
306c1b9
add matrix to download
brandon-b-miller Nov 6, 2023
7ea3af1
+x to test_wheel.sh
brandon-b-miller Nov 6, 2023
1b2ce4d
expand test_wheel.sh
brandon-b-miller Nov 6, 2023
02906bb
try reading artifact name
brandon-b-miller Nov 6, 2023
91ac502
a debugging commit
brandon-b-miller Nov 8, 2023
08842b1
update wheel install command
brandon-b-miller Nov 8, 2023
1dc9460
missing ..
brandon-b-miller Nov 8, 2023
a70ea01
fix more paths
brandon-b-miller Nov 8, 2023
a635037
build in verbose mode
brandon-b-miller Nov 8, 2023
8dcceef
cmake.verbose=True in pyproject.toml
brandon-b-miller Nov 8, 2023
5690d0a
require cuda 12.3
brandon-b-miller Nov 9, 2023
667691f
explicitly install gcc
brandon-b-miller Nov 9, 2023
e253487
switch containers, remove extra gcc install
brandon-b-miller Nov 14, 2023
b0b0410
endquote
brandon-b-miller Nov 14, 2023
e0f71aa
base image name on python version, dont install python in test container
brandon-b-miller Nov 14, 2023
ab57c23
adjust matrix
brandon-b-miller Nov 14, 2023
e2a2a3b
install pytest using pip
brandon-b-miller Nov 14, 2023
8c318ff
use gpu runners
brandon-b-miller Nov 14, 2023
8dfd9b9
install gcc again
brandon-b-miller Nov 14, 2023
d1eac13
enforce compiler versions in cmake
brandon-b-miller Nov 14, 2023
78566e0
try installing system compilers
brandon-b-miller Nov 15, 2023
effab51
set and unset e
brandon-b-miller Nov 15, 2023
b80ad0a
start expanding test matrix
brandon-b-miller Nov 15, 2023
ee7841d
do not set e for now
brandon-b-miller Nov 15, 2023
774b2c8
use SCCACHE_S3_NO_CREDENTIALS=1
brandon-b-miller Nov 15, 2023
7449fcb
use ci-wheel containers to get arm images
brandon-b-miller Nov 15, 2023
e11cc48
get python from the image instead of conda
brandon-b-miller Nov 15, 2023
7ef7ce7
use inputs.node_type
brandon-b-miller Nov 15, 2023
ce3bb98
build on cpu runners
brandon-b-miller Nov 15, 2023
6497681
Update .github/workflows/pr.yaml
brandon-b-miller Nov 15, 2023
f340a61
remove python version from path
brandon-b-miller Nov 15, 2023
4614362
separate testing into arm64 and amd64
brandon-b-miller Nov 15, 2023
6baf691
add to needs section
brandon-b-miller Nov 15, 2023
b4db71a
use citestwheel test images
brandon-b-miller Nov 15, 2023
ca82b03
skip bad test
brandon-b-miller Nov 15, 2023
5ea590e
Apply suggestions from code review
bdice Nov 15, 2023
110af93
Update pynvjitlink/tests/test_pynvjitlink.py
brandon-b-miller Nov 16, 2023
c9f57b4
build amd and arm in two separate jobs
brandon-b-miller Nov 16, 2023
0789176
don't test ubuntu 18.04 with arm
brandon-b-miller Nov 16, 2023
f016805
address reviews
brandon-b-miller Nov 16, 2023
1f647ba
fix cmake
brandon-b-miller Nov 16, 2023
78b0b41
try removing cache again
brandon-b-miller Nov 16, 2023
7e48e2b
Revert "try removing cache again"
brandon-b-miller Nov 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Copyright (c) 2023, NVIDIA CORPORATION.
name: pr

on:
push:
branches:
- "pull-request/[0-9]+"

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
pr-builder:
needs:
- checks
secrets: inherit
uses: rapidsai/shared-action-workflows/.github/workflows/[email protected]
checks:
secrets: inherit
uses: rapidsai/shared-action-workflows/.github/workflows/[email protected]
with:
enable_check_generated_files: false
wheel-build:
runs-on: ubuntu-latest
container:
image: "rapidsai/ci-conda:latest"
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Get PR Info
id: get-pr-info
uses: rapidsai/shared-actions/get-pr-info@main
- name: Run build_wheel.sh
run: ci/build_wheel.sh
28 changes: 28 additions & 0 deletions ci/build_wheel.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash
# Copyright (c) 2023, NVIDIA CORPORATION

# install CUDAToolKit version 12.2
apt-get update
apt-get -y install cuda-toolkit-12-2

rapids-logger "Check conda environment"

conda list

rapids-logger "Building wheel"

pip wheel .

rapids-logger "Installing wheel"

pip install pynvjitlink-0.1.0-cp310-cp310-manylinux_2_35_x86_64.whl
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to use an older CI image than what is pinned at rapidsai/ci-conda:latest because we want to use the oldest OS to ensure that wheels are compatible with an older manylinux tag. This should be manylinux_2_17 if at all possible. Try using a CentOS 7 image and see if that helps.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed that we're going to need a Python version matrix. This wheel only support Python 3.10. We need to ship pynvjitlink on Python 3.9, 3.10 for RAPIDS, and also 3.11 and 3.12 if we want to make external consumers from Numba happy. We don't have Python 3.11 / 3.12 Docker images yet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gmarkall Can you comment on the expected Python version support for this package?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping to follow SPEC 0, so at the present time 3.10-3.12 support would be my aim. If 3.9 support is also needed for RAPIDS then that seems OK. If we can't initially provide wheels for 3.11 and 3.12 then I think that's also OK.

My reasoning is that for non-RAPIDS use cases, it will always be possible for users to build pynvjitlink themselves (unlike with cubinlinker), and we're covering the RAPIDS deployment scenarios in supporting 3.9 and 3.10 from the start. If there's no short-term route to subsequently adding 3.11 and 3.12, I'd like to look into how to get to supporting them eventually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there aren't ci-conda centos images spanning the range of python versions we need. Should we be installing the right version of python ourselves within the container or is there something else I should be doing instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add newer Python CI images, or just use Python versions from conda. Perhaps it will suffice to have a single build job that iterates over all the Python versions, since the builds should be fast.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd lean towards having a separate job for each build, just so that if something goes wrong with just one of them one doesn't have to parse through the log to figure out which one.

Installing and using the correct python version via conda seems like it's a fairly easy, so I could investigate down that route. What do you mean by "add newer python CI images"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by "add newer python CI images"?

Today, our CI images only cover Python 3.9 and 3.10. https://github.com/rapidsai/ci-imgs/blob/bf7d540855e7e94695a54e40080c3acb8681764a/matrix.yaml#L8-L10

Copy link
Contributor

@bdice bdice Nov 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it will suffice to have a single build job that iterates over all the Python versions, since the builds should be fast.

I had in mind using a conda variant build like in ucxx, where the conda_build_config.yaml has a list of Python versions and conda build knows to build all of them. It can be quite convenient, especially since pynvjitlink may have a wider support matrix of Python versions than RAPIDS as a whole (and is already likely to be using a newer CUDA than we might be using in RAPIDS).

https://github.com/rapidsai/ucxx/blob/4464ca942d5ac71aabe4492e2a6d59da453f06dc/conda/recipes/ucxx/conda_build_config.yaml#L19-L21


rapids-logger "Building tests"

cd test_binary_generation && make

rapids-logger "Running Tests"

cd ..
conda install -y pytest
py.test pynvjitlink/tests
2 changes: 2 additions & 0 deletions ci/check_style.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/bash
# Copyright (c) 2023, NVIDIA CORPORATION