add workflow to upload conda packages to legate channel #173

jameslamb · 2024-10-28T18:47:38Z

Contributes to #101
Contributes to #115

Follow-up to #171 (comment)

adds a workflow for publishing conda packages to the legate channel
switches dev versions to YY.MM.{patch}.dev{n}, instead of YY.MM.{patch}dev{n}, to match what legate and cunumeric are doing
- e.g. look at https://anaconda.org/legate/legate/files?channel=experimental ... you'll see versions like 24.09.00.dev329, not 24.09.00dev329
cleans up a bit more CI configuration as a result of split up PR and nightly/release CI #169

Notes for Reviewers

I've left comments inline describing specific changes in more detail.

jameslamb · 2024-10-28T21:16:29Z

🎉 🎉 🎉 just pushed conda packages to the experimental label on the legate channel!

https://anaconda.org/legate/legate-boost/files

This will now work (try it for yourself)

conda create --name test-legate-boost \
  --dry-run \
  --override-channels \
  -c legate/label/experimental \
  -c legate \
  -c conda-forge \
  -c nvidia \
    legate-boost=24.09.*  \
    python=3.11

This at least shows that as of this branch and moment in time, we have all the pieces in place to start publishing conda packages from this repo to the legate channel. I'll get this PR updated and ready to review 😁

jameslamb · 2024-10-30T15:51:45Z

.github/workflows/build.yaml

+    needs:
+      - conda-python-build
+    uses: ./.github/workflows/conda-upload-packages.yaml
+    secrets: inherit


Adding this here, with these triggers:

legate-boost/.github/workflows/build.yaml

Lines 7 to 17 in 166c98d

on:

# run on pushes to certain branches

push:

branches:

- "main"

- "release/v[0-9][0-9].[0-9][0-9].[0-9][0-9]"

# run on pushes of new release tags

tags:

- v[0-9][0-9].[0-9][0-9].[0-9][0-9]

# run by clicking buttons in the GitHub Actions UI

workflow_dispatch:

Means that new legate-boost packages will automatically be published to the legate channel in the following situations.

Scenario 1: New packages pushed to the experimental label, with version like {YY}.{MM}.{patch}.dev{n}

new commit pushed to main

new commit pushed to any branch named release/v{YY}.{MM}.{patch}

new tag pushed which does not exactly match the pattern v[0-9][0-9].[0-9][0-9].[0-9][0-9]

someone manually clicked the "run workflow" button in GitHub Actions UI

Scenario 2: New packages pushed to the main label, with version like {YY}.{MM}.{patch} (i.e. a stable release)

new tag pushed exactly matching the pattern v[0-9][0-9].[0-9][0-9].[0-9][0-9]

This sounds reasonable to me. The only downside would be that release branch nightlies are probably not used much and use disk-space/build time.

The only downside would be that release branch nightlies are probably not used much and use disk-space/build time.

The release/ branches should only be used for hotfixes, in the way I've proposed setting this up:

legate-boost/contributing.md

Line 233 in 166c98d

### Hotfixes

I'm thinking of the case where @danielfrg or someone reports a bug, and we merge a fix and want to say "ok we have a release candidate of like 24.09.01.dev3, can you test it?"

It'd be useful to have a nightly package for that testing, I think, so we can build confidence in that fix before officially releasing a new version and so the tester doesn't have to build from source or download a GitHub Actions artifact.

I don't know if we really need nightlies for all tags

Here, I'm thinking about the specific case where, say, we just cut stable release 24.09.00 and are ready to begin work on 24.12.00.dev{n}. Was thinking that in that case, we want a first build to run when development starts, so you immediately have nightlies with that version (and the corresponding versions of cunumeric / legate) to use.

See #171 (comment)

jameslamb · 2024-10-30T15:52:15Z

.github/workflows/github-actions.yml

@@ -23,8 +23,6 @@ jobs:
  # group together all jobs that must pass for a PR to be merged
  # (for use by branch protections)
  pr-builder:
-    # skip on builds from merges to main
-    if: github.ref != 'refs/heads/main'


Now that this workflow file is only triggered on pull requests, this condition is unnecessary.

jameslamb · 2024-10-30T15:54:12Z

conda/recipes/legate-boost/conda_build_config.yaml

-  - "=24.09.*,>=0.0.0.dev0"
+  # .dev319: https://github.com/nv-legate/legate.core.internal/pull/1401
+  # .dev329: https://github.com/nv-legate/legate.core.internal/issues/1409
+  - "=24.09.*,>=0.0.0.dev0,!=24.09.00.dev319,!=24.09.00.dev329"


I think we should try this approach of !=-ing past nightlies that cause incompatibilities here. This is preferable to == pinning to an older version, because it means that if the bugs we've reported upstream get fixed, then CI will just automatically start pulling in the new nightlies.

And if they aren't, then we'll get feedback from CI of the form "the bug we reported has still not been fixed".

I agree with the != approach. When we start using a newer version, we can convert the != to > (just to not pile up blocklisted versions)

convert the != to >

Yes definitely, agreed!

seberg

Thanks @jameslamb! I looked through and the logic seems right to me. I don't know if we really need nightlies for all tags and release branches (rather than just main).
But, there is also no harm, unless disk space is a problem and since versions must match very precisely, it could be useful.

seberg · 2024-10-30T16:53:37Z

conda/recipes/legate-boost/conda_build_config.yaml

-  - "=24.09.*,>=0.0.0.dev0"
+  # .dev319: https://github.com/nv-legate/legate.core.internal/pull/1401
+  # .dev329: https://github.com/nv-legate/legate.core.internal/issues/1409
+  - "=24.09.*,>=0.0.0.dev0,!=24.09.00.dev319,!=24.09.00.dev329"


I agree with the != approach. When we start using a newer version, we can convert the != to > (just to not pile up blocklisted versions)

seberg · 2024-10-30T16:55:43Z

.github/workflows/build.yaml

+    needs:
+      - conda-python-build
+    uses: ./.github/workflows/conda-upload-packages.yaml
+    secrets: inherit


This sounds reasonable to me. The only downside would be that release branch nightlies are probably not used much and use disk-space/build time.

jameslamb · 2024-10-30T18:49:39Z

I'm going to merge this on the strength of your approval @seberg , to test that it does what we want on merge-to-main, and to move on to the next step in this.

All of these things are easily reversible, so @RAMitchell please do let me know, when you see this, if there are any changes you'd like me to make.

jameslamb · 2024-10-30T19:14:04Z

Cool yeah this worked!

The merge triggered this build: https://github.com/rapidsai/legate-boost/actions/runs/11599749514/job/32298585013

That published 6 new packages to the legate channel (3 Python versions x [cpu, gpu]), with version 24.09.00.dev1

https://anaconda.org/legate/legate-boost/files

RAMitchell · 2024-10-31T08:17:22Z

Looks great thank you @jameslamb!

add workflow to upload conda packages to legate channel

9f3ec4b

jameslamb added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Oct 28, 2024

jameslamb added 5 commits October 28, 2024 14:00

fix pr-builder

ea15675

fix upload-to-anaconda script

2392376

temporarily skip import tests

0b4aaf8

use .dev, pin back to older legate and cunumeric nightlies

d2fafbc

try actually uploading packages

b63549b

jameslamb added 2 commits October 28, 2024 16:43

do not publish on every PR

f394171

more cleanup

1cce078

jameslamb mentioned this pull request Oct 28, 2024

start v24.09.00 development [skip ci] #171

Merged

jameslamb added 3 commits October 28, 2024 17:03

update install instructions

046e95a

skip latest nightlies

010bf51

skip more nightlies

e593ae6

jameslamb commented Oct 30, 2024

View reviewed changes

jameslamb changed the title ~~WIP: [DO NOT MERGE] add workflow to upload conda packages to legate channel~~ add workflow to upload conda packages to legate channel Oct 30, 2024

jameslamb marked this pull request as ready for review October 30, 2024 16:15

jameslamb requested review from seberg and RAMitchell October 30, 2024 16:15

seberg approved these changes Oct 30, 2024

View reviewed changes

jameslamb merged commit e998041 into rapidsai:main Oct 30, 2024
9 checks passed

jameslamb deleted the package-publishing branch October 30, 2024 18:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add workflow to upload conda packages to legate channel #173

add workflow to upload conda packages to legate channel #173

jameslamb commented Oct 28, 2024 •

edited

Loading

jameslamb commented Oct 28, 2024

jameslamb Oct 30, 2024

seberg Oct 30, 2024

jameslamb Oct 30, 2024

jameslamb Oct 30, 2024

jameslamb Oct 30, 2024

seberg Oct 30, 2024

jameslamb Oct 30, 2024

seberg left a comment

seberg Oct 30, 2024

seberg Oct 30, 2024

jameslamb commented Oct 30, 2024

jameslamb commented Oct 30, 2024

RAMitchell commented Oct 31, 2024

	on:
	# run on pushes to certain branches
	push:
	branches:
	- "main"
	- "release/v[0-9][0-9].[0-9][0-9].[0-9][0-9]"
	# run on pushes of new release tags
	tags:
	- v[0-9][0-9].[0-9][0-9].[0-9][0-9]
	# run by clicking buttons in the GitHub Actions UI
	workflow_dispatch:

add workflow to upload conda packages to legate channel #173

add workflow to upload conda packages to legate channel #173

Conversation

jameslamb commented Oct 28, 2024 • edited Loading

Notes for Reviewers

jameslamb commented Oct 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameslamb commented Oct 30, 2024

jameslamb commented Oct 30, 2024

RAMitchell commented Oct 31, 2024

jameslamb commented Oct 28, 2024 •

edited

Loading