Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: refactor using data platform workflows #665

Merged
merged 12 commits into from
Jan 30, 2025
Merged

ci: refactor using data platform workflows #665

merged 12 commits into from
Jan 30, 2025

Conversation

NohaIhab
Copy link
Contributor

This PR merges KF-6684-refactor-ci-with-build branch into main . The changes implemented in the dev branch are:

  1. Refactors the charmcraft.yaml files to use st124 shorthand notation and python plugin instead of charm plugin for compatibility with charmcraftcache
  2. Replaces the integrate.yaml and on_pull_request.yaml workflows into ci.yaml, a workflow that does the following:
  • Gets the paths for the charms in the multi-charm repo
  • Builds each charm in a separate runner with charmcraftcache
  • Passes the built charm artifacts to test jobs
  • Runs the individual integration tests for each charm
  • Runs the bundle integration tests
  • Publishes the built charms to <track>/edge/pr-xx channel
  1. Similarly, replaces on_push.yaml with release.yaml workflow, which:
  • Re-uses ci.yaml to run the builds, checks, and tests
  • Published the built charms to <track>/edge only if the tests pass
  1. Renames the old release.yaml to promote.yaml
  2. Refactors the integration test .py files to deploy from the pre-built charms, and keeps the option for local builds

DnPlas and others added 12 commits January 6, 2025 18:34
@DnPlas
ci: use get_charms_build_with_cache.yaml workflow (#642)
7b8ce0d 
* ci: use get_charms_build_with_cache.yaml workflow

This workflow enables the usage of canonical/charmed-kubeflow-workflows/.github/workflows/get_charms_build_with_cache.yaml
for this CI, meaning that the integration tests and publish jobs are now able to download charm artefacts and use them
consistently throught the various operations of this CI.

Fixes #641
@DnPlas
ci, tests: pass charm artefacts to deploy and test charms (#640)
9c543d8 
* ci, tests: pass charm artefacts to deploy and test charms

This commit enables the "--charm-path" option to pass .charm artefacts
to be deployed and tested at an individual level.
This change enables the option to pass pre-built charms to the tests
to avoid building them on the same test. It also ensures compatibility
with the build_charm.py reusable workflow (from canonical/data-platform-workflows).

Fixes: #639
@NohaIhab
Refactor charmcraft.yaml files to use python plugin and st124 not…
9df3478 
…ation (#648)

* use python plugin and add licenses to charms
* use charmcraft channel 3.x/edge
* use branch for quality checks workflow with charmcraft 3.x/edge
* update charmcraft channel to latest/candidate
@DnPlas @NohaIhab
ci: remove wrapper around get charm paths and build with cache workfl…
1a58b13 
…ows (#651)

* ci: remove wrapper around get charm paths and build with cache workflows

This commit removes the extra abstraction of the get-charm-paths action and
build_charm.yaml reusable workflow to avoid potential incompatibilities
and sync issues with the data platforms workflows.

* update workflows for dpw v29

* fix: use correct artifact name

* nit: address comments for consistency with data repos
---------

Co-authored-by: NohaIhab <[email protected]>
@NohaIhab
Use stage instead of prime in charmcraft files part (#658)
9985853
@NohaIhab @DnPlas
ci, tests: pass charm artefacts to deploy and test charms (#640) (#657)
3f1efc3 
Use single (cached) build for integration tests & release

Fix hardcoded path for `platforms` syntax

remove workflow dispatch from release.yaml not needed

add description for outputs

rename get charm paths job and channel output

pin quality checks back to main due to merging canonical/charmed-kubeflow-workflows#95

Use stage instead of prime in charmcraft files part (#658)

Co-authored-by: Daniela Plascencia <[email protected]>
@NohaIhab @orfeas-k
ci: use cache built charms in bundle tests (#662)
6fefbc4 
* init: pass cached charms to v1 bundle tests

* add build dependency for bundle tests

* add step to download packed charms

* fix: wait-for units instead of applications (#659)

* remove no_build option and refactor v1 tests

* update v2 bundle tests

* implement helper function to update context + comments

* fix: update logs action (#663)

Ref canonical/kubeflow-ci#152

* fix: modify ci.yaml release job to need get-charm-paths-channel

* update tests README

---------

Co-authored-by: Orfeas Kourkakis <[email protected]>
@NohaIhab
remove charmcraft pin to latest/candidate (#664)
4b93d27
@NohaIhab
Merge branch 'main' into KF-6684-refactor-ci-with-build
52d5783
@NohaIhab @carlcsaposs-canonical
fix: remove destination_channel from release workflow
b642cab 
Co-authored-by: Carl Csaposs <[email protected]>
@NohaIhab
add Copyright in license files
c65db64
@NohaIhab
add Copyright in license files
4fb8ec7
@NohaIhab NohaIhab mentioned this pull request Jan 29, 2025
Closed
@NohaIhab
Copy link
Contributor Author

the CI failed intermittently due to canonical/data-platform-workflows#22, re-running Get charm paths job and its dependents resolved it.

orfeas-k
orfeas-k approved these changes Jan 30, 2025
View reviewed changes
Copy link
Contributor

@orfeas-k orfeas-k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @NohaIhab and everyone involved in this effort. It's so great to see this refactor. Can't wait to use it!

@NohaIhab NohaIhab merged commit e59744f into main Jan 30, 2025
182 checks passed
@NohaIhab NohaIhab deleted the KF-6684-refactor-ci-with-build branch January 30, 2025 09:34
This was referenced Jan 30, 2025
Closed
Merged
carlcsaposs-canonical pushed a commit to canonical/charmcraftcache-hub that referenced this pull request Jan 30, 2025
@NohaIhab
Update kfp charms branch to main (#291)
829ac62 
Pointing to main now that
canonical/kfp-operators#665 is merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@orfeas-k orfeas-k orfeas-k approved these changes

Assignees
No one assigned
Labels
Libraries: Out of sync
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@NohaIhab @orfeas-k @DnPlas