Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch to rclone to upload artifacts #25

Merged
merged 2 commits into from
Jun 3, 2024

Conversation

flokli
Copy link
Contributor

@flokli flokli commented May 24, 2024

Requires a Jenkins with tiiuae/ghaf-infra#160 and tiiuae/ghaf-infra#161 applied.

Replaces the archiveArtifacts commands with rclone sync commands, uploading artifacts to a WebDAV endpoint (backed by Azure Blob Storage on the other side).

Links to the artifact downloads is provided by editing the Build description, as can be seen here.

ghaf-build-pipeline.groovy Outdated Show resolved Hide resolved
@flokli
Copy link
Contributor Author

flokli commented May 31, 2024

This needs to be extended to also support the other pipelines that appeared in the meantime.

@flokli
Copy link
Contributor Author

flokli commented May 31, 2024

Okay, ghaf-pre-merge-pipeline.groovy does not make use of the artifacts functionality, and ghaf-nightly-parallel.groovy has a lot of these (36!)

I think there should be some sort of deduplication/simplification, it was already mentioned in #24.

As the two approaches are not mutually exclusive, I'd prefer leaving refactoring up to such future PR, and have ghaf-nightly-parallel.groovy upload to local disk until it's refactored (cc @mkaapu).

This stops using archiveArtifacts, and instead shells out to `rclone` to
upload artifacts to the webdav endpoint provided via unix socket on
/run/rclone-jenkins-artifacts.sock.

The two wildcards `ghaf/result-*/**` and `ghaf/result-aarch64*/**` are
overlapping, at least for the way rclone does do globbing, so are merged
together.

Signed-off-by: Florian Klink <[email protected]>
@flokli
Copy link
Contributor Author

flokli commented May 31, 2024

I replaced the two individual rclone invocations with a single one, using ghaf/ as the source, and --include statements to do the globbing (as well as -L to follow the result- symlinks into the store).

We don't set JENKINS_URL, so we cannot use `${env.JENKINS_URL}` here,
but use an absolute path, which also works.

Signed-off-by: Florian Klink <[email protected]>
@henrirosten henrirosten requested a review from a team May 31, 2024 12:03
@flokli
Copy link
Contributor Author

flokli commented May 31, 2024

This can be merged once all Jenkins instances using the script (that we care about) have the changes from tiiuae/ghaf-infra#161 deployed.

I cannot merge here anyways, so please hit the button once done :-)

@henrirosten henrirosten merged commit eb8ad9a into tiiuae:main Jun 3, 2024
1 check passed
@flokli flokli deleted the flokli-rclone branch June 3, 2024 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants