jenkins: preparations to upload and serve artifacts #161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Disabling the setup wizard is done in #160, this is unrelated to this PR. |
|
Drafting until #160 is merged. |
Adds per-environment storage account and storage container for storing Jenkins artifacts. Signed-off-by: Florian Klink <[email protected]>
This adds two rclone services, one for uploads, one for browsing. Both listen on unix sockets. The first one is used by (and limited to) Jenkins to upload artifacts, the second one is exposed (over HTTPS) through caddy, exposing artifacts at the /artifacts/* subpath. Signed-off-by: Florian Klink <[email protected]>
rclone also doesn't support connecting to a webdav endpoint that's exposed via an HTTP socket, so apply that patch too. The feature also has been sent upstream, we link to the PR, but vendor a patch, as it doesn't apply cleanly on our rclone version. Signed-off-by: Florian Klink <[email protected]>
We'll shell out to our patched rclone from the pipeline definition. Signed-off-by: Florian Klink <[email protected]>
|
With #160 merged, I rebased this. |
|
Corresponding pipeline PR: tiiuae/ghaf-jenkins-pipeline#25 |
We don't want to cache directory listings for 5 minutes (the default), as that will mean artifacts won't be visible for way too long. Set to 5 secs, which will still give us some request deduplication, but not too much. We cannot apply this to the binary cache configs yet, as rclone seems to always want to fetch a listing, which is very slow for that. Signed-off-by: Florian Klink <[email protected]>
|
Reviewed and verified working by deploying to a private environment. I guess once this is merged, and all the Jenkins pipelines are changed to no longer use the |
henrirosten
approved these changes
May 31, 2024
|
Yeah, that's blocked on everything being switched over, and we might need to schedule some time to properly sync some existing state (if we want to keep the disk contents) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #160, will be rebased once merged (so ignore the first 2 commits).
This adds the necessary services running on the jenkins-controller machine to upload and serve artifacts from Azure storage.
$PATH.