Stage local files to an OSDF origin #19
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is designed to be run at an HTCSS AP and stage a copy of the input file into a remote origin server.
When `stashcp` is run with an `osdf://` source without a third slash (i.e., `osdf://foo/bar` instead of `osdf:///foo/bar`), then the path fed into the namespace lookup lacked a leading `/` (i.e., `foo/bar`) resulting in a failed lookup.
- If MatchNamespaces fails, report it as a process-wide error. - If no process-wide errors are reported, use the last returned error as the error message.
If a prefix is not in a registered namespace, instead of effectively falling back to a default set of caches, return a failure. This is meant to provide reasonable error messages when there are typos in the URL instead of making it look like there's an issue with the caches. Here's an example output from `stashcp`: ``` stashcp osdf://typo/staging/bbockelm/foo /dev/null ERROR[2023-06-03T10:58:53-05:00] Failure downloading osdf://typo/staging/bbockelm/foo: Attempt htcondor#1: OSDF namespace not known for path /typo/staging/bbockelm/foo (100ms since start) ```
With this, end-to-end tests of the staging binary works for a simple job ad.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR creates a small command line tool to stage a local file to a unique name on a remote OSDF origin. It is meant to be idempotent; that is, it can be run many times and will only upload once as long as the file contents are unchanged.
If you replace the local filename with the generated OSDF URL, the user is guaranteed to get the current copy of the file from the OSDF (as the name is unique) and not a prior one
It is intended to be used with a AP-side job hook launched by the shadow prior to the job being activated. This way, the shadow will upload the file to the origin and the EP will download it via the OSDF.
An example use case would be a user who prefers to update their SIF file in-place -- they would always get the "correct as of time of job launch" version of their SIF.