CASMCMS-9201 - Fix orphaned artifacts in S3. #155
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jsl-hpe
approved these changes
Jan 29, 2025
dlaine-hpe
force-pushed
the
CASMCMS-9201
branch
from
d000b37 to
fb1d5d1
Compare
mharding-hpe
approved these changes
Jan 29, 2025
jsollom-hpe
reviewed
Jan 29, 2025
| S3_STS_ENDPOINT = os.getenv('S3_STS_ENDPOINT') | ||
| S3_STS_ACCESS_KEY = os.getenv('S3_STS_ACCESS_KEY') | ||
| S3_STS_SECRET_KEY = os.getenv('S3_STS_SECRET_KEY') | ||
| S3_STS_SSL_VALIDATE = \ |
There was a problem hiding this comment.
You're hashing all of these potential false values to False, but you're not doing the same for, presumably, the true values. Why not? You're just taking whatever is in that environment variable.
| # use a multi-part copy operation. This is automatically handled by the Object.copy | ||
| # function. The multi-part copy requires greater levels of permissions than the single | ||
| # transfer copy. It will currenly not work if the owner of the object is something | ||
| # other than IMS. When an object is copied into S3 via 'cray artifacts create ...' it |
There was a problem hiding this comment.
Suggested change
| # other than IMS. When an object is copied into S3 via 'cray artifacts create ...' it | |
| # other than IMS. When an object is copied into S3 via 'cray artifacts create ...', it |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary and Scope
The 'soft delete' of IMS objects used the S3 'copy_from' command. This command does the copy in a single atomic operation but is limited to artifacts less than 5Gb in size. When the rootfs files got bigger than 5Gb, the copy into the 'deleted objects' location would fail, leaving them behind. Then when the 'deleted image' was deleted, these artifacts were not removed from S3.
The fix is to use the S3 'copy' command which uses a multi-part copy if the object is too larger for the single operation copy. There is an issue with S3/boto3 where the multi-part copy fails if the artifact is owned by a different user than the 'IMS' user. I had to put code in to handle the copy differently if the artifact is owned by 'STS' which is the case if 'cray artifacts create' is used to load the artifact into S3.
Issues and Related PRs
Testing
Tested on:
SurturTest description:
The new service was upgraded via helm chart and verified that the delete / undelete / hard delete worked as expected with all size artifacts and the 'STS' owner.
Risks and Mitigations
This should be fairly low risk.
Pull Request Checklist