Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filemanager: lifecycle transitioning objects #605

Open
mmalenic opened this issue Oct 14, 2024 · 2 comments
Open

filemanager: lifecycle transitioning objects #605

mmalenic opened this issue Oct 14, 2024 · 2 comments
Assignees
Labels
feature New feature filemanager an issue relating to the filemanager

Comments

@mmalenic
Copy link
Member

The filemanager should support a mechanism to transition objects to an expired state or a different storage class. This involves accepting a transition request via the API or the EventBus for a set of objects, and then performing the transition after the specified time period.

I think this can be accomplished with object tagging, and S3 lifecycle rules. For example, the filemanager could tag objects with predefined values matching lifecycle rules. This should only require one tag per transition. Moving should also not be an issue, because the filemanager can remove the lifecycle tag when an object is moved, as it's tracking moves via the filemanager_id tag.

A limitation of this approach is that it requires additional tags for each transition, and there is a maximum of 10 tags per objects. It doesn't look like there is a way to pack tag values into a single key to support multiple transitions, because the transition rules expect exact values for tags.

As an alternative, the filemanager could control the lifecycle transitions using it's own database state and DeleteObject. This could be more complicated but would remove the tag requirement, and allow supporting transitions using more complex logic like wildcards.

@mmalenic mmalenic self-assigned this Oct 14, 2024
@mmalenic mmalenic added filemanager an issue relating to the filemanager feature New feature labels Oct 14, 2024
@mmalenic
Copy link
Member Author

It is possible to specify multiple transitions in a single lifecycle rule, so it should be possible to define rules that transition more than once based on a single tag. For example, a tag with the value standard_ia_30&glacier_90 could match a rule with a transition for standard_ia after 30 days and glacier after 90 days.

@mmalenic
Copy link
Member Author

mmalenic commented Oct 17, 2024

Tasks:

  • Add the FileStateChange schema to support transitioning objects via the event bus.
  • Implement the logic in file manager to tag objects and transition them.
  • Add the S3 lifecycle rules which do the transitions in the infrastructure repo: https://github.com/umccr/infrastructure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature filemanager an issue relating to the filemanager
Projects
None yet
Development

No branches or pull requests

1 participant