Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize replay file deletion #779

Closed
tsa96 opened this issue Jul 5, 2023 · 3 comments
Closed

Optimize replay file deletion #779

tsa96 opened this issue Jul 5, 2023 · 3 comments
Labels
Blocked: Needs more info This is lacking essential information like steps to recreate or a video showing reported behavior. For: Backend This is something to do for the backend (server folder) of the website. Priority: Medium Something that should be tried to be done in this release, but not as important as High priority. Size: Medium Something that may take a few days or so to implement. Type: Enhancement Something that builds on top of what already exists

Comments

@tsa96
Copy link
Member

tsa96 commented Jul 5, 2023

We'll be deleting a lot of replay files from S3, when a user PBs, and when maps are updated (changing state during map review, patching a map to fix a catastrophic bug). In the latter case we'd be reaaaaally hammering S3/R2 if we do individual deletes.

Also, we're going to be adding a status flag to each run in the future, likely PENDING_VALIDATION, APPROVED. Can easily add another, PENDING_DELETION. When a run is deleted, we just update the service to PENDING_DELETION.

Then, have a new service that uses NestJS's @Cron decorator from the scheduler that periodically, within an interactive transaction gets all runs marked as PENDING_DELETION, creates a S3 delete multiple call, and assuming that's successful, actually deletes all those runs from DB, interactive transaction ends.

@tsa96 tsa96 added Type: Enhancement Something that builds on top of what already exists Priority: Medium Something that should be tried to be done in this release, but not as important as High priority. Size: Medium Something that may take a few days or so to implement. For: Backend This is something to do for the backend (server folder) of the website. labels Jul 5, 2023
@tsa96 tsa96 changed the title Optimize replayfile deletion Optimize replay file deletion Jul 5, 2023
@tsa96 tsa96 added the Blocked: Needs more info This is lacking essential information like steps to recreate or a video showing reported behavior. label Jul 31, 2024
@tsa96
Copy link
Member Author

tsa96 commented Jul 31, 2024

Blocking for now; with ongoing replay work it's possible the size of replays end up decreasing to a point where we don't need to delete

@Gocnak
Copy link
Member

Gocnak commented Jul 31, 2024

Deleting in R2 is completely free: https://developers.cloudflare.com/r2/pricing/#free-operations

@tsa96
Copy link
Member Author

tsa96 commented Jul 31, 2024

Oh I hadn't realised. Closing in that case, only case I'm concerned about multi-deletes will be when doing whole leaderboard deletions #916, can tackle that there.

@tsa96 tsa96 closed this as not planned Won't fix, can't repro, duplicate, stale Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocked: Needs more info This is lacking essential information like steps to recreate or a video showing reported behavior. For: Backend This is something to do for the backend (server folder) of the website. Priority: Medium Something that should be tried to be done in this release, but not as important as High priority. Size: Medium Something that may take a few days or so to implement. Type: Enhancement Something that builds on top of what already exists
Projects
None yet
Development

No branches or pull requests

2 participants