Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement custom storage for orgs #2093

Open
wants to merge 66 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
cc464b6
Add back custom storage endpoints
tw4l Sep 20, 2024
eaacd4d
Flush out tests for setting custom storage
tw4l Sep 20, 2024
8893885
Fix test issue with bucket not existing for now
tw4l Sep 20, 2024
a530bc6
Add additional tests
tw4l Sep 23, 2024
9f58b64
Fix custom storage so it works as expected
tw4l Sep 24, 2024
abadda5
Actually unset custom replica storage before deleting
tw4l Sep 24, 2024
dbcd47e
Add TODO where custom storage deletion is failing
tw4l Sep 24, 2024
bc73f32
Fix check for whether storage label is in use
tw4l Sep 24, 2024
ad81cc5
Remove todo on endpoint that's fine
tw4l Sep 24, 2024
c03b675
Add todos re: tasks necessary to change storage
tw4l Sep 24, 2024
156bfe2
Check that no crawls are running before updating storage
tw4l Sep 25, 2024
512d6e7
Start adding post-storage update logic
tw4l Sep 25, 2024
0c5a1ee
WIP: Add background job to copy old s3 bucket to new
tw4l Sep 25, 2024
be6bb2a
WIP: Start adding logic to handle replica location updates
tw4l Sep 25, 2024
2f7c722
Add additional note
tw4l Sep 25, 2024
2e04a93
Fix argument
tw4l Sep 25, 2024
5bd3f66
Fix another argument
tw4l Sep 25, 2024
3b4995c
Fixups
tw4l Sep 25, 2024
ed2e2a3
Fix linting
tw4l Sep 25, 2024
5e6f698
More linting fixes
tw4l Sep 25, 2024
0e1dd89
Refactor, seperate storage and replicas updates
tw4l Sep 26, 2024
87f0751
More refactoring
tw4l Sep 26, 2024
bac456b
Make post-update task methods private
tw4l Sep 26, 2024
a63b08e
Check if any bg jobs running before changing storage
tw4l Sep 26, 2024
d24b90a
Check bg job finished as well
tw4l Sep 26, 2024
4d85f26
Fixups
tw4l Sep 26, 2024
9ad4fe6
Storage update improvements
tw4l Sep 26, 2024
8cbf28f
Fixup
tw4l Sep 26, 2024
fe51f48
Remove TODO
tw4l Sep 26, 2024
c132bd0
Remove another todo
tw4l Sep 26, 2024
201c0c6
More fixups
tw4l Sep 26, 2024
4bdca83
Add provider to s3storage for rclone
tw4l Sep 26, 2024
1bf224f
Fix typo
tw4l Sep 26, 2024
97aa1c5
Make API endpoints that change storage superuser-only for now
tw4l Sep 30, 2024
0989c66
Add typing for init_storages_api, import Callable
tw4l Sep 30, 2024
fa9161d
Add missing User import
tw4l Sep 30, 2024
a62144a
Fix StorageOps in operator main
tw4l Oct 1, 2024
1282fb8
Always use oid prefix in s3 storage
tw4l Oct 1, 2024
96e10eb
Post-rebase fixups and remove create bucket fallback
tw4l Oct 10, 2024
8ac9182
Create extra test buckets in CI
tw4l Oct 15, 2024
9fedf85
Add test for non-verified custom storage
tw4l Oct 15, 2024
f779507
Refactor to move updates to FastAPI background tasks
tw4l Oct 15, 2024
df1cb41
Include default replicas in /storage response if no org replicas
tw4l Oct 15, 2024
82eb46c
Fix unsetting of presigned URLs
tw4l Oct 16, 2024
5db34ae
Add --progress flag to rclone copy command
tw4l Oct 16, 2024
164b53b
Increase ttl seconds after finished for testing on dev
tw4l Oct 17, 2024
b19edcc
Ensure there are no double slashes between bucket name and oid
tw4l Oct 17, 2024
ff5fc2c
Increase memory limit/request for copy job to 500Mi
tw4l Oct 17, 2024
51f2404
Reduce copy job ttlSecondsAfterFinished to 60
tw4l Oct 17, 2024
f145ce4
Add storage tag to API endpoints
tw4l Oct 17, 2024
ce002d4
Add flags to rclone to reduce memory usage, set limit to 350Mi
tw4l Oct 17, 2024
aa3f571
Fix positional operator in storage ref update
tw4l Oct 17, 2024
315d23b
One more positional operator fix
tw4l Oct 17, 2024
01cc836
Update docstrings and comments
tw4l Oct 17, 2024
61d4e21
Make all-storages response valid JSON with response model
tw4l Oct 17, 2024
761e52c
Add admin docs for storage
tw4l Oct 17, 2024
3a52832
Fix API endpoint path in docs example
tw4l Oct 17, 2024
c1fffb0
Docs typo fix
tw4l Oct 17, 2024
de605f3
Add provider field note
tw4l Oct 17, 2024
4f31c41
Docs language cleanup
tw4l Oct 17, 2024
42c1c6b
Check /all-storages in backend tests
tw4l Oct 17, 2024
131b8a2
Add API endpoint for background job progress
tw4l Oct 18, 2024
d8486c7
Fix linting
tw4l Oct 18, 2024
74c8fd6
Format post-rebase with Black
tw4l Dec 3, 2024
a7d20e9
Format with Black
tw4l Jan 24, 2025
8fbc995
Fix linting
tw4l Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Increase memory limit/request for copy job to 500Mi
tw4l committed Jan 24, 2025
commit ff5fc2c7a772db223d613b6a7c24cf9525e9527e
4 changes: 2 additions & 2 deletions chart/app-templates/copy_job.yaml
Original file line number Diff line number Diff line change
@@ -89,8 +89,8 @@ spec:
command: ["rclone", "-vv", "--progress", "copy", "--checksum", "prev:{{ prev_bucket }}{{ oid }}", "new:{{ new_bucket }}{{ oid }}"]
resources:
limits:
memory: "200Mi"
memory: "500Mi"

requests:
memory: "200Mi"
memory: "500Mi"
cpu: "50m"