Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement custom storage for orgs #2093

Open
wants to merge 66 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
cc464b6
Add back custom storage endpoints
tw4l Sep 20, 2024
eaacd4d
Flush out tests for setting custom storage
tw4l Sep 20, 2024
8893885
Fix test issue with bucket not existing for now
tw4l Sep 20, 2024
a530bc6
Add additional tests
tw4l Sep 23, 2024
9f58b64
Fix custom storage so it works as expected
tw4l Sep 24, 2024
abadda5
Actually unset custom replica storage before deleting
tw4l Sep 24, 2024
dbcd47e
Add TODO where custom storage deletion is failing
tw4l Sep 24, 2024
bc73f32
Fix check for whether storage label is in use
tw4l Sep 24, 2024
ad81cc5
Remove todo on endpoint that's fine
tw4l Sep 24, 2024
c03b675
Add todos re: tasks necessary to change storage
tw4l Sep 24, 2024
156bfe2
Check that no crawls are running before updating storage
tw4l Sep 25, 2024
512d6e7
Start adding post-storage update logic
tw4l Sep 25, 2024
0c5a1ee
WIP: Add background job to copy old s3 bucket to new
tw4l Sep 25, 2024
be6bb2a
WIP: Start adding logic to handle replica location updates
tw4l Sep 25, 2024
2f7c722
Add additional note
tw4l Sep 25, 2024
2e04a93
Fix argument
tw4l Sep 25, 2024
5bd3f66
Fix another argument
tw4l Sep 25, 2024
3b4995c
Fixups
tw4l Sep 25, 2024
ed2e2a3
Fix linting
tw4l Sep 25, 2024
5e6f698
More linting fixes
tw4l Sep 25, 2024
0e1dd89
Refactor, seperate storage and replicas updates
tw4l Sep 26, 2024
87f0751
More refactoring
tw4l Sep 26, 2024
bac456b
Make post-update task methods private
tw4l Sep 26, 2024
a63b08e
Check if any bg jobs running before changing storage
tw4l Sep 26, 2024
d24b90a
Check bg job finished as well
tw4l Sep 26, 2024
4d85f26
Fixups
tw4l Sep 26, 2024
9ad4fe6
Storage update improvements
tw4l Sep 26, 2024
8cbf28f
Fixup
tw4l Sep 26, 2024
fe51f48
Remove TODO
tw4l Sep 26, 2024
c132bd0
Remove another todo
tw4l Sep 26, 2024
201c0c6
More fixups
tw4l Sep 26, 2024
4bdca83
Add provider to s3storage for rclone
tw4l Sep 26, 2024
1bf224f
Fix typo
tw4l Sep 26, 2024
97aa1c5
Make API endpoints that change storage superuser-only for now
tw4l Sep 30, 2024
0989c66
Add typing for init_storages_api, import Callable
tw4l Sep 30, 2024
fa9161d
Add missing User import
tw4l Sep 30, 2024
a62144a
Fix StorageOps in operator main
tw4l Oct 1, 2024
1282fb8
Always use oid prefix in s3 storage
tw4l Oct 1, 2024
96e10eb
Post-rebase fixups and remove create bucket fallback
tw4l Oct 10, 2024
8ac9182
Create extra test buckets in CI
tw4l Oct 15, 2024
9fedf85
Add test for non-verified custom storage
tw4l Oct 15, 2024
f779507
Refactor to move updates to FastAPI background tasks
tw4l Oct 15, 2024
df1cb41
Include default replicas in /storage response if no org replicas
tw4l Oct 15, 2024
82eb46c
Fix unsetting of presigned URLs
tw4l Oct 16, 2024
5db34ae
Add --progress flag to rclone copy command
tw4l Oct 16, 2024
164b53b
Increase ttl seconds after finished for testing on dev
tw4l Oct 17, 2024
b19edcc
Ensure there are no double slashes between bucket name and oid
tw4l Oct 17, 2024
ff5fc2c
Increase memory limit/request for copy job to 500Mi
tw4l Oct 17, 2024
51f2404
Reduce copy job ttlSecondsAfterFinished to 60
tw4l Oct 17, 2024
f145ce4
Add storage tag to API endpoints
tw4l Oct 17, 2024
ce002d4
Add flags to rclone to reduce memory usage, set limit to 350Mi
tw4l Oct 17, 2024
aa3f571
Fix positional operator in storage ref update
tw4l Oct 17, 2024
315d23b
One more positional operator fix
tw4l Oct 17, 2024
01cc836
Update docstrings and comments
tw4l Oct 17, 2024
61d4e21
Make all-storages response valid JSON with response model
tw4l Oct 17, 2024
761e52c
Add admin docs for storage
tw4l Oct 17, 2024
3a52832
Fix API endpoint path in docs example
tw4l Oct 17, 2024
c1fffb0
Docs typo fix
tw4l Oct 17, 2024
de605f3
Add provider field note
tw4l Oct 17, 2024
4f31c41
Docs language cleanup
tw4l Oct 17, 2024
42c1c6b
Check /all-storages in backend tests
tw4l Oct 17, 2024
131b8a2
Add API endpoint for background job progress
tw4l Oct 18, 2024
d8486c7
Fix linting
tw4l Oct 18, 2024
74c8fd6
Format post-rebase with Black
tw4l Dec 3, 2024
a7d20e9
Format with Black
tw4l Jan 24, 2025
8fbc995
Fix linting
tw4l Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Post-rebase fixups and remove create bucket fallback
Creating a bucket in the verification stage for adding custom
storages if it didn't exist was useful for testing but is an
anti-pattern for production, so we remove it here.
tw4l committed Jan 24, 2025
commit 96e10eb17d2b38259b142e61105cdec310cda621
8 changes: 5 additions & 3 deletions backend/btrixcloud/background_jobs.py
Original file line number Diff line number Diff line change
@@ -296,9 +296,11 @@ async def create_delete_org_job(
self,
org: Organization,
existing_job_id: Optional[str] = None,
) -> Optional[str]:
) -> str:
"""Create background job to delete org and its data"""

job_type = BgJobType.DELETE_ORG.value

try:
job_id = await self.crawl_manager.run_delete_org_job(
oid=str(org.id),
@@ -333,7 +335,7 @@ async def create_delete_org_job(
except Exception as exc:
# pylint: disable=raise-missing-from
print(f"warning: delete org job could not be started: {exc}")
return None
return ""

async def create_recalculate_org_stats_job(
self,
@@ -800,7 +802,7 @@ async def get_background_job(
"""Retrieve information for background job"""
return await ops.get_background_job(job_id, org.id)

@app.get("/orgs/all/jobs/{job_id}", response_model=SuccessResponse, tags=["jobs"])
@app.get("/orgs/all/jobs/{job_id}", response_model=AnyJob, tags=["jobs"])
async def get_background_job_all_orgs(job_id: str, user: User = Depends(user_dep)):
"""Get background job from any org"""
if not user.is_superuser:
7 changes: 5 additions & 2 deletions backend/btrixcloud/crawlmanager.py
Original file line number Diff line number Diff line change
@@ -125,7 +125,9 @@ async def run_delete_org_job(
if existing_job_id:
job_id = existing_job_id
else:
job_id = f"delete-org-{oid}-{secrets.token_hex(5)}"
job_id_prefix = f"delete-org-{oid}"
# ensure name is <=63 characters
job_id = f"{job_id_prefix[:52]}-{secrets.token_hex(5)}"

return await self._run_bg_job_with_ops_classes(
oid, job_id, job_type=BgJobType.DELETE_ORG.value
@@ -187,6 +189,8 @@ async def _run_bg_job_with_ops_classes(

await self.create_from_yaml(data, namespace=DEFAULT_NAMESPACE)

return job_id

async def run_copy_bucket_job(
self,
oid: str,
@@ -220,7 +224,6 @@ async def run_copy_bucket_job(
"new_secret_name": new_storage.get_storage_secret_name(oid),
"new_endpoint": new_endpoint,
"new_bucket": new_bucket,
"BgJobType": BgJobType,
}

data = self.templates.env.get_template("copy_job.yaml").render(params)
18 changes: 5 additions & 13 deletions backend/btrixcloud/storages.py
Original file line number Diff line number Diff line change
@@ -269,7 +269,7 @@ async def update_storage_ref(

await self.org_ops.update_storage_refs(org)

# TODO: Run in asyncio task or background job?
# Run in background jobs (1 to copy files, 1 for db updates)
await self._run_post_storage_update_tasks(
prev_storage_ref,
storage_ref,
@@ -336,7 +336,7 @@ async def update_storage_replica_refs(

await self.org_ops.update_storage_refs(org, replicas=True)

# TODO: Run in asyncio task or background job?
# Run in background job? or just kick off background jobs?
await self._run_post_storage_replica_update_tasks(
prev_storage_replicas, replicas, org
)
@@ -363,7 +363,7 @@ async def _run_post_storage_replica_update_tasks(
await self.org_ops.add_file_replica_storage_refs(org, replica_storage)

# Delete no-longer-used replica storage refs from files
# TODO: Determine if we want to delete files from the buckets as well
# Determine if we want to delete files from the buckets as well
for replica_storage in prev_replica_refs:
if replica_storage not in new_replica_refs:
await self.org_ops.remove_file_replica_storage_refs(
@@ -418,16 +418,8 @@ async def verify_storage_upload(self, storage: S3Storage, filename: str) -> None
key += filename
data = b""

try:
resp = await client.put_object(Bucket=bucket, Key=key, Body=data)
assert resp["ResponseMetadata"]["HTTPStatusCode"] == 200
except Exception:
# create bucket if it doesn't yet exist and then try again
resp = await client.create_bucket(Bucket=bucket)
assert resp["ResponseMetadata"]["HTTPStatusCode"] == 200

resp = await client.put_object(Bucket=bucket, Key=key, Body=data)
assert resp["ResponseMetadata"]["HTTPStatusCode"] == 200
resp = await client.put_object(Bucket=bucket, Key=key, Body=data)
assert resp["ResponseMetadata"]["HTTPStatusCode"] == 200

def resolve_internal_access_path(self, path):
"""Resolve relative path for internal access to minio bucket"""