Code

Implement custom storage for orgs#2093

Code

Open

tw4l wants to merge 66 commits intomainwebrecorder/browsertrix:mainfrom issue-578-custom-storagewebrecorder/browsertrix:issue-578-custom-storageCopy head branch name to clipboard

+938-92

Commits

Commits on Jan 24, 2025

• Add back custom storage endpoints

  tw4l

  committed
  cc464b6

  Copy full SHA for cc464b6

• Flush out tests for setting custom storage

  tw4l

  committed
  eaacd4d

  Copy full SHA for eaacd4d

• Fix test issue with bucket not existing for now

  tw4l

  committed
  8893885

  Copy full SHA for 8893885

• Add additional tests

  tw4l

  committed
  a530bc6

  Copy full SHA for a530bc6

• Fix custom storage so it works as expected

  Show description for 9f58b64
  tw4l

  committed
  9f58b64

  Copy full SHA for 9f58b64

• Actually unset custom replica storage before deleting

  tw4l

  committed
  abadda5

  Copy full SHA for abadda5

• Add TODO where custom storage deletion is failing

  tw4l

  committed
  dbcd47e

  Copy full SHA for dbcd47e

• Fix check for whether storage label is in use

  tw4l

  committed
  bc73f32

  Copy full SHA for bc73f32

• Remove todo on endpoint that's fine

  tw4l

  committed
  ad81cc5

  Copy full SHA for ad81cc5

• Add todos re: tasks necessary to change storage

  tw4l

  committed
  c03b675

  Copy full SHA for c03b675

• Check that no crawls are running before updating storage

  tw4l

  committed
  156bfe2

  Copy full SHA for 156bfe2

• Start adding post-storage update logic

  tw4l

  committed
  512d6e7

  Copy full SHA for 512d6e7

• WIP: Add background job to copy old s3 bucket to new

  tw4l

  committed
  0c5a1ee

  Copy full SHA for 0c5a1ee

• WIP: Start adding logic to handle replica location updates

  tw4l

  committed
  be6bb2a

  Copy full SHA for be6bb2a

• Add additional note

  tw4l

  committed
  2f7c722

  Copy full SHA for 2f7c722

• Fix argument

  tw4l

  committed
  2e04a93

  Copy full SHA for 2e04a93

• Fix another argument

  tw4l

  committed
  5bd3f66

  Copy full SHA for 5bd3f66

• Fixups

  tw4l

  committed
  3b4995c

  Copy full SHA for 3b4995c

• Fix linting

  tw4l

  committed
  ed2e2a3

  Copy full SHA for ed2e2a3

• More linting fixes

  tw4l

  committed
  5e6f698

  Copy full SHA for 5e6f698

• Refactor, seperate storage and replicas updates

  tw4l

  committed
  0e1dd89

  Copy full SHA for 0e1dd89

• More refactoring

  tw4l

  committed
  87f0751

  Copy full SHA for 87f0751

• Make post-update task methods private

  tw4l

  committed
  bac456b

  Copy full SHA for bac456b

• Check if any bg jobs running before changing storage

  tw4l

  committed
  a63b08e

  Copy full SHA for a63b08e

• Check bg job finished as well

  tw4l

  committed
  d24b90a

  Copy full SHA for d24b90a

• Fixups

  tw4l

  committed
  4d85f26

  Copy full SHA for 4d85f26

• Storage update improvements

  tw4l

  committed
  9ad4fe6

  Copy full SHA for 9ad4fe6

• Fixup

  tw4l

  committed
  8cbf28f

  Copy full SHA for 8cbf28f

• Remove TODO

  tw4l

  committed
  fe51f48

  Copy full SHA for fe51f48

• Remove another todo

  tw4l

  committed
  c132bd0

  Copy full SHA for c132bd0

• More fixups

  tw4l

  committed
  201c0c6

  Copy full SHA for 201c0c6

• Add provider to s3storage for rclone

  tw4l

  committed
  4bdca83

  Copy full SHA for 4bdca83

• Fix typo

  tw4l

  committed
  1bf224f

  Copy full SHA for 1bf224f

• Make API endpoints that change storage superuser-only for now

  tw4l

  committed
  97aa1c5

  Copy full SHA for 97aa1c5

• Add typing for init_storages_api, import Callable

  tw4l

  committed
  0989c66

  Copy full SHA for 0989c66

• Add missing User import

  tw4l

  committed
  fa9161d

  Copy full SHA for fa9161d

• Fix StorageOps in operator main

  tw4l

  committed
  a62144a

  Copy full SHA for a62144a

• Always use oid prefix in s3 storage

  Show description for 1282fb8
  tw4l

  committed
  1282fb8

  Copy full SHA for 1282fb8

• Post-rebase fixups and remove create bucket fallback

  Show description for 96e10eb
  tw4l

  committed
  96e10eb

  Copy full SHA for 96e10eb

• Create extra test buckets in CI

  tw4l

  committed
  8ac9182

  Copy full SHA for 8ac9182

• Add test for non-verified custom storage

  tw4l

  committed
  9fedf85

  Copy full SHA for 9fedf85

• Refactor to move updates to FastAPI background tasks

  tw4l

  committed
  f779507

  Copy full SHA for f779507

• Include default replicas in /storage response if no org replicas

  tw4l

  committed
  df1cb41

  Copy full SHA for df1cb41

• Fix unsetting of presigned URLs

  tw4l

  committed
  82eb46c

  Copy full SHA for 82eb46c

• Add --progress flag to rclone copy command

  tw4l

  committed
  5db34ae

  Copy full SHA for 5db34ae

• Increase ttl seconds after finished for testing on dev

  tw4l

  committed
  164b53b

  Copy full SHA for 164b53b

• Ensure there are no double slashes between bucket name and oid

  tw4l

  committed
  b19edcc

  Copy full SHA for b19edcc

• Increase memory limit/request for copy job to 500Mi

  tw4l

  committed
  ff5fc2c

  Copy full SHA for ff5fc2c

• Reduce copy job ttlSecondsAfterFinished to 60

  tw4l

  committed
  51f2404

  Copy full SHA for 51f2404

• Add storage tag to API endpoints

  tw4l

  committed
  f145ce4

  Copy full SHA for f145ce4

• Add flags to rclone to reduce memory usage, set limit to 350Mi

  tw4l

  committed
  ce002d4

  Copy full SHA for ce002d4

• Fix positional operator in storage ref update

  tw4l

  committed
  aa3f571

  Copy full SHA for aa3f571

• One more positional operator fix

  tw4l

  committed
  315d23b

  Copy full SHA for 315d23b

• Update docstrings and comments

  tw4l

  committed
  01cc836

  Copy full SHA for 01cc836

• Make all-storages response valid JSON with response model

  tw4l

  committed
  61d4e21

  Copy full SHA for 61d4e21

• Add admin docs for storage

  tw4l

  committed
  761e52c

  Copy full SHA for 761e52c

• Fix API endpoint path in docs example

  tw4l

  committed
  3a52832

  Copy full SHA for 3a52832

• Docs typo fix

  tw4l

  committed
  c1fffb0

  Copy full SHA for c1fffb0

• Add provider field note

  tw4l

  committed
  de605f3

  Copy full SHA for de605f3

• Docs language cleanup

  tw4l

  committed
  4f31c41

  Copy full SHA for 4f31c41

• Check /all-storages in backend tests

  tw4l

  committed
  42c1c6b

  Copy full SHA for 42c1c6b

• Add API endpoint for background job progress

  tw4l

  committed
  131b8a2

  Copy full SHA for 131b8a2

• Fix linting

  tw4l

  committed
  d8486c7

  Copy full SHA for d8486c7

• Format post-rebase with Black

  tw4l

  committed
  74c8fd6

  Copy full SHA for 74c8fd6

• Format with Black

  tw4l

  committed
  a7d20e9

  Copy full SHA for a7d20e9

• Fix linting

  tw4l

  committed
  8fbc995

  Copy full SHA for 8fbc995