Skip to content
Permalink

Comparing changes

This is a direct comparison between two commits made in this repository or its related repositories. View the default comparison for this range or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: webrecorder/browsertrix
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: c5e88e3c18b9d6fb49a0a2e99731bbd6013db7d6
Choose a base ref
..
head repository: webrecorder/browsertrix
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: a867411511394679b2de77e51665a393e82c83c1
Choose a head ref
Showing with 4,034 additions and 1,652 deletions.
  1. +17 −0 .github/ISSUE_TEMPLATE/localization-request.yml
  2. +5 −5 .github/PULL_REQUEST_TEMPLATE/ui_changes.md
  3. +11 −2 .github/workflows/docs-publish.yaml
  4. +1 −1 .github/workflows/k3d-nightly-ci.yaml
  5. +13 −10 README.md
  6. +1 −1 assets/browsertrix-lockup-color-dynamic.svg
  7. +62 −1 backend/btrixcloud/background_jobs.py
  8. +31 −20 backend/btrixcloud/crawlconfigs.py
  9. +37 −1 backend/btrixcloud/crawlmanager.py
  10. +2 −2 backend/btrixcloud/main.py
  11. +20 −97 backend/btrixcloud/main_bg.py
  12. +10 −65 backend/btrixcloud/main_op.py
  13. +22 −2 backend/btrixcloud/models.py
  14. +124 −0 backend/btrixcloud/ops.py
  15. +8 −4 backend/btrixcloud/orgs.py
  16. +1 −1 backend/btrixcloud/storages.py
  17. +1 −0 backend/test-requirements.txt
  18. +30 −0 backend/test/conftest.py
  19. +58 −60 backend/test/test_qa.py
  20. +6 −6 backend/test/test_run_crawl.py
  21. +19 −10 backend/test/test_uploads.py
  22. +32 −1 backend/test/test_z_delete_org.py
  23. +3 −0 backend/test_nightly/test_crawlconfig_crawl_stats.py
  24. +90 −6 backend/test_nightly/test_org_deletion.py
  25. +21 −8 backend/test_nightly/test_storage_quota.py
  26. +11 −0 chart/email-templates/failed_bg_job
  27. +1 −1 chart/values.yaml
  28. +1 −0 frontend/.yarnrc
  29. +2 −2 frontend/Dockerfile
  30. +15 −0 frontend/docs/copy-api-docs.sh
  31. +1 −1 frontend/docs/docs/deploy/index.md
  32. +34 −16 frontend/docs/docs/develop/docs.md
  33. +0 −41 frontend/docs/docs/develop/frontend-dev.md
  34. +57 −0 frontend/docs/docs/develop/localization.md
  35. +4 −3 frontend/docs/docs/index.md
  36. +49 −0 frontend/docs/docs/overrides/partials/actions.html
  37. +2 −0 frontend/docs/docs/overrides/partials/integrations/analytics/custom.html
  38. +4 −0 frontend/docs/docs/overrides/partials/integrations/analytics/plausible.html
  39. +1 −1 frontend/docs/docs/user-guide/signup.md
  40. +34 −25 frontend/docs/mkdocs.yml
  41. +6 −3 frontend/package.json
  42. +3 −0 frontend/playwright.config.ts
  43. +1,326 −0 frontend/src/__generated__/locales/es.ts
  44. +1 −1 frontend/src/assets/brand/browsertrix-lockup-color.svg
  45. +1 −1 frontend/src/assets/favicons/favicon.svg
  46. +3 −0 frontend/src/assets/icons/flask-fill.svg
  47. +2 −0 frontend/src/classes/BtrixElement.ts
  48. +6 −1 frontend/src/classes/TailwindElement.ts
  49. +1 −1 frontend/src/components/beta-badges.stylesheet.css
  50. +17 −10 frontend/src/components/beta-badges.ts
  51. +2 −2 frontend/src/components/detail-page-title.ts
  52. +3 −5 frontend/src/components/orgs-list.ts
  53. +2 −1 frontend/src/components/screencast.ts
  54. +10 −11 frontend/src/components/ui/config-details.ts
  55. +1 −2 frontend/src/components/ui/index.ts
  56. +0 −1 frontend/src/components/ui/input/index.ts
  57. +0 −55 frontend/src/components/ui/input/input.css
  58. +0 −120 frontend/src/components/ui/input/input.ts
  59. +5 −4 frontend/src/components/ui/language-select.ts
  60. +0 −74 frontend/src/components/ui/locale-picker.ts
  61. +3 −4 frontend/src/components/ui/markdown-editor.ts
  62. +2 −2 frontend/src/components/ui/navigation/navigation-button.ts
  63. +85 −0 frontend/src/components/ui/user-language-select.ts
  64. +47 −0 frontend/src/controllers/localize.ts
  65. +17 −11 frontend/src/features/accounts/sign-up-form.ts
  66. +9 −18 frontend/src/features/archived-items/archived-item-list.ts
  67. +4 −13 frontend/src/features/archived-items/crawl-list.ts
  68. +0 −2 frontend/src/features/archived-items/crawl-logs.ts
  69. +4 −3 frontend/src/features/archived-items/crawl-pending-exclusions.ts
  70. +10 −10 frontend/src/features/archived-items/crawl-queue.ts
  71. +6 −6 frontend/src/features/archived-items/crawl-status.ts
  72. +8 −11 frontend/src/features/browser-profiles/select-browser-profile.ts
  73. +11 −8 frontend/src/features/collections/collection-items-dialog.ts
  74. +7 −9 frontend/src/features/collections/collection-workflow-list.ts
  75. +1 −1 frontend/src/features/collections/collections-add.ts
  76. +14 −6 frontend/src/features/crawl-workflows/exclusion-editor.ts
  77. +27 −23 frontend/src/features/crawl-workflows/queue-exclusion-form.ts
  78. +78 −63 frontend/src/features/crawl-workflows/workflow-editor.ts
  79. +9 −25 frontend/src/features/crawl-workflows/workflow-list.ts
  80. +60 −30 frontend/src/features/org/org-status-banner.ts
  81. +0 −2 frontend/src/features/org/usage-history-table.ts
  82. +4 −4 frontend/src/features/qa/page-list/page-list.ts
  83. +4 −5 frontend/src/features/qa/qa-run-dropdown.ts
  84. +9 −2 frontend/src/index.test.ts
  85. +64 −36 frontend/src/index.ts
  86. +268 −179 frontend/src/pages/account-settings.ts
  87. +18 −20 frontend/src/pages/admin.ts
  88. +14 −11 frontend/src/pages/log-in.ts
  89. +7 −10 frontend/src/pages/org/archived-item-detail/archived-item-detail.ts
  90. +6 −9 frontend/src/pages/org/archived-item-detail/ui/qa.ts
  91. +5 −6 frontend/src/pages/org/archived-item-qa/archived-item-qa.ts
  92. +9 −8 frontend/src/pages/org/archived-item-qa/ui/resources.ts
  93. +2 −5 frontend/src/pages/org/browser-profiles-detail.ts
  94. +1 −4 frontend/src/pages/org/browser-profiles-list.ts
  95. +1 −1 frontend/src/pages/org/browser-profiles-new.ts
  96. +21 −6 frontend/src/pages/org/collection-detail.ts
  97. +15 −14 frontend/src/pages/org/collections-list.ts
  98. +12 −10 frontend/src/pages/org/dashboard.ts
  99. +68 −46 frontend/src/pages/org/settings/components/billing.ts
  100. +12 −0 frontend/src/pages/org/settings/settings.ts
  101. +64 −54 frontend/src/pages/org/workflow-detail.ts
  102. +51 −38 frontend/src/pages/org/workflows-list.ts
  103. +13 −10 frontend/src/pages/reset-password.ts
  104. +5 −1 frontend/src/replayWebPage.d.ts
  105. +1 −1 frontend/src/routes.ts
  106. +15 −1 frontend/src/theme.stylesheet.css
  107. +1 −0 frontend/src/types/billing.ts
  108. +19 −0 frontend/src/types/localization.ts
  109. +2 −0 frontend/src/types/user.ts
  110. +2 −0 frontend/src/utils/LiteElement.ts
  111. +2 −2 frontend/src/utils/crawler.ts
  112. +9 −10 frontend/src/utils/cron.ts
  113. +2 −2 frontend/src/utils/executionTimeFormatter.ts
  114. +4 −4 frontend/src/utils/form.ts
  115. +0 −68 frontend/src/utils/localization.ts
  116. +119 −0 frontend/src/utils/localize.test.ts
  117. +147 −0 frontend/src/utils/localize.ts
  118. +5 −1 frontend/src/utils/number.ts
  119. +10 −1 frontend/src/utils/pluralize.ts
  120. +6 −2 frontend/src/utils/slugify.ts
  121. +0 −2 frontend/src/utils/state.ts
  122. +3 −3 frontend/src/utils/workflow.ts
  123. +25 −40 frontend/tailwind.config.js
  124. +10 −0 frontend/web-test-runner.config.mjs
  125. +1 −1 frontend/webpack.config.js
  126. +139 −80 frontend/xliff/es.xlf
  127. +207 −23 frontend/yarn.lock
17 changes: 17 additions & 0 deletions .github/ISSUE_TEMPLATE/localization-request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: Localization Request
description: Request a new language or translation.
title: "[L10N]: "
labels: ["localization"]
body:
- type: textarea
attributes:
label: Language
description: Specify the language you'd like to add or translate. A list of currently supported languages can be found in our [Weblate project](https://hosted.weblate.org/engage/browsertrix/).
validations:
required: true
- type: textarea
attributes:
label: Context
description: Any background information that helps us understand the request.
validations:
required: true
10 changes: 5 additions & 5 deletions .github/PULL_REQUEST_TEMPLATE/ui_changes.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
Resolves #issue_number

<!-- Fixes #issue_number -->
## Changes

### Changes

### Manual testing

## Manual testing

1.

### Screenshots
## Screenshots

| Page | Image/video |
| ---- | ----------- |
| | |

<!-- ### Follow-ups -->
<!-- ## Follow-ups -->
13 changes: 11 additions & 2 deletions .github/workflows/docs-publish.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Publish Docs
on:
workflow_dispatch:

release:
types: [published]

@@ -21,4 +21,13 @@ jobs:
- name: Generate Helm Chart Index
run: python ./scripts/generate-helm-index.py > ./frontend/docs/docs/helm-repo/index.yaml

- run: cd frontend/docs; mkdocs gh-deploy --force
- name: Copy Docs Files
run: frontend/docs/copy-api-docs.sh
env:
DOCS_SOURCE_URL: https://app.browsertrix.com
ENABLE_ANALYTICS: true

- name: Build Docs
run: cd frontend/docs; mkdocs gh-deploy --force
env:
ENABLE_ANALYTICS: true
2 changes: 1 addition & 1 deletion .github/workflows/k3d-nightly-ci.yaml
Original file line number Diff line number Diff line change
@@ -79,7 +79,7 @@ jobs:
run: kubectl wait --for=condition=ready pod --all --timeout=240s

- name: Create Extra Test Buckets
run: kubectl exec -i deployment/local-minio -c minio mkdir /data/replica-0
run: kubectl exec -i deployment/local-minio -c minio -- mkdir /data/replica-0

- name: Run Tests
run: pytest -vv ./backend/test_nightly/test_*.py
23 changes: 13 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -6,38 +6,41 @@

&nbsp;

Browsertrix is a cloud-native, high-fidelity, browser-based crawling service designed
to make web archiving easier and more accessible for everyone.
Browsertrix is a cloud-native, high-fidelity, browser-based crawling service designed to make web archiving easier and more accessible for everyone.

The service provides an API and UI for scheduling crawls and viewing results, and managing all aspects of crawling process. This system provides the orchestration and management around crawling, while the actual crawling is performed using [Browsertrix Crawler](https://github.com/webrecorder/browsertrix-crawler) containers, which are launched for each crawl.

See [browsertrix.com](https://browsertrix.com) for a feature overview and information about Browsertrix hosting.
See [webrecorder.net/browsertrix](https://webrecorder.net/browsertrix) for a feature overview and information about how to sign up for Webrecorder's hosted Browsertrix service.

## Documentation

The full docs for using, deploying, and developing Browsertrix are available at: [https://docs.browsertrix.com](https://docs.browsertrix.com)
The full docs for using, deploying, and developing Browsertrix are available at [docs.browsertrix.com](https://docs.browsertrix.com).

Our docs are created with [Material for MKDocs](https://squidfunk.github.io/mkdocs-material/).

## Deployment

The latest deployment documentation is available at: [https://docs.browsertrix.com/deploy](https://docs.browsertrix.com/deploy)
The latest deployment documentation is available at [docs.browsertrix.com/deploy](https://docs.browsertrix.com/deploy).

The docs cover deploying Browsertrix in different environments using Kubernetes, from a single-node setup to scalable clusters in the cloud.

Previously, Browsertrix also supported Docker Compose and podman-based deployment. This has been deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.
Early on, Browsertrix also supported Docker Compose and podman-based deployment. This was deprecated due to the complexity of maintaining feature parity across different setups, and with various Kubernetes deployment options being available and easy to deploy, even on a single machine.

Making deployment of Browsertrix as easy as possible remains a key goal, and we welcome suggestions for how we can further improve our Kubernetes deployment options.

If you are looking to just try running a single crawl, you may want to try [Browsertrix Crawler](https://github.com/webrecorder/browsertrix-crawler) first to test out the crawling capabilities.

## Development Status
## Contributing

Browsertrix is currently in a beta, though the system and backend API is fairly stable, we are working on many additional features.
Though the system and backend API is fairly stable, we are working on many additional features. Please see the GitHub issues and [this GitHub Project](https://github.com/orgs/webrecorder/projects/9) for our current project plan and tasks.

Additional developer documentation is available at [https://docs.browsertrix.com/develop](https://docs.browsertrix.com/develop/)
Guides for getting started with local development are available at [docs.browsertrix.com/develop](https://docs.browsertrix.com/develop/).

Please see the GitHub issues and [this GitHub Project](https://github.com/orgs/webrecorder/projects/9) for our current project plan and tasks.
### Translation

We use [Weblate](https://hosted.weblate.org/engage/browsertrix/) to manage translation contributions.

<img src="https://hosted.weblate.org/widget/browsertrix/browsertrix-ui/multi-auto.svg" alt="Translation status" />

## License

2 changes: 1 addition & 1 deletion assets/browsertrix-lockup-color-dynamic.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading