Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression fix: add profiles/ prefix to profile filenames #1365

Merged
merged 8 commits into from
Nov 10, 2023

Conversation

tw4l
Copy link
Member

@tw4l tw4l commented Nov 9, 2023

Fixes #1364

Regression fix for issue introduced in storage refactoring (see issue for more details).

Changes:

  1. Add profiles/ prefix to profile filename passed in to crawler for profile creation and written into db
  2. Remove hardcoded profiles/ prefix from crawler YAML
  3. Add migration to add profiles/ prefix to profile filenames that don't already have it

This way between the related storage document and the profile filename, we have the full path to the object in the database rather than relying on additional prefixes hardcoded into k8s job YAML files.

Note that this as a follow-up it'll be necessary to manually move any profiles that had been written into the <oid> "directory" in object storage rather than <oid>/profiles to the latter. This should only affect profiles created very recently in a 1.8.0-beta release.

Manually tested but please double check my work!

@tw4l tw4l requested a review from ikreymer November 9, 2023 23:54
@tw4l tw4l force-pushed the issue-1364-profiles-prefix branch from 8d98baa to c0b34c4 Compare November 10, 2023 01:01
@ikreymer ikreymer merged commit 82a5d1e into main Nov 10, 2023
4 checks passed
@ikreymer ikreymer deleted the issue-1364-profiles-prefix branch November 10, 2023 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Using browser profiles results in crawl failures
2 participants