Add option to add and update additional scan profile details #5660

flozia · 2025-02-25T17:22:39Z

References:

Jira: MNTOR-4065
Figma: https://www.figma.com/design/2iCgADpFXKOFZTEgCPkQsR/Settings?node-id=597-10052&t=c2UkGyKPdbvneuz5-4

Description

Adds additional scan profile details and the option to update them:

The migration file for the changes in this PR is split out to PR Migration for PR #5660 #5664.
Updates the /admin/dev panel to add the option to test the backend changes.

How to test

Run the migrations: npm run db:migrate
Run an initial welcome scan
Enable the feature flag EditScanProfileDetails
Make sure you can still run an initial scan
Visit /admin/dev
Make changes to the scan profile
Confirm any profile updates are applied to the table onerep_profiles and also show for the remote profile

Checklist (Definition of Done)

Commits in this PR are minimal and have descriptive commit messages.
I've added or updated the relevant sections in code comments
I've added a unit test to test for potential regressions of this bug.
All acceptance criteria are met.
Jira ticket has been updated with suggestions for QA when this PR is deployed to stage.

…le details to the onerep_profiles table

github-actions · 2025-02-26T10:58:05Z

Preview URL 🚀 : https://blurts-server-pr-5660-mgjlpikfea-uk.a.run.app

Vinnl

I didn't closely review the admin UI code since it's running locally only anyway, but it's looking really neat! And the functionality seemed to work here.

src/app/functions/server/onerep.ts

src/knex-tables.d.ts

src/app/functions/server/updateDataBrokerScanProfile.ts

src/app/functions/server/updateDataBrokerScanProfile.test.ts

src/db/tables/onerep_scans.ts

rhelmer

lgtm overall, I reviewed but did not test this locally, @flozia do you want me to help with that?

The only really important question I have is how you intend to enforce the limits on the number of allowable fields, since this involves both DB storage and external API calls it is probably more complex, but I included a reference to some prior art we have in this area (the limit on secondary emails in the settings screen).

...ated)/user/(dashboard)/dashboard/fix/data-broker-profiles/removal-under-maintenance/page.tsx

src/app/functions/server/updateDataBrokerScanProfile.test.ts

rhelmer · 2025-03-05T17:56:01Z

src/constants.ts

+  "phone_numbers",
+  "addresses",
+] as const;
+export const CONST_DATA_BROKER_PROFILE_DETAIL_LIMITS = {


How are these limits enforced? We found in the past that is was insufficient to do this on the front-end, and we ended up enforcing this at the database level using transactions with a strong isolation level

blurts-server/src/db/tables/emailAddresses.ts

Line 49 in d92af9a

await trx.raw("SET TRANSACTION ISOLATION LEVEL SERIALIZABLE");

The database entries are checked for their limits before writing the updates here. Since this still leaves a window of opportunity to insert more entries than the limit allows for setting the transaction isolation level is a good idea. Added in 27b9cc8.

rhelmer · 2025-03-05T17:57:18Z

src/knex-tables.d.ts

@@ -320,7 +320,6 @@ declare module "knex/types/tables" {
  interface OnerepProfileRow {
    id: number;
    onerep_profile_id: null | number;
-    name_suffix: null | string;


I think I missed the memo on this, are we deprecating the name_suffix field entirely?

No, we are keeping it for now: 41912ff.

Vinnl

The changes look good to me too. (Except the transaction level stuff, since I don't know what that is, but I'll leave that to you and Rob :) )

Migration for PR #5660

flozia · 2025-03-11T10:24:28Z

The Deploy Preview Action is also stuck for this PR. Closing in favor of PR #5697.

github-actions · 2025-03-11T10:26:27Z

Cleanup completed - database 'blurts-server-pr-5660' destroyed, cloud run service 'blurts-server-pr-5660' destroyed

github-actions · 2025-03-11T11:02:10Z

Cleanup completed - database 'blurts-server-pr-5660' destroyed, cloud run service 'blurts-server-pr-5660' destroyed

flozia added 10 commits February 17, 2025 18:41

chore: Extend interface OnerepProfileRow to include alternate items

28b19b9

WIP: Update profile data

dbc2b5d

migration: Add additional profile details to onerep_profiles

bfaf6cc

migration: Alter and add profile detail addresses to onerep_profiles

af18e56

WIP: Update profile data

2d60905

feat: Set local and remote profile details

874dbea

feat: Add unit tests for updateDataBrokerScanProfile

b8bd65d

merge: main -> mntor-4065

b1b269a

fix: Update use of address for when creating a profile

2a6b69b

fix: Unit test coverage for file onerep_scans.ts

a321f77

flozia requested review from rhelmer and mansaj and removed request for rhelmer and mansaj February 25, 2025 17:22

flozia changed the title ~~Add option to add and update additional scan profile details~~ WIP: Add option to add and update additional scan profile details Feb 25, 2025

flozia added 2 commits February 26, 2025 11:13

chore: Merge migration files for altering and adding additional profi…

e1c8d90

…le details to the onerep_profiles table

chore: Add feature flag EditScanProfileDetails

0de73e5

flozia force-pushed the mntor-4065 branch from 0e6c4fe to 0de73e5 Compare February 26, 2025 10:48

flozia marked this pull request as ready for review February 26, 2025 10:53

flozia requested review from rhelmer, Vinnl and mansaj February 26, 2025 10:53

flozia changed the title ~~WIP: Add option to add and update additional scan profile details~~ Add option to add and update additional scan profile details Feb 26, 2025

flozia self-assigned this Feb 26, 2025

flozia mentioned this pull request Feb 26, 2025

Migration for PR #5660 #5664

Closed

flozia force-pushed the mntor-4065 branch from 04cf48a to 47ec48b Compare February 26, 2025 12:58

flozia marked this pull request as draft February 26, 2025 13:06

flozia removed request for rhelmer and Vinnl February 26, 2025 13:06

flozia requested a review from mansaj February 26, 2025 14:47

flozia force-pushed the mntor-4065 branch from 70a8e4b to 29955e0 Compare February 26, 2025 15:20

flozia added 2 commits February 26, 2025 16:31

Merge branch 'mntor-4065-migrations' into mntor-4065

8b7874b

Merge branch 'main' into mntor-4065

eb5c1f0

Vinnl approved these changes Mar 3, 2025

View reviewed changes

flozia added 6 commits March 4, 2025 11:36

chore: Explicitly selected to profile info properties to PUT

93b35f4

chore: Move type UpdateableProfileDetails

cc99fff

chore: Wrap logged variable in delimiters

ad97615

chore: Throw at first detail that exceeds the limit

74c8838

chore: Stop at first profile hit in

953e0f1

chore: Update unit tests for updateDataBrokerScanProfile

29a8735

rhelmer requested changes Mar 5, 2025

View reviewed changes

flozia mentioned this pull request Mar 6, 2025

Migrate addresses in table onerep_profiles #5688

Draft

flozia added 2 commits March 10, 2025 11:24

chore: Readd name_suffix

41912ff

chore: Set transaction isolation level for updateProfileDetails

27b9cc8

flozia requested a review from rhelmer March 10, 2025 10:43

Vinnl approved these changes Mar 10, 2025

View reviewed changes

flozia mentioned this pull request Mar 10, 2025

Migration for PR #5660 #5696

Merged

rhelmer approved these changes Mar 11, 2025

View reviewed changes

flozia added a commit that referenced this pull request Mar 11, 2025

Merge pull request #5696 from mozilla/mntor-4065-migrations

f209944

Migration for PR #5660

Base automatically changed from mntor-4065-migrations to main March 11, 2025 10:06

merge: main -> mntor-4065

f3e6f5f

flozia closed this Mar 11, 2025

flozia mentioned this pull request Mar 11, 2025

Add option to add and update additional scan profile details #5697

Merged

5 tasks

flozia reopened this Mar 11, 2025

Vinnl closed this Mar 11, 2025

flozia deleted the mntor-4065 branch March 11, 2025 11:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to add and update additional scan profile details #5660

Add option to add and update additional scan profile details #5660

flozia commented Feb 25, 2025 •

edited

Loading

github-actions bot commented Feb 26, 2025

Vinnl left a comment

rhelmer left a comment

rhelmer Mar 5, 2025

flozia Mar 10, 2025

rhelmer Mar 5, 2025

flozia Mar 10, 2025

Vinnl left a comment

flozia commented Mar 11, 2025 •

edited

Loading

github-actions bot commented Mar 11, 2025

github-actions bot commented Mar 11, 2025

Add option to add and update additional scan profile details #5660

Add option to add and update additional scan profile details #5660

Conversation

flozia commented Feb 25, 2025 • edited Loading

References:

Description

How to test

Checklist (Definition of Done)

github-actions bot commented Feb 26, 2025

Vinnl left a comment

Choose a reason for hiding this comment

rhelmer left a comment

Choose a reason for hiding this comment

rhelmer Mar 5, 2025

Choose a reason for hiding this comment

flozia Mar 10, 2025

Choose a reason for hiding this comment

rhelmer Mar 5, 2025

Choose a reason for hiding this comment

flozia Mar 10, 2025

Choose a reason for hiding this comment

Vinnl left a comment

Choose a reason for hiding this comment

flozia commented Mar 11, 2025 • edited Loading

github-actions bot commented Mar 11, 2025

github-actions bot commented Mar 11, 2025

flozia commented Feb 25, 2025 •

edited

Loading

flozia commented Mar 11, 2025 •

edited

Loading