Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LG-15217 Add submit_attempts property to doc auth troubleshooting option actions #11682

Merged
merged 6 commits into from
Jan 2, 2025
Merged

LG-15217 Add submit_attempts property to doc auth troubleshooting option actions #11682

merged 6 commits into from
Jan 2, 2025

Conversation

jennyverdeyen
Copy link
Member

@jennyverdeyen jennyverdeyen commented Dec 20, 2024 •
edited
Loading

🎫 Ticket

Link to the relevant ticket:
LG-15217

🛠 Summary of changes

We want to be able to track metrics for users who choose to "try again online" or fall back to "try in person" after not being able to verify their ID in the remote flow. In particular we want to know how many submission attempts they've made before eventually trying in person. This PR adds submit_attempts to the analytics for these events: "Frontend: IdV: Capture troubleshooting dismissed" and "Frontend: IdV: Verify in person troubleshooting option clicked"

📜 Testing Plan

Provide a checklist of steps to confirm the changes.

  • Run make watch_events in your terminal to view the analytics events

  • Enter the application through Sinatra, choosing "identity-verified"

  • Begin the remote flow by choosing "Continue online" on /verify/how_to_verify

  • Choose "upload photos" to reach the document capture step

  • Upload failing images and submit

  • Choose "Try Again"

  • Observe that the event "Frontend: IdV: Capture troubleshooting dismissed" has been logged, with submit_attempts included under event_properties. It should accurately reflect how many times you've tried to submit the doc. (So far, 1)

  • Back on the document capture page, upload another erroring/failing ID (you may need a separate failing YML or reload the page to upload the same ones you just uploaded)

  • This time, choose "Try in person"

  • Observe that the event "IdV: verify in person troubleshooting option clicked" has been logged, with submit_attempts included under event_properties. It should accurately reflect how many times you've tried to submit the doc (This time, 2)

👀 Screenshots

Screen.Recording.2024-12-20.at.11.17.17.AM.mov

@jennyverdeyen jennyverdeyen requested review from a team and gina-yamada December 20, 2024 18:09
@gina-yamada
Copy link
Contributor

gina-yamada commented Dec 23, 2024 •
edited
Loading

📓 Testing Notes:

❓If a user starts over, should the count start over? It makes sense that is does not start over- your comments say user so I bet not but just checking
✅ Count increases with every doc auth attempt and is as I'd expect
✅ Count increased when I was using hybrid (tested w/ telephony) and is as I'd expect
✅ Tested if a user fails doc auth, then starts ID-IPP and decides they don't want to do it- and tries doc auth again (only to fail) and then starts ID-IPP - count as I'd expect
✏️ I think the last sentence in Summary of changes needs a small update. You state This PR adds submit_attempts to the analytics for these events: "Frontend: IdV: Capture troubleshooting dismissed" and "Frontend: IdV: Capture troubleshooting dismissed" You mentioned Frontend: IdV: Capture troubleshooting dismissed twice but not IdV: Verify in person troubleshooting option clicked.
❌ I think the name submit_attempts is vague, especially on IdV: Verify in person troubleshooting option clicked. Can we give this a more descriptive name? (I recently had to look up key/value pairs on a handful of events to confirm what each was collecting.)

gina-yamada
gina-yamada reviewed Dec 23, 2024
View reviewed changes
app/services/analytics_events.rb Outdated
@@ -1198,12 +1198,14 @@ def idv_cancellation_visited(
# @param ["hybrid","standard"] flow_path Document capture user flow
# @param [String] use_alternate_sdk
# @param [Boolean] liveness_checking_required
# @param [Integer] submit_attempts Times that user has tried submitting
Copy link
Contributor

@gina-yamada gina-yamada Dec 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe try... number of doc auth submit_attempts for user

gina-yamada
gina-yamada reviewed Dec 23, 2024
View reviewed changes
app/services/analytics_events.rb
@@ -5268,16 +5271,19 @@ def idv_verify_by_mail_enter_code_visited(

# @param ["hybrid","standard"] flow_path Document capture user flow
# @param [Boolean] opted_in_to_in_person_proofing User opted into in person proofing
# @param [Integer] submit_attempts Times that user has tried submitting
# The user clicked the troubleshooting option to start in-person proofing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you update the description above, maybe do so here too

@jennyverdeyen
LG-15217 Add submit_attempts property to idv_verify_in_person_trouble…
b3f2875 
…shooting_option_clicked and idv_capture_troubleshooting_dismissed

changelog: Internal, In-person proofing, add submit_attempts property to try again and IPP fallback doc auth troubleshooting events
@jennyverdeyen
Clean up and fix tests
59649a3
@jennyverdeyen
Update more tests to include new event property
609397f
@jennyverdeyen
use identityconfig instead of hardcoding submit_attempts in a test
40f6346
@jennyverdeyen
Update description for submit_attempts in upload.tsx
dd5c117
@jennyverdeyen jennyverdeyen force-pushed the jverdeyen/LG-15217-add-submit-attempts-event-property branch from 2f96070 to dd5c117 Compare January 2, 2025 16:16
shanechesnutt-ft
shanechesnutt-ft reviewed Jan 2, 2025
View reviewed changes
Copy link
Contributor

@shanechesnutt-ft shanechesnutt-ft left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through the test plan things look good. I agree with @gina-yamada's doc and rename comments. I will approve after those comments are addressed.

@jennyverdeyen
Copy link
Member Author

❓If a user starts over, should the count start over? It makes sense that is does not start over- your comments say user so I bet not but just checking

Right, I think it makes sense that the count doesn't start over if the user starts over. It seems like the count only resets after the 6-hour expiration period from the first attempt (set by the config variable doc_auth_attempt_window_in_minutes)

✏️ I think the last sentence in Summary of changes needs a small update. You state This PR adds submit_attempts to the analytics for these events: "Frontend: IdV: Capture troubleshooting dismissed" and "Frontend: IdV: Capture troubleshooting dismissed" You mentioned Frontend: IdV: Capture troubleshooting dismissed twice but not IdV: Verify in person troubleshooting option clicked.

🤦‍♀️ Thank you, updated the description!

❌ I think the name submit_attempts is vague, especially on IdV: Verify in person troubleshooting option clicked. Can we give this a more descriptive name? (I recently had to look up key/value pairs on a handful of events to confirm what each was collecting.)

This one got me thinking! submit_attempts is the name of the property in many of the of the doc auth related analytics events which is why I chose that same name here - BUT the event names don't have mention of doc auth/doc capture in them, making it a little less clear what submit_attempts refers to, compared to the other events. I think "IdV: verify in person troubleshooting option clicked" might be clearer named it if were called "IdV: Document capture troubleshooting - in person option clicked" or something. Not in scope for this so I'm going to leave the property name as is so it's consistent with the other similar events. Also we discussed this on slack but I'm commenting here for visibility!

@gina-yamada gina-yamada self-requested a review January 2, 2025 17:13
gina-yamada
gina-yamada approved these changes Jan 2, 2025
View reviewed changes
Copy link
Contributor

@gina-yamada gina-yamada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing feedback. Other than fixing broken specs, I think you are all set. Nice work @jennyverdeyen ! 🚀

shanechesnutt-ft
shanechesnutt-ft approved these changes Jan 2, 2025
View reviewed changes
Copy link
Contributor

@shanechesnutt-ft shanechesnutt-ft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me! 👍🏻

@jennyverdeyen jennyverdeyen merged commit 2f476bb into main Jan 2, 2025
2 checks passed
@jennyverdeyen jennyverdeyen deleted the jverdeyen/LG-15217-add-submit-attempts-event-property branch January 2, 2025 20:52
@solipet solipet mentioned this pull request Jan 7, 2025
Merged
matthinz pushed a commit that referenced this pull request Jan 7, 2025
@jennyverdeyen @matthinz
LG-15217 Add submit_attempts property to doc auth troubleshooting opt…
9ea877d 
…ion actions (#11682)

* LG-15217 Add submit_attempts property to idv_verify_in_person_troubleshooting_option_clicked and idv_capture_troubleshooting_dismissed

changelog: Internal, In-person proofing, add submit_attempts property to try again and IPP fallback doc auth troubleshooting events
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shanechesnutt-ft shanechesnutt-ft shanechesnutt-ft approved these changes

@gina-yamada gina-yamada gina-yamada approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jennyverdeyen @gina-yamada @shanechesnutt-ft