Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change work_histories migration to insert via SQL #9853

Merged
merged 1 commit into from
Sep 26, 2024

Conversation

CatalinVoineag
Copy link
Contributor

@CatalinVoineag CatalinVoineag commented Sep 20, 2024

Context

The data migration BackfillApplicationChoicesWithWorkExperiences was inserting records via
ActiveRecord. This caused an incident, known as The Big Touch. Many application choices, forms, and candidates had their updated_at updated and this caused some of our APIs to be slow.

We still need to run this migration in sandbox so we thought it's a good idea to change this migration to insert records via SQL avoiding any callbacks.

This doesn't need to run again in production, only sandbox, which has around 38k records that need updating

Ran this locally with 120k records, the workers finished their 5k batch insert in around 20 seconds, much faster than before when we were using ActiveRecord.

Changes proposed in this pull request

Changed MigrateApplicationChoicesWorker

Guidance to review

Things to check

  • If the code removes any existing feature flags, a data migration has also been added to delete the entry from the database
  • This code does not rely on migrations in the same Pull Request
  • If this code includes a migration adding or changing columns, it also backfills existing records for consistency
  • If this code adds a column to the DB, decide whether it needs to be in analytics yml file or analytics blocklist
  • API release notes have been updated if necessary
  • If it adds a significant user-facing change, is it documented in the CHANGELOG?
  • Required environment variables have been updated added to the Azure KeyVault
  • Inform data insights team due to database changes
  • Make sure all information from the Trello card is in here
  • Rebased main
  • Cleaned commit history
  • Tested by running locally
  • Add PR link to Trello card

Copy link
Contributor

You have one or more flakey tests on this branch! ❄️ ❄️ ❄️

Failed 1 out of 3 times at ./spec/system/provider_interface/provider_changes_conditions_on_an_offer_spec.rb:8: ⚠️ expected not to find text "and another" in "Cookies on Manage teacher training applications We use some essential cookies to make this service work. We’d also like to use analytics cookies so we can understand how you use the service and make improvements. Accept analytics cookies Reject analytics cookies View cookies Skip to main content GOV.UK Organisation settings Your account Sign out Test Applications Interview schedule Reports Activity log Back Update offer - John Smith Check and send new offer Course details Training provider Chuck Koss College-13 (GI8) Course Commerce (QZPB) Full time or part time Full time Location Flowerlake High 334 948 White Fields North Cammy Central Z5 5JB Qualification QTS with PGCE Funding type Fee Conditions of offer Candidates are required to receive 2 references. Add or change conditions Ex aut nostrum illum. Pending condition Pending and another Pending Send new offer Cancel Get help Email [email protected] You will get a response within 5 working days, or one working day for urgent requests. Support links How to use this service Changes to this service Accessibility Cookies Privacy Guidance for using AI © Crown copyright"

2 similar comments
Copy link
Contributor

You have one or more flakey tests on this branch! ❄️ ❄️ ❄️

Failed 1 out of 3 times at ./spec/system/provider_interface/provider_changes_conditions_on_an_offer_spec.rb:8: ⚠️ expected not to find text "and another" in "Cookies on Manage teacher training applications We use some essential cookies to make this service work. We’d also like to use analytics cookies so we can understand how you use the service and make improvements. Accept analytics cookies Reject analytics cookies View cookies Skip to main content GOV.UK Organisation settings Your account Sign out Test Applications Interview schedule Reports Activity log Back Update offer - John Smith Check and send new offer Course details Training provider Chuck Koss College-13 (GI8) Course Commerce (QZPB) Full time or part time Full time Location Flowerlake High 334 948 White Fields North Cammy Central Z5 5JB Qualification QTS with PGCE Funding type Fee Conditions of offer Candidates are required to receive 2 references. Add or change conditions Ex aut nostrum illum. Pending condition Pending and another Pending Send new offer Cancel Get help Email [email protected] You will get a response within 5 working days, or one working day for urgent requests. Support links How to use this service Changes to this service Accessibility Cookies Privacy Guidance for using AI © Crown copyright"

Copy link
Contributor

You have one or more flakey tests on this branch! ❄️ ❄️ ❄️

Failed 1 out of 3 times at ./spec/system/provider_interface/provider_changes_conditions_on_an_offer_spec.rb:8: ⚠️ expected not to find text "and another" in "Cookies on Manage teacher training applications We use some essential cookies to make this service work. We’d also like to use analytics cookies so we can understand how you use the service and make improvements. Accept analytics cookies Reject analytics cookies View cookies Skip to main content GOV.UK Organisation settings Your account Sign out Test Applications Interview schedule Reports Activity log Back Update offer - John Smith Check and send new offer Course details Training provider Chuck Koss College-13 (GI8) Course Commerce (QZPB) Full time or part time Full time Location Flowerlake High 334 948 White Fields North Cammy Central Z5 5JB Qualification QTS with PGCE Funding type Fee Conditions of offer Candidates are required to receive 2 references. Add or change conditions Ex aut nostrum illum. Pending condition Pending and another Pending Send new offer Cancel Get help Email [email protected] You will get a response within 5 working days, or one working day for urgent requests. Support links How to use this service Changes to this service Accessibility Cookies Privacy Guidance for using AI © Crown copyright"

@CatalinVoineag CatalinVoineag marked this pull request as ready for review September 20, 2024 16:01
@CatalinVoineag
Copy link
Contributor Author

@CatalinVoineag CatalinVoineag force-pushed the cv/backfill-remaining-choices branch from f70060d to d8e77b0 Compare September 23, 2024 08:54
@CatalinVoineag CatalinVoineag marked this pull request as draft September 23, 2024 08:57
Copy link
Contributor

You have one or more flakey tests on this branch! ❄️ ❄️ ❄️

Failed 1 out of 3 times at ./spec/system/provider_interface/provider_views_a_teacher_degree_apprenticeship_application_spec.rb:25: ⚠️ expected not to find text "Nursing" in "Cookies on Manage teacher training applications We use cookies to make this service work and collect analytics information. To accept or reject cookies, turn on JavaScript in your browser settings or reload this page. Skip to main content GOV.UK Organisation settings Your account Sign out Test Applications Interview schedule Reports Activity log Received This application was received today. You should try and respond to the candidate within 30 days. Application References Notes Timeline Application will be able to edit some sections of their application. You will get a notification if they add any new information. Download application (PDF) Application details Submitted 9 October 2024 at 9am Recruitment cycle 2024 to 2025 - current Application number13 Personal information First name Last name Date of birthNot provided Nationality Contact information Phone numberNot provided Email [email protected] Address Sandbox feature See what this application looks like from the candidate side by signing in as : Sign in as this candidate Course Training provider Adelaide Bogan Academy-2 Course Nursing (SQ8S) Cycle2025 Full time or part time Full time Location Icelyn High School 539 (3N8Y3) 68043 Stamm Camp Estherfurt Bedfordshire UQ1 1BZ Accredited body Adelaide Bogan Academy-2 Qualification QTS with UNDERGRADUATE_DEGREE Funding type Apprenticeship Criminal record and professional misconduct Do you want to declare any safeguarding issues such as a criminal record or professional misconduct? No Work history and unpaid experience Do you have any work history No Do you have any unpaid experience No Qualifications Degree A degree is not required for a teacher degree apprenticeship (TDA). GCSEs or equivalent A levels and other qualifications Do you want to add A levels and other qualifications? No Personal statement Guidance given to candidates You can write about these 3 areas in your personal statement to show how your knowledge, qualifications and experience make you suited to teaching. Include any information that you think supports your application. Explain why you want to be a teacher. Reflect on what motivates you about a career in teaching. Describe why you want to teach this subject or specialism. For primary, describe why you want to teach this age group. Highlight your expertise in a subject gained through your qualifications or work experience. Describe your relevant experience and personal characteristics and explain how they would make you a great teacher. Discuss any relevant experience of working or volunteering with young people and any transferable skills or strengths gained through life experience that would support you in the classroom. The most successful personal statements are between 500 and 1,000 words long, so make sure you include enough detail. Read more about what to include in your personal statement. Qui libero modi. Dolorum nesciunt itaque. Quo rer. Disability support Do you want to ask for help to become a teacher? No Interview availability Do you have any times you cannot be available for interviews? No Sex, disability and ethnicity Get help Email [email protected] You will get a response within 5 working days, or one working day for urgent requests. Support links How to use this service Changes to this service Accessibility Cookies Privacy Guidance for using AI © Crown copyright"

@CatalinVoineag CatalinVoineag force-pushed the cv/backfill-remaining-choices branch from 55004b2 to e19e72d Compare September 25, 2024 15:55
@CatalinVoineag CatalinVoineag force-pushed the cv/backfill-remaining-choices branch from e19e72d to 4604369 Compare September 25, 2024 16:21
@CatalinVoineag CatalinVoineag changed the title Backfill remaining work_histories for choices Change work_histories migration to insert via SQL Sep 25, 2024
@CatalinVoineag
Copy link
Contributor Author

@CatalinVoineag CatalinVoineag marked this pull request as ready for review September 26, 2024 08:59
@CatalinVoineag CatalinVoineag requested a review from a team September 26, 2024 09:00
Copy link
Contributor

@elceebee elceebee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one!

Previously, the data migration
BackfillApplicationChoicesWithWorkExperiences was inserting records via
ActiveRecord. This caused an incident, known as 'The Big Touch'. A lot
of application choices, forms and candidates had their updated_at
updated and this caused some of our APIs to be slow.

We still need to run this migration in sandbox so we thought it's a good
idea to change this migration to insert records via SQL avoiding any
callbacks.
@CatalinVoineag CatalinVoineag force-pushed the cv/backfill-remaining-choices branch from 2e23b0b to 174ed50 Compare September 26, 2024 12:53
@CatalinVoineag CatalinVoineag merged commit 0b05ab8 into main Sep 26, 2024
59 checks passed
@CatalinVoineag CatalinVoineag deleted the cv/backfill-remaining-choices branch September 26, 2024 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants