Refresh Epic: Wagtail in Bedrock

[stevejalim]

Wagtail in Bedrock

This epic outlines the work needed to integrate Wagtail into Bedrock in a way that's compliant with our RRA and generally effective.

Necessary tasks

These will be split out into individual Issues as we prep them

• Move from django_jinja to core Django Jinja2 support - this will make it easer to integrate Wagtail Decision: stick with django-jinja, because its feature set is better
• Refactor i18n mechanism to gain compatibility with wagtail-localize Refactor legacy, custom i18n mechanism to use core django i18n #14256
• 🚧 Integrate Wagtail into Bedrock
  • Groundwork Wagtail in Bedrock: main groundwork #14250
  • Ensure SessionMiddleware doesn't get run in the main Web deployment - it must remain 100% stateless
  • Add a simple migration to change text content of the root page, to avoid user confusion Issue here
  • Disable the Wagtail Userbar extension for jinja #14657
  • CSP tuning for Wagtail admin
    • Support different CSP policies for different URL paths #14576
  • Support pre-generating image renditions
    • Core behaviour Support pre-generating renditions #14578
    • Add django-rq support so we can run background tasks #14601
    • move rendition pre-generation to a background task via queue worker
    • Disable rendition generation if doesn't exist already - catch the request early in middleware on the web deployment invalid concern
    • add workers in k8s to proccess the queue
      • Dev: https://github.com/mozilla-it/webservices-infra/pull/2138
      • Stage: TODO after Dev proven working
      • Prod: TODO after Dev proven working
    • Support scheduled publishing
• Tooling to support local and community dev easier
  • Support filling postgres from sqlite and dumping postgres to sqlite #14541
    • Minorly refactor ProductRelease.notes field #14574 (blocks the above)
  • Add tooling to give local developers an easy way to get appropriate CMS/Bedrock data to work with #14659
  • Add a test that checks for missing migrations #14665
• Plan changes needed to bedrock's startup/management scripts
  • Support uploading the exported-from-Postgres sqlite DB to cloud storage #14660
• Integrate Bedrock's Wagtail with translation vendor
  • Custom work agreed and started
  • Integrate and evaluate wagtail-localize-smartling #14774
  • Add a scheduled task to sync Smartling translations #14776
  • Set up new Smartling project credentials for CMS-backed production #14802
  • Integration hooked in to Bedrock and working end-to-end
• Risk assessment
  • Request inital RRA for updated Bedrock + Wagtail service
  • Act on RRA and adjust plan accordingly
  • Request follow-up RRA ahead of go-live and respond accordingly
• SSO
  • Request SSO credentials for Wagtail in Bedrock -> https://mozilla-hub.atlassian.net/browse/IAM-1350
  • Add SSO support locally
    • Ensure sqlite DB export definitely excludes OIDC tables
  • Apply credentials to k8s, ready for use
  • Add convenience links to Wagtail Admin to link to Django admin and RQ queues #14650
• Infra update enable secure and stable editing of content for Bedrock
  • Confirm and document infra changes
  • JIRA tickets raised (parent epic here)
  • Only expose REDIS_URL to Editing Deployment, not Web Deployment
  • Infra updates started on
    • Dev
    • Stage
    • Prod
• Infra fully updated on
  • Dev
  • Stage
  • Prod
• Update from 5.2.x LTS to 6.1.x ahead of next LTS
• Content management and access planning (Will move to own Epic)

Rollout plan

• Use an SSH tunnel to connect to the relevant postgres DB as the appropriate user and run the pre-fill-postgres script

  • Done for Dev
  • Done for Stage
  • Done for Prod

• Stand up Editing deployment, connected to postgres

• Set MEDIA_URL in infra and also align with the CDN Add helper to get only the hostname out of MEDIA_URL #14723

• CMS mode tuning:

  • Backport K8S config changes so that we capture custom-tuned values for workers and blocking threads
  • Expand CSP for the editing deployment, but only on the editing deployment
  • Add custom CSP for Wagtail that allows just-uploaded images to be shown #14733

• Add new command to keep the editing deployment's DB up to date #14715

  • Update healthz-cron page re postgres backend #14737
  • https://github.com/mozilla-it/webservices-infra/pull/2232

• Once we're happy that the editing deployment is consuming data properly

  • Switch web pods to use the readonly postgres DB replicas and GCS storage bucket
    • Initially, add DATABASE_URL and MEDIA_URL via secrets
    • GRANT pg_read_all_data TO <BEDROCK READONLY USER NAME>;
    • once stable, move MEDIA_URL to env vars and out of secrets
  • Disable script to download new sqlite DB from the bucket if postgres is in play (this is now redundant for Dev/Stage/Prod, but still relevant to Bedrock-test and Demos)

• Add tooling to allow devs to download a sqlite version of the PG database

Final push tasks (ordered in priority terms)

Beta Give feedback

1. Add option to disable case-changing of language codes wagtail-localize-smartling#18
   +0
   
2. Fine tune wagtail-localize fields to exclude slug from translated fields
3. Ensure live wagtail pages are in the sitemap #15022
   Backend Wagtail +2
   
4. Update the postgres-to-sqlite-export script to be a lot more cautious #14937
   +0
   
5. Language selector on CMS pages is showing too many languages #15037
   Backend Bug 🐛 L10N P2 +4
   
6. Support uploading the exported-from-Postgres sqlite DB to cloud storage #14660
   Backend WIP 🚧 +2
   
7. Add tooling to give local developers an easy way to get appropriate CMS/Bedrock data to work with #14659
   3 of 3
   Backend Wagtail +2
   
8. Stage Web deployment to use postgres
9. Prod CMS mode
10. Do not add Aliases for translated pages if they are not ticked for actual translation
11. Ensure DB downloaded is the production one, not the dev one mozmeao/www-sitemap-generator#11
    +0
    
12. DB exporter script fails when workflows are configured #15084
    +0
    
13. Prod Web deployment to use postgres
14. Document the DB export process and enable on stage and prod
15. Disable prod clock pod
16. Update k8s config to capture granian env vars that we've set manually so far

Post-final push CMS tasks

Beta Give feedback

1. Moderation flow tweaks #15087
   2 of 2
   +0
   
2. Add a snitch to the DB export script #15221
   +0
   
3. Support any _permitted_ person to preview content in the CMS before go-live #15181
   WIP 🚧 +1
   
4. CMS deployments are missing assets due to CORS error #15184
   Bug 🐛 +1
   
5. Add cms_url helper to provide lang-prefixed relative path to a CMS page from any page #14871
   +0
   
6. Stop auto-publishing of new locale variants when a page is first sent to translation #15345
   +0
   
7. Support notifications or auto-publishing when a translation lands back #15347
   +0
8. Page tree problems after translating including subtree wagtail/wagtail-localize#773
   +0
9. Support admin/CMS access for demo servers #15375
   5 of 6
   +0
   
10. Update bedrock to no longer need to download the DB to start, if DATABASE_URL is configured
11. Infra cleanup
12. Support private pages
13. Add signal handler to alert when translations land, so they can be published #14934
    L10N Wagtail +2
14. Support CAT tool in Smartling, including support for draft/unpublished pages if possible #14938
    L10N Needs Info WIP 🚧 Wagtail +4
    
15. No indication in UI of pages that have had new, unpublished translations land? wagtail-localize-smartling#19
    +0
    
16. Switch to en as the default langage for pages, not en-US #15039
    +0
    
17. Strikethrough is not supported in Smartling #15498
    +0
    
18. Support automatic linting of translated content #14948
    L10N Needs Info Wagtail +3
19. Explore a CMS flow/UX that ensures core pages are all sent for translation #14969
    +0
20. CMS admin: solve challenge of finding/filtering out pages in various locales #14968
    Backend Wagtail +2
21. String concatenation problems with strings pulled back from Smartling wagtail-localize-smartling#31
    +0
    
22. Add animated GIF support for Wagtail-managed images #15330
    +0
    
23. Wagtail <-> L10N workflow improvements #15348
    2 of 5
    +0
    
24. URLs that use prefer_cms should merge active_locales data #15349
    Bug 🐛 +1
    
25. Update to Wagtail 6.3 LTS #15368
    +0
    
26. Support Smartling's CAT tool wagtail-localize-smartling#34
    enhancement +1
    
27. Improve DB export behaviour around Draft pages
28. Simplify job titles OR Support customisable job titles via settings wagtail-localize-smartling#36
    +0
    
29. Provide a custom callback in Bedrock for wagtail-localize-smartling to tune each Job's description
30. Support syncing down complete translations even if the Job is not yet closed wagtail-localize-smartling#37
    enhancement +1
31. Can we support a note to translators when submitting a job? wagtail-localize-smartling#38
    enhancement +1
    
32. Language picker on VPN resource center index page lists unreachable locales for that page #15521
    Bug 🐛 +1