Every time a new releaseduty cycle begins, new RelEng people are (re)ramping up. To ease the transition, we're keeping this CHANGELOG that should trace all the tools/productivity/infrastructure changes. This includes high-level changes and should come into compliance with the rest of the documentation.
This page best serves the people that have previously been into the releaseduty cycle. Starting from the baseline, people coming back after N cycles can ramp up incrementally with the latest changes.
As releaseduty squirrels are the ones with the best context when it comes to releases, they are the ones to edit this page and amend it accordingly. Keep in mind that changes should be in compliance with the other pieces of documentation.
- Adding new
base_revandbase_refparameters in taskgraph (bug 1784232)- Deployed new
build-decisionimage (D155195) - List to be expanded once new changes are deployed.
- Deployed new
- Balrog 3.20 deployed
- Deployed new iscript + scriptworker version to mac signers
- FirefoxCI cluster upgraded to Taskcluster 44.18.0
- relduty no longer announces minor upgrades on dev-platform
- update-verify sends requests with "Cache-Control: max-stale=0" to avoid getting out of date responses from the CDN fronting balrog (https://bugzilla.mozilla.org/show_bug.cgi?id=1778728)
- Android:
- Android-Components gets released off of https://shipit.mozilla-releng.net/ instead of Github Releases (RELENG-11)
- Fenix/Focus are not immediately shipped anymore. We now have a 2-step process where builds are promoted and sent to our testing population before shipping them to Github or archive.mozilla.org (bug 1776830 and bug 1778056)
- Version number now follow the gecko scheme. This means 104.0.0-beta.1 will instead be called 104.0b1 (bug 1777255)
- L10n string uplift are automatically approved by mergify (bug 1779797)
- Android-Components bumps are also automatically approved (bug 1780740)
- FirefoxCI cluster upgraded to Taskcluster 44.14.0
- https://github.com/mozilla/delivery-dashboard/ deployed (first time since 2018!)
- Deployed balrog v3.18 to production (https://bugzilla.mozilla.org/show_bug.cgi?id=1772799)
- (ahal) Copied new scriptworker CoT private key to mac signers
- (ahal) Deployed scriptworker pools to pick up new CoT private key
- (ahal) Updated dependencies in image-task docker image
- Treestatus updated for esr-102 and deployed
- signingscript update for authenticode timestamping
- scriptworker-scripts and shipit updated for esr-102 and deployed
- new docker-worker pools to fix ssl certs
- Widevine 4.10.2449.0 final rollout
- aus5.mozilla.org (balrog prod client domain) certificate renewed (https://bugzilla.mozilla.org/show_bug.cgi?id=1768253
- FirefoxCI cluster upgraded to Taskcluster 44.13.7
- Updated Mac certificates on all prod mac hosts. + Nightly builds now use the same keychain file as release.
- FirefoxCI cluster upgraded from Taskcluster 44.8.4 to 44.11.1; Postgres DB instance doubled in RAM.
- Updated all iOS apps in Bitrise to use new cert and provisioning profiles
- https://pypi.org/project/mozilla-version/ updated to 1.0.1 -> beetmover, bouncer, pushapk, treescript deps updated
- Experimental change to timestamp server for signing windows firefox, mozilla-releng/scriptworker-scripts#501
- Allowed System Addons to be served from archive.m.o, mozilla-releng/balrog#2520
- Deployed Balrog v3.16 to production
- FirefoxCI cluster upgraded from 43.2.0 -> 44.8.1
beetmover-geckoviewjobs (copying geckoview artifacts to maven) now run as part of thepromote_firefoxgraphs on release and beta instead ofpush_firefox- Switched SENTRY_DSN for Balrog -> sentry.io
- SHA-256 signing of Windows Installers, bug 1745467
- Widevine update, bug 1758423 (nightlytest, nightly, and beta channels)
- Treescript merge day CLOBBER datestring
- Internal pypi uploads for numpy/scipy/pillow, bug 1757792
- Deployed pollbot v1.4.7 to production
- Deployed balrog v3.14 to production
- Deployed balrog v3.13 to production
- Minor refactor to release_artifacts in
gecko_taskgraph - Re-imaged mac-v1-signing3 and mac-v1-signing8. Quarantined mac-v1-signing19 for MozillaVPN work.
- Signingscript retries for winsign (timestamp.digicert.com), bug 1590599
- RELENG-659 - Py3 support on xpi/addons and updated the taskcluster version used
- Change ShipIt to point to xpi-manifest "main" branch
- Deployed new version of iscript to mac signers
- Deployed balrog v3.12 to production
- Deployed signingscript to production
- Modified mac-signing releng ssh allowlist
- l10n-cross-channel-quarantine now nukes comm/ at the end of the run
- Updated puppet dependencies for mac signers
- fixed treescript merge-automation to be idempotent
- refactored the system addons balrog rules
- removed pushsnap scriptworker pools
- focus-android releases switched to taskgraph
- firefox desktop update-verify watershed
- new firefox desktop msix packages
- update-verify async download PR
- partner attribution configs landed
- balrog v2 submission
- ESR68 EOL
- central-to-beta merges should automatically l10n-bump
- L10n bumper will look at the hg pushlog for the default revision, rather than Elmo. If we need to pin a previous revision due to bustage, land a change updating the revision and changing
pintotruefor that locale. - ESR68 l10n bumper disabled.
- partner builds moved from b8 to b5 to adjust for shorter release cycles
- partner attributions landed
- Lots of accounts disabled; watch for fallout!
- Changelog moved to rtd
- Mac Notarization is now single-zip instead of multi-account due to bug 1620697. This may slow down the success case by a few minutes, but reduce or remove the multi-hour manual-intervention failure case.
- Scriptworker now catches a number of
asyncio.TimeoutErrors, and should be more robust.
- added support for shipping Firefox Flatpaks to Flathub
- py38 rollout for scriptworkers, tooltool, balrog, and other release services
- aws -> gcp, then back again, for tier 1 workers
- mac 3-tier notarization
- (still-in-progress) signing servers are being shut down. we've already stopped using them, however
- mergeduty processed is back to two stages like it used to be before Quantum: beta -> release in the first iteration while central -> beta and bumping central in the second week
- l10n bumper for central is moving to taskcluster cron shortly; will uplift to beta and esr68 later
- switched to GCP scriptworkers in all gecko trees (including esr68)
- after TCW of 9th Nov, switched to GCP scriptworkers in
commrepos too, but also Github mobile projects - secrets are now managed in
sops, no longer inhiera - decommissioning of the AWS infrastructure
- treescript has been refactored
- scriptworker-scripts docker images now mount the repo at
/appinstead of the script (we may revisit this to lower disk usage) - mergeduty process change for upcoming automated betas. tl;dr - no longer Devedition b1 and b2 during RC week. central merges to beta a week later
- mac signing now happens on the mac notarization pool.
- mac nightly and release builds are now notarized as well as signed.
- mac signing tasks also create pkg installers
- update & final verify will check SSL certificates of Balrog, archive.m.o etc
- the release promotion action is now a hook.
- chain of trust no longer uses gpg to create and verify signatures.
- scriptworker 22 now signs and verifies ed25519 signatures; gpg signatures are deprecated.
- partner repacks are now enabled on ESR60
- adding list of whatsnewpage locales in-tree is now part of mergeduty
- removed thunderbird docs. thunderbird is now managed by thunderbird team and they have support for shipit v2
- MARs are now signed via autograph (hashes only)
- Mergeduty tasks are now tracked in Trello
- Releaseduty now assists CIDuty with puppet pyup PRs once per week
- added 6 new mac signing servers
- switched to ship-it v2
- reduced mac signing concurrency from 4 to 2 to reduce server churn
- switched to Ship-it v2 for beta releases
- releasewarrior local automation to add WNP
- added partner off-cycle support + docs
- widevine docs
- update verify only blocks on required platforms to start (eg: win32 update verify only blocks on win32 beetmover and balrog)
- python 2.7.15 rollout to a portion of our infra
- python 3.6.5 rollout completed to our scriptworkers
- update verify no longer fails on expected differences, but will fail on any files found only in source or target; bug 1461490
- releaseduty now responsible for periodic file update approvals in phabricator. Change this line to update those responsible.
- added
esr52-latest*bouncer aliases - added
firefox-esr-next-*bouncer aliases - added esr60 support
- rerun action task in Treeherder - hopefully will help avoid duplicate reruns
- KEY file to releases directory in bug 1446816 for >=60.0b11
- added addonscript for langpack pushing to amo
- re-added source tarball for >=60.0b6
- updated scriptworkers to share more client code
- update verify now fails on any differences, and creates a summary log for >=60.0b8. additional documentation on how to interpret failures are available.
- split snap repackage and push task in bug 1447263
- updated rr3 add partner configs for fx60+
- tc partner repacks!
- split esr52 and esr60 templates
- fixed beta+fennec bouncer aliases
- push-apk-breakpoint removed in bug 1432817
- we track all major changes in this newly added CHANGELOG
- separate RC docs for both Fennec and Firefox
- new-style WNP setup, as documented.
- we now have support for snaps in
releasechannel
- old release promotion based on releasetasks is dead. Everything is in-tree scheduled now. Changes are riding the trains.
- all releaseduty related documentation has been moved out of the wiki under warrior's
docssubfolder. - use
release graphidto enter graphids, rather than editing the .json manually. - in-tree release notifications going away in favor of native taskcluster notifications. pulse-notify is to be retired whenever we kill esr52
- Balrog toplevel submission and release scheduling now happens in balrog scriptworker for Fx60+.
- the use of the new releasewarrior
- old how-tos are now gathered under one roof in releasewarrior-2.0/old-howtosreleasewarrior-2.0/old-how-tos but they are gradually being migrated to the wiki which becomes the single source of truth
- authentication method for taskcluster-cli changed. If you are upgrading from an older version of taskcluster-cli, you may have to remove the
${HOME}/.config/taskcluster.ymlfile to work with the new authentication method.
- dropped support for old releasewarrior
- dropped the need for QE signoffs in Balrog
- a merge day instance has been added to ease the handover during mergedays but also for speed and reliability. More on this here. When connecting to the merge day instance, keep in mind to ensure its environment is clean and ready for use
- simplified documentation for mid-betas and release checklists. If not already, duplicate an existing sheet in this google docs checklist and clear out the status that was carried over from the previous release.
- mergeduty dates shifted. We now merge
betatoreleaseandcetraltobetain the same day, keeping a relbranch forbetaand holdcentralfrom version bump until before the release. - because of 57 and the aforementioned mergeduty change, the exception becomes rule so that for each new beta cycle X, we have X.0b1 and X.b2 just for
Devedition, whileFirefoxstarts at X.b3
- dropped support for tctalker and solely rely on taskcluster-cli. More on its installation here
- releaseduty now checks on status of central nightlies when not undertaking other duties
- releaseduty now handles signing of arbitrary .apk's as requested by MoCo teams (Klar/Focus, Rocket)
- release-drivers mailing list is no longer used for signoffs/gtb's, please subscribe to release-signoffs@
- QE will now send an explicit GO for DevEdition Beta's to be pushed live, usually in the same e-mail as the GO for Beta
- Fennec single locales no longer shipped