Releases

Release and Hotfix Procedures

A release consists of packaging and deploying the application from the Development Environment (tdm-dev.azurewebsites.net) (which runs the deploy branch of the client and web api server code against the shared development database) to the User Acceptance Test UAT (tdm-uat.azurewebsites.net) and Production (tdm.ladot.lacity.org) environments.

• Development Environment https://tdm-dev.azurewebsites.net
• UAT Environment: https://tdm-uat.azurewebsites.net
• Production Environment https://tdm.ladot.lacity.org

Automated (Well, Mostly Automated) Deployment to the Development Environment

Changes to the develop code branch in GitHub triggers a GitHub Action that builds a Docker container from the develop branch code and publishes it to Docker Hub (a public Docker Container repository). We have a Microsoft Azure account (owned and managed by John Darragh) that detects publication of a new version of the application Docker container, then pulls it from Docker Hub and automatically replaces the prior container in an Azure App Service on this Azure account to deploy the new application version to tdm-dev.azurewebsites.net. Progress of the container build can be verified from the Actions Tab in the GitHub repo. The part that is not automated is scripts required to make changes to the structure or content of the database. These are effected by Database Migration scripts, and need to be run manually by a developer from a command line terminal with the current directory set to the /server folder of the project on his/her local machine. The steps are:

1. Edit the /server/.env file to uncomment the connection string environment variables for the shared Azure database. Make sure that all other connection string information for other database environments are commented out (i.e., lines that start with #).
2. run the command `npm run flyway migrate` from the command line.
3. In some cases, the terminal will tell you it is downloading the flyway packages. If so, you will need to run the same command again to get the migrations to actually run. Otherwise, you should get messages indicating that the migration ran successfully, indicating that none were needed, or telling you how many scripts were run.  If any errors occur, the migration failed and you will need to debug why the migrations failed. If the commit does  include a database migration script, running the migration may or may not run the script, since the developer may or may not have run the script against the shared development database prior to creating the pull request.

A few minutes after the GitHub action is complete, you should be able to run the application at tdm-dev.azurewebsites.org and verify that the new changes are complete (by noting the description of the commit and verifying any observable change in behavior).

Developers who merge PRs should always check to see if a database migration is part of the PR and, if so, run the database migration. If the PR includes other code changes, the developer doing the merging should verify that the GitHub action runs without errors, to be sure that the container has been built and deployed.

Creating a Release to the UAT and Production Environments

Creating a release should only be done by a release manager! The deployment environment is similar to that for the Development Environment, but includes additional steps to

• Tag the code with a formal release number in GitHub (and visible on the About page)
• Publish the release notes
• Make sure that code changes made as part of the release (usually just updating the release number) are merged back into the develop branch

Release branches are created from the develop branch.

• Make sure your local machine has an up-to-date version of the develop branch:

git checkout develop
git pull

Decide on a release number for the next release, using semver conventions. For example, if the current release (as shown in the pacakge.json files in the /, /server and /client directories) is 1.0.0 and the changes in this release are minor, the new release number would be 1.0.1.

• Create a new release branch from develop with the name release-<release#>:

git checkout -b release-1.0.1

• Update the release number in the application. This typically entails updating the package.json files' version properties (in the /, /client and /server folders).

• Commit the version number change:

git add -A
git commit -m "Bumped version number to 1.0.1"

• Run the application (locally and/or in a deployment environment) and make any fixes necessary. These should be very minor changes - significant changes should be made by creating a feature release based on the develop branch as described above. When the application is ready for release...
• Merge the release branch into main:

git checkout main
git pull origin main
git merge --no-ff release-1.0.0
git tag -a 1.0.1 -m "Release version 1.0.1"
git push origin HEAD
git push origin 1.0.1

Committing code to the main branch like this will trigger a GitHub action which builds the UAT/Production Docker container and publishes it to the TDM Docker Hub repository. Both UAT and Production environments are hosted on a Microsoft Azure account that is owned and managed by the city. Lon Soh is the point of contact for information and requests related to the Azure account, but you generally will not need Access to this Azure account to perform a release. There are separate Azure App Services for UAT (tdm-uat.azurewebsites.org) and Production (tdm.ladot.lacity.org) sites, but both are configured to detect publication of a new release container of TDM and automatically deploy them.

• Go to the GitHub Actions Tab and verify that the container build completes successfully (this may take a few minutes.
• After completion of the GitHub action it may take several more minutes for the application to be deployed to Azure. You can verify this by launching the sites and verifying that the release number on the About page is updated with the new release number.

While the above actions are taking place, perform database migrations to both environments (each uses ar different database). To do this, you will need to have the connection string information for the UAT and Production databases, and have sections in your /server/.env file for each. The connection string information is highly sensitive, and therefore stored in the 1Password Vault "TDM Dev", so you will need to get this information and add it to the /server/.env file on your development machine:

1. Edit the /server/.env file to uncomment the connection string environment variables for the UAT database. Make sure that all other connection string information for other database environments are commented out (i.e., lines that start with #).
2. run the command npm run flyway:migrate from the command line.
3. In some cases, the terminal will tell you it is downloading the flyway packages. If so, you will need to run the same command again to get the migrations to actually run. Otherwise, you should get messages indicating that the migration ran successfully, indicating that none were needed, or telling you how many scripts were run. If any errors occur, the migration failed and you will need to debug why the migrations failed. If the commit does include a database migration script, running the migration may or may not run the script, since the developer may or may not have run the script against the shared development database prior to creating the pull request.
4. Edit your /server/.env file again, commenting out the UAT connection string variables, and uncommenting the Production connection string variables.
5. Run the command npm run flyway:migrate again from the command line. You should see the same migrations applied as you saw for the UAT environment.
6. Be sure to edit the /server/.env file to comment out the Production connection string variables and uncomment the connection string you would normally use (i.e., either to a local database or the shared development database).

The release is now done and tagged for future reference.

• Merge the release branch into develop:

git checkout develop
git merge --no-ff release-1.0.1

(Resolve any merge conflicts)

git push origin HEAD

It will automatically be published to https://tdm-dev.azurewebsites.net. Please be sure to run the application here and make sure your changes are reflected in this deployed version of the develop branch.

• We are now done with this release and can delete the release branch:

git branch -d release-1.0.1

Generating release notes

We are using gren with .grenrc.json config file and generating the log from Github Issues. See this page on the recommended convention for writing issue titles. We use the enhancement and bug labels to categorize issues for release notes.

First, go to the GitHub Project Board and look at the "On Dev - Not Yet Pushed to Production" column. Any development issues you find should have one of the following labels: enhancement, bug, system update, or refactor. If not, it will not be picked up for the Release Notes, so you want to add one of these labels to any development issues that do not already have one.

1. Follow instructions on the gren setup section to generate and install your Github token.

2. Be in the project root directory

3. Run the script to update the notes

npm run release-notes

1. This script is a bit finicky. The first time it is run, it associates ALL the historical release notes with the current release. To fix this, just run the same script again. When the script is complete, you should be able to see the correct release notes here
2. After the release notes look correct, go back to the project board and drag all the issues from the "On Dev - not yet pushed to Prod" column to the "Released" column.

Creating a HotFix

A HotFix is a quick fix to a release. In almost all cases, a full release is preferred, but it may be necessary if a small fix is necessary to the production release without including changes that have already been merged with the develop branch.

Creating a HotFix should only be done by the release manager! A HotFix should only include very minor patches to the application, and is always based on the main branch. The new release number will be the same as the main branch number with the patch number incremented by one. For example a patch to release 34.67.22 should be 34.67.23.

• Make sure your local machine has an up-to-date version of the main branch:

git checkout main
git pull origin main

• Create a new release branch from main with the name hotfix-<release#>:

git checkout -b hotfix-34.67.23 main

• Update the release number in the application. This typically entails updating the package.json file version properties, and perhaps other locations where the release number might appear (For now, I just added it to the About.js component, though we should probably put it in a site footer or some inconspicuous place, so it can be viewed from the UI.)

• Commit the version number change:

git add -A
git commit -m "Bumped version number to 34.67.23"

• Run the application (locally and/or in a deployment environment) and modify the code to implement the fixe(s). These should be very minor changes - significant changes should be made by creating a feature release based on the develop branch as described above. When the hotfix is ready for release...
• Merge the hotfix branch into main:

git checkout main
git pull origin main
git merge --no-ff hotfix-34.67.23
git tag -a 34.67.23 -m "Release version 34.67.23"
git push origin HEAD

At present, a github action is configured to detect the commit to main and automatically deploy the application to production. You should navigate to https://tdm-uat.azurewebsites.net after giving the Docker webhook time to deploy and verify that the application runs, and any visible release # has been incremented.

The hotfix is now done and tagged for future reference.

• Merge the hotfix branch into develop:

git checkout develop
git merge --no-ff hotfix-34.67.23

(Resolve any merge conflicts)

git push origin HEAD

It will automatically be published to https://tdm-dev.azurewebsites.net. Please be sure to run the application here and make sure your changes are reflected in this deployed version of the develop branch.

• We are now done with this release and can delete the release branch:

git branch -d hotfix-34.67.23

You should also regenerate the release notes as described in the Release Procedure above.