Release Procedures

!!!!!!!!! Copy and pasted from JWST. Needs editing to be STCAL specific. !!!!!!!!!

We assume you have a local clone of your fork of https://github.com/spacetelescope/jwst and that origin points to your fork and upstream points the central repo. I.e. the following (or ssh equivalents):

$ git remote -v
origin	https://github.com/jdavies-st/jwst.git (fetch)
origin	https://github.com/jdavies-st/jwst.git (push)
upstream	https://github.com/spacetelescope/jwst.git (fetch)
upstream	https://github.com/spacetelescope/jwst.git (push)

Prepare master branch for release

These steps should be undertaken on the master branch:

Update CHANGES.rst to have the correct release version and date. Change "unreleased" next to the intended release to the current date in YYYY-MM-DD format.
Update README.md to show the latest release version in the "Software vs DMS build version map" and be sure to update the date and CRDS_CONTEXT this release is being currently tested with.
Create a PR against upstream/master and merge it.

Prepare the release branch

If you're making a major or minor version release, then the release branch will not yet exist. If you're releasing a patch version, then a release branch will already exist and patch may include everything on master or need cherry-picking. Following one of the next 3 options accordingly. The following steps assume that the spacetelescope/jwst remote is named upstream, just as above.

Option 1: Major or minor version release

Update pointer to master:

$ git fetch upstream
$ git checkout master
$ git pull upstream master

Verify that the last commit in the log is as expected:
```
$ git log
```
Create a new release branch. The name of the release branch should share the major and minor version of your release version, but the patch version should be x. For example, when releasing 0.16.0, name the branch 0.16.x.
```
$ git checkout -b a.b.x upstream/master
```
Push the branch to the upstream remote.
```
$ git push -u upstream HEAD
```

Github actions CI should notice the updated branch and run the tests which you can check here. Wait until the build passes before proceeding.

Option 2: Patch release (including all of `master`)

If your patch release will include everything on master, the procedure is very similar to a major/minor release, but the release branch will already exist. So if you need to deliver a 0.16.1 patch to 0.16.0, then you will do your work in branch 0.16.x.

Checkout and freshen release branch (this assumes that your local branch is already tracking upstream/a.b.x):
```
$ git checkout a.b.x
$ git pull
```

Pull all changes on master into the release branch:

$ git fetch upstream
$ git pull upstream master

Push updates to the upstream remote:
```
$ git push upstream HEAD
```

Github actions CI should notice the updated branch and run the tests which you can check here. Wait until the build passes before proceeding.

Option 3: Patch release (requiring cherry-picking from `master`)

In the case of a patch release, the release branch will already exist. So if you need to deliver a 0.16.1 patch to 0.16.0, then you will do your work in branch 0.16.x.

Checkout and freshen release branch (this assumes that your local branch is already tracking upstream/a.b.x):
```
$ git checkout a.b.x
$ git pull
```
Cherry-pick relevant commits from master that should be included in the patch release (including the new changelog commit):
```
$ git cherry-pick ...
```
Push updates to the upstream remote:
```
$ git push upstream HEAD
```

Github actions CI should notice the updated branch and run the tests which you can check here. Wait until the build passes before proceeding.

Cherry-picking tips

A bug needs to be fixed in 0.16.0. It is also present on master.

Fix the bug on master through a PR. Merge it and get the hash of the commit. As an example, let's say the commit hash is f41120e.

Cherry pick all commits that need to be included in the patch release.

$ git cherry-pick -x f41120e

If there are conflicts resolve them before you continue with other commits. In general start with the oldest commit so that conflicts are minimized.

Sometimes a commit is a merge commit with more than one parent. The above command will fail with an error message:

error: Commit f41120e is a merge but no -m option was given.
fatal: cherry-pick failed

This discussion is helpful:

In this case look at the parents and choose which one is the ancestor. The command then is

git cherry-pick -x -m 1 f41120e

This tells git to use the first parent.

Use git show to verify what changes are going in the branch.

Github actions CI should notice the updated branch and run the tests which you can check here. Wait until the build passes before proceeding.

Create the release tag locally

At this point, you should have the release branch checked out and ready to tag.

Create an annotated tag with a name that matches your intended release and a useful message. For our purposes, having the DMS Build number that this tag goes with is the current useful message:
```
$ git tag -a a.b.c -m "JWST DMS Build x.y"
```

Publish to test.pypi.org

Before proceeding, you will need to pip install twine build.

Checkout the release tag, clean the directory, and make sure umask and permissions are set correctly:
```
$ git checkout a.b.c
$ git clean -xdf
$ umask 0022
$ chmod -R a+Xr .
```
Check the package setup and create the package sdist and wheel:
```
$ python -m build .
```
Upload the package to PyPi's test server (you need an account and be added as maintainer):
```
$ twine check --strict dist/*
$ twine upload --repository testpypi dist/*
```
Check that it looks good on the test server. Make sure it installs without errors in a new virtual env:
```
$ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple jwst[test]
```
Run the tests on the installed package. Change to a directory that does not contain the jwst source and confirm that tests pass on the installed package.
```
$ pushd /
$ pytest --pyargs jwst
$ popd
```
If the package looks good on test.pypi.org and installs OK and the tests pass, then proceed.

Make the release official on Github and Pypi

If publishing to the PyPi test server above worked well, push the new tag to the upstream remote on Github:
```
$ git push upstream a.b.c
```
Go to https://github.com/spacetelescope/jwst/releases and draft a new release with the existing a.b.c tag used above. Use a release title "JWST Build x.y rcN" where N is the current release candidate for DMS, and include the changelog. We can always remove the "rcN" part from the title later if, say, rc3 turns out to be the final release. When saved, this will kick off the Github action to deliver the package to Pypi officially.
Check that it was delivered to Pypi (https://pypi.org/project/jwst/).

Maintain the stable branch (if necessary)

The stable branch points to the latest official release of jwst. If the current release has become the latest, then the next step is to rewrite the stable branch to point our new tag.

$ git checkout stable
$ git reset --hard a.b.c
$ git push upstream stable --force

Make the official release to DMS

Run the Jenkins JWSTDP release job

This assumes that the build artifacts from the passing regression tests have been produced here

Just put the tag a.b.c in the top box and hit the BUILD button. Add yourself to the "notification_recipients" if you're not already there. If there are any problems, contact Matt Rendina.

If the regression tests have not passed, you will have to trawl jwcalibdev and boyle (currently) to get the linux and MacOS build artifacts. Current locations (which get wiped daily) are
```
jwcalibdev:/data1/jenkins/workspace/RT/JWST/clone/*stable*.txt
boyle:/users/iraf/build/boyle/workspace/RT/JWST/clone/*stable*.txt
```
And push these up to Artifactory at
```
https://bytesalad.stsci.edu/artifactory/jwst-pipeline-results/
```
And then run the Jenkins JWSTDP release job as above.

The release job will edit the requirements files so that jwst will have the specified tag (instead of the git hash from the regression tests run), and it will populate the new release here.

And it will send out an email to "notificaiton_recipients" listed in the Jenkins JWSTDP release job.
If this is the final release for a build, an announcement email should be sent that includes all of JWST DMS, INS, and Mission Office. This email should include the link to the SDN documentation and installations for how to install the specific build.

Prepare master for further development

Create and merge a PR to master updating CHANGES.rst, opening it up for new features and bugfixes. The diff should look something like this

$ git diff
diff --git a/CHANGES.rst b/CHANGES.rst
index e8771c7b..d7fac87e 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -1,3 +1,8 @@
+0.16.1 (unreleased)
+===================
+
+
+
0.16.0 (2020-05-04)
===================

Tag master for further development (if necessary)

If we're not already working on a later release on master (i.e. if we had to cherry-pick from master onto our release branch), then we'll need to tag the HEAD of master with a development tag. This allows the setuptools-scm plugin to identify code installed from master as the latest version. I.e. this only really needs to be done if we've done a patch release on a release branch that has cherry-picked commits on it.

$ git fetch upstream
$ git checkout upstream/master
$ git tag -a a.b.d.dev -m "Tagging a.b.d.dev"
$ git push upstream a.b.d.dev

Update artifactory regression test data

Go to the Artifactory web interface and copy the passing test input and truth data from

 jwst-pipeline/dev

to a new directory named the same as the release tag

 jwst-pipeline/a.b.c

images/bytesalad_copy_dev.png

images/bytesalad_copy_dev2.png

CRDS-related Updates

CRDS maintains a reference file, CALVER, and metadata that are tied to JWST software releases. Review the two substeps that are part of the CRDS S/W Release Process. If either require updates due to a JWST release, file a CCD issue to update CRDS as needed. The two subseps are:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release Procedures

Prepare master branch for release

Prepare the release branch

Option 1: Major or minor version release

Option 2: Patch release (including all of `master`)

Option 3: Patch release (requiring cherry-picking from `master`)

Cherry-picking tips

Create the release tag locally

Publish to test.pypi.org

Make the release official on Github and Pypi

Maintain the stable branch (if necessary)

Make the official release to DMS

Prepare master for further development

Tag master for further development (if necessary)

Update artifactory regression test data

CRDS-related Updates

Clone this wiki locally

Release Procedures

Prepare master branch for release

Prepare the release branch

Option 1: Major or minor version release

Option 2: Patch release (including all of master)

Option 3: Patch release (requiring cherry-picking from master)

Cherry-picking tips

Create the release tag locally

Publish to test.pypi.org

Make the release official on Github and Pypi

Maintain the stable branch (if necessary)

Make the official release to DMS

Prepare master for further development

Tag master for further development (if necessary)

Update artifactory regression test data

CRDS-related Updates

Clone this wiki locally

Option 2: Patch release (including all of `master`)

Option 3: Patch release (requiring cherry-picking from `master`)