Develop consistent method for capturing release notes across all PDS repositories

[jordanpadams]

💡 Description

Looking to standardize some common way for we, the PDS, should capture release notes for releases of software components.

Right now, we use the CHANGELOG as our primary view of what has changed, but that tends to be too technical/granular for the everyday user. We need some simple, manually curated release notes to describe in more user-centered terms what is new, what is fixed, and any backwards incompatibilities. Let's use this issue to throw out some ideas, and we can talk at the 2/3/22 breakout. What I am looking for:

(1) common mechanism for capturing these notes (some file in the repo, a page in the online docs, etc.)
(2) what do the release notes actually look like? - a consistent format for how we present this to this to the user

Let's start the discussion on (1), and then we can switchover to (2) once we have some ideas. If folks know of good best practices out there for this, that would be ideal so we are not re-inventing the wheel.

Some ideas to get us started:
a. Add a RELEASE_NOTES.md at the top of every repo that we link to from our documentation.
b. Use Github discussion threads - I did this for a validate release to just see how it might work
c. Include the notes in the actual tagged release on Github - You can actually put a custom description on these release pages

(a) seems like the easiest solution with the least amount of effort. we would just need to add this to our procedures for our "release engineer". (b) or (c) would be nice, but would require more manual clicking through the repos and/or update to roundup

[kgrimes2]

I've seen (c) used the most. It forces teams to distill commit messages into a few bullet points, explaining at a high-level what was changed.

One disadvantage of using release notes, is that it's somewhat hidden away in Github. It may make sense to update the release notes, but also to maintain a separate markdown document, linked to from the README.

[jpl-jengelke]

Usually the biggest challenge in release notes that I've experienced is with content. Someone actually needs to input the data regarding the changes somewhere, whether it is in commit messages or tickets.

First of all, "release notes" I've seen for Lab projects have been markdown documents at the top-level of the repository named CHANGELOG.md, while the README.md has been for specific information about application usage or maintenance. So CHANGELOG.md has been of the form of a date/version heading followed by a bulleted list of fixes and enhancements. Actual release notes can also be composed from the CHANGELOG.md document but those are usually longer and more expository, in my experience.

Are there automated ways to generate the bulleted list? Yes, but there will be some level of (perhaps minimal) added developer commitment. We know about the hard way (reliance on one expert) which generates notes manually by relying on a single person creating a mini report of relevant tickets. Conversely, an automated way is to rely on the team with just a little more commit information. A Git commit hook requires ticket numbers to be inserted into any code commit message (excluding merge or release). At release time a script is run that collects commit messages, parses them for (required) ticket information and collects detail from the Jira/GitHub ticketing API. Then notes are composed using ticket titles, or perhaps the first sentence of the description, or anything in the ticketing system. Another very similar option is simply collect the first 100 chars of any commit message without querying a ticketing API for detail.

[jordanpadams]

@jpl-jengelke my concern with commit messages is a few things:

1. The audience - our customers don't care about what changed in the code or what this particular commit message is doing, all they want to know there is some new functionality that you can use with this new flag.
2. Too much curation of commit messages, not enough pushing to GitHub - I tend to prefer commits and pushes to repos early and often to ensure traceability to changes in the code for debugging purposes. Relying on commit messages for user-centered functionality descriptions will remove this paradigm and code will start getting committed in huge chunks.
3. That all being said, we could leverage the PR messages since they are what is put into the merge request, but even then our CI pipeline adds a change log to the commit history as soon as we merge to main...

[ramesh-maddegoda]

I like the suggestion of @jpl-jengelke on "A Git commit hook requires ticket numbers to be inserted into any code commit message". I have seen successful uses of this method to capture the traceability from the requirements (user stories) to implementation.

I also agree with the concerns of @jordanpadams on commit messages. I think instead of the commit messages, we can pickup the high-level user stories (issues) as follows.

Some suggestions:

1. Use customer friendly Issue Titles, that look good enough to directly include in the release notes.
2. Include the issue number in all commit messages (enforced by Git commit hooks to avoid mistakes).
3. Find the issue numbers delivered in a release bundle using the commit messages
4. Filter only the issue numbers, that are closed (where issue status = closed)
5. Include the list of closed issue titles in the release notes.

[jpl-jengelke]

@jordanpadams wrote:

my concern with commit messages is a few things:

2. Too much curation of commit messages, not enough pushing to GitHub

I totally get the concerns, and we spent a lot of time discussing this approach. In the end, though, it wasn't a big burden. Once a ticket number was required in commit messages, it became second nature. If a general commit was needed for something like code cleanup, devs had a generic ticket number on hand. In the end it boiled down to typing a #<number> into the commit message. And, overall, a ticket should reflect any meaningful change. But yes it's something extra and new that initially felt like overhead.

[jordanpadams]

@ramesh-maddegoda @jpl-jengelke thanks for the feedback.

I guess to rephrase the concern a bit, we pretty much do something like this already and our users don't find it helpful.

Our changelogs are auto-generated for ever merged PR, and every PR should (and most the time does) include the issue number. if not, we are usually pretty good about going back to the issue and referencing the specific PR or commit when we close the issue.

Here are some examples:

https://github.com/NASA-PDS/validate/blob/main/CHANGELOG.md
https://github.com/NASA-PDS/harvest/blob/main/CHANGELOG.md
https://github.com/NASA-PDS/deep-archive/blob/main/CHANGELOG.md

Additionally, we have the added complexity of issues crossing repo boundaries. We could always ensure we have a ticket in the repo we are committing to, but then replicating tickets isn't necessary ideal either.

I guess my point is, if we go with issues or commit messages, I'm not sure what we do better than we already do?

For the issue titles, we could create different issues titles that talk about what is actually done, versus the requirements, but then we lose the ability to autogenerate requirements from our issues (which almost an equally laborious manual task).

[jordanpadams]

That being said, maybe we do just need to change the way we are managing our requirements...

[tloubrieu-jpl]

@jordanpadams is the usage of the release description in the github form implied by one of your proposed options ?

There is a also an 'auto-generate description' button which I think is based on pull requests but I did not manage to make that work.

For the comments before, to be honest I am not paying a lot of attention to the generated changelog. Maybe we should validate them more carefully when we create a release if they are not too far from what the users expect. The issue then is going to be that if we want to adjust the tickets titles or presence (harvest does not have any ticket for release 3.5.2) we would have to regenerate the changelog multiple time manually for the same release which is not easy with the current CICD.

[jordanpadams]

@jordanpadams is the usage of the release description in the github form implied by one of your proposed options ?

@tloubrieu-jpl yes. I kind of meant that by (c). The auto-generate description won't really work because our last push to the repo is almost always the changelog updates from the CICD

[nutjob4life]

I don't know if this is something that can or even should be automated. Maybe how they're stored or presented, sure. But their content? Nah.

Good release notes should assuage fears about changes to things they like, detail what's changed since last time, provide motivation to update, give only those technical detail so folks can make a mental picture of what the effort's going to be. They should be grouped so people can gravitate to the parts that are relevant to them, such as "new features", "bug fixes", "security updatess", etc.

On top of it all, they: should be a joy to read: sufficiently engaging to keep the interest, brief enough to get people through them all, and a hint of playfulness to make the entire ordeal seem not so bad.

"Playful has no place in software."

Really? If that were the case, we wouldn't have the Google Doodle!

This doesn't mean every bullet item in the release notes has to be a joke. But having a little humor in at least one of them can help make the medicine go down.

Because of all this, I don't see a good way to automate it. I'd have a good technical writer look at the changelog for a release and turn that into a decent set of release notes.

Example Release Notes

What follows are excerpts of actual release notes I've read recently that I think stand out as good examples.

Example Release Notes for 1Password on MacOS

Today's release brings a number of improvements to file attachments for 1Password accounts. We've also made a number of improvements to browser integration, making sure 1Password is filling fields as deftly as possible.

Improvements

• Improved the file attachment interface so that a newly added item shows up immediately and displays progress uploading the file.
• Attachments now maintain a consistent order between view mode and edit mode. {#5555}
• Attachments are now included during export. {#5530}
• Improved iCloud export for one-time passwords so that we export a proper otpauth URL when we only have a raw secret value in 1Password. {#5575}
• Added a dismiss button to the migration offer and a way to bring it back via Help menu. {#5562}

Fixed

• Fixed an issue that caused detached windows to errantly claim to be focused all the time.
• Fixed an issue that prevented the share item sheet from being presented on detached windows.
• Fixes a possible crash on launch relating to periodic downloads.
  Fixed a crash that could occur when running in an unregistered state. {#5565}

Example Release Notes for Slack on macOS

• Those on MacOS 12 reported that notification sounds were still coming through despite Notification Sounds being set to “None.” We have updated the internal logic to the more explicit “None, not under any circumstances, not even if it’s very important, never, not once, nil, absolutely not.” We hope it takes the hint.
• Some external URLs were occasionally launching within the Slack app window, but will now load in your default web browser, as is proper. Slack cannot have a little web page, even as a treat.
• We’ve tinkered with the internal workings and polished some rough edges. The app is now better than it was.

Example Release Notes for ProductPlan

See 'em here: https://www.productplan.com/release-notes/

[jpl-jengelke]

@jordanpadams wrote:

Additionally, we have the added complexity of issues crossing repo boundaries. We could always ensure we have a ticket in the repo we are committing to, but then replicating tickets isn't necessary ideal either.

Just to throw this out there, tickets can be cross referenced from any repo. The syntax is this:
NASA-PDS/software-issues-repo#20 which is demonstrated below ...
#20

[jordanpadams]

proposing release notes in release descriptions. started a release guide here: https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Release-Notes-Guide