Skip to content

Latest commit

 

History

History
543 lines (348 loc) · 44.7 KB

RELEASE_PROCESS_OPENSEARCH.md

File metadata and controls

543 lines (348 loc) · 44.7 KB

OpenSearch Release Process

Preparation

Release Terminology and Knowledge Center

Definitions

OpenSearch Project: OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.

OpenSearch/OpenSearch Dashboards Bundle: An OpenSearch/OpenSearch Dashboards bundle refers to a compressed file that encompasses both the OpenSearch/OpenSearch Dashboards distribution and accompanying plugins. This compressed file acts as a container for the OpenSearch/OpenSearch Dashboards software and its associated extensions, enabling users to conveniently package and deploy the entire system.

OpenSearch/OpenSearch Dashboards Components/Plugins: OpenSearch/OpenSearch Dashboards components/plugins are extensions that can be added to OpenSearch/OpenSearch Dashboards, to add new features or functionality. There are a wide variety of plugins available refer OpenSearch available plugins and OpenSearch Dashboards available plugins sections.

Types of Manifests

Input Manifest

These manifests serve as the initial input for driving the build workflow. These manifests are located in build repository and are generated by the manifest workflow. Once the manifest is created, the maintainers of the build repository review and merge it into the main branch. The entire build repository operates exclusively on the main branch.

Test Manifest

These manifests are integral to the comprehensive testing of the core and components/plugins, as they contain numerous settings and configurations that facilitate various tests within the test_workflow.

Sample Test Manifests
OpenSearch OpenSearch Dashboards
opensearch-2.8.0-test.yml opensearch-dashboards-2.8.0-test.yml
Build Manifest

Output of the build workflow, used as input to the assemble workflow.

Sample Generated Build Manifests
deb rpm tar windows
os-arm64, os-x64 os-arm64, os-x64 os-x64, os-arm64 os-x64
Bundle Manifest

The final output of the assemble workflow and manifest that is added to the final distribution, this has the commitID (can be used for reproducible builds) and the artifact file links. This final bundle manifest incorporates the assembled components and ensures traceability through the commit ID and accessibility to the artifact files.

Sample Generated Bundle Manifests
deb rpm tar windows
os-arm64, os-x64 os-arm64, os-x64 os-x64, os-arm64 os-x64

AUTOCUT issues

These are the issues created by automation with the distribution build and integ-test workflows failure, the automation detects the component failure and raises an issue in the respective component repo. Sample integ-test failure AUTOCUT issue and distribution build failure AUTOCUT issue. The created AUTOCUT issues will have the updated information with latest build failure details, the automation also detects if the component build has passed and closes the issues automatically. For more details refer the closeBuildSuccessGithubIssue.groovy and createGithubIssue.groovy libraries part of the distribution build and integ-test workflows.

Build Workflows

Workflow Description
Check for Build Workflow that periodically triggers the distribution workflows using parameterized cron.
OpenSearch Distribution Build Workflow that is responsible to build/assemble the OpenSearch and its components.
OpenSearch Dashboards Distribution Build Workflow that is responsible to build/assemble the OpenSearch Dashboards and its components.
OpenSearch Integ Test Workflow that runs integ tests for OpenSearch and its components.
OpenSearch Dashboards Integ Test Workflow that runs integ tests for OpenSearch Dashboards and its components.
Benchmark Tests Workflow that runs Performance tests using opensearch-benchmark on a cluster created with a given version.
BWC Tests Workflow that runs backward compatibility tests on a cluster created with a given version.
RPM Validation Workflow that validates the RPM distribution
Docker Build Workflow that builds the OpenSearch and OpenSearch Dashboards docker images
Docker Copy Workflow that copies the created docker images to multiple DockerHub and ECR repositories
Docker Scan Workflow that checks vulnerabilities for a given docker image as an input.
1.x Maven Publish Workflow that published snapshot maven artifacts, used only for 1.3.x versions. For more details check this issue-319.

Release Workflows

Workflow Description
Release Notes Tracker Workflow that identifies if a component has a release notes added based on the commit history.
Promote Repos Workflow that signs and promotes the APT/YUM repos to the production buckets accessed via the cloudfront.
Promote artifacts Workflow that signs and promotes all the release artifacts to the production buckets accessed via the cloudfront.
Publish to Maven Workflow that signs and publishes to the central maven repository.
Docker Promotion Workflow that promotes the docker images to production docker repositories.
Validation Workflow Workflow that validates the released distribution.

Creating a New Version

Each new OpenSearch release process starts when any one component increments a version, typically on the main branch. For example, OpenSearch#1192 incremented the version to 2.0. The version check automation workflow will notice this change or it can be triggered manually, and make a pull request (e.g. opensearch-build#514) that adds a new manifest (e.g. opensearch-2.9.0.yml). After that's merged, a GitHub issue is automatically opened by this workflow to make a new release using this release template (e.g. opensearch-build#566). Existing and new components (re)onboard into every release by submitting pull requests to each version's manifest.

Release Manager

The release manager to a specific OpenSearch release will be assigned through volunteer model. The request for a release manager will be posted in OpenSearch public Slack #releases channel and selected on first come first served (FCFS) model. Note: The release manager should be a maintainer of a repo under OpenSearch GitHub organization.

Release Label

The release label creation is part of the version increment workflows running in the build repo OpenSearch Version Increment Workflow, OpenSearch Dashboards Version Increment Workflow. These workflows not only raise a pull request for a version increment but also verify if the release label exists for a given version. If it doesn't, they proceed to create it.

Release Issue

This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section.

Release Issue Update

The release issue is created by an automation workflow. Once the release manager is finalized, the release manager should be updating the created release issue. Sample Release Issue 2.8.0. Update the release issue so all __REPLACE_RELEASE-__ placeholders have actual dates.

Increase the Build Frequency

Increase the build frequency for the release from once a day (H 1 * * *) to once every hour (H/60 * * * *) in check-for-build.jenkinsfile. This will ensure the Distribution Build workflow is called every hour to build and detect the components failure early that are part of the Input Manifest.

Update the Maven Publish Workflow

This step is necessary solely for the 1.3.x release process; you can find more information in the issue. Please make sure to update and execute the 1.x Maven Publish workflow; you can find the workflow specifics in the Build Workflows section. Additionally, you can refer to a relevant sample PR.

Campaigns

This section is not required for a patch release.

Component Release Issues

The component release issues are auto created by the workflows part of the build repo OpenSearch components, OpenSearch Dashboards components. These workflows create the release issues based on the template component_release_template.md and links back the global release issue part of the build. Sample component release issues created for 2.10.0.

Issue Creation Process Overview

Inside the template component_release_template.md, replace the fields RELEASE_VERSION, RELEASE_BRANCH_X, RELEASE_BRANCH and RELEASE_ISSUE to desired release values before creating the release issues across the component/plugin repos. Once the fields are replaced, use the meta and gh cli to create the issues. Find the list of components/plugins from the opensearch-plugins repo (for OpenSearch, for OpenSearch Dashboards) and use the meta cli to create the release issues. For more details check the create-an-issue-in-all-plugin-repos section.

meta exec "gh issue create --label v2.8.0 --title 'Release version 2.8.0' --body-file /tmp/opensearch-build/.github/ISSUE_TEMPLATE/component_release_template.md"

Release Campaigns

If exists any release specific issues/campaigns, link it back to the release issue. Sample linked issues/campaigns

Release Branch Readiness

The Release Branch Readiness date is determined as the release date minus 4 days.

Release Branch

This release branch creation is not applicable for patch release.

Core

This step requires both OpenSearch and OpenSearch Dashboards to create a release branch that will be used for the release.

Components

This step requires that every team participating in a release has their release branch created for the corresponding release by the date listed on this step. The Distribution Build workflow will also start using the release branch to create release candidate instead of .x branches.

Version Increment

Versions are incremented as soon as development starts on a given version to avoid confusion. Following is the example that depicts the version increment.

  • OpenSearch: main = 3.0.0, 2.x = 2.9.0, and 2.8 = 2.8.1
  • common-utils: main = 3.0.0.0, 2.x = 2.9.0.0 and 2.8 = 2.8.1.0

Core Version Increment

To ensure the version incrementation process is handled correctly, it is important to follow to increment the version of the release branch. Currently, the version incrementation is being done manually by an individual from the core repositories. Sample OpenSearch Version Increment PR for the release branch.

Components Version Increment

The next step is to identify the pending component version increment pull requests. The creation of component version increment pull requests is automated with workflows running in the build repo OpenSearch Version Increment Workflow, OpenSearch Dashboards Version Increment Workflow. Sample component version increment PR.

The objective is to merge these pull requests in order to synchronize all the components with the core version. It is not as simple as submitting a pull request and having it merged. When a pull request is submitted, CI checks are triggered for each repository. These CI checks assess the dependencies. Thus, prior to merging the pull requests that involve version increments, the release manager must confirm that the dependencies have been updated, built, and pushed to Maven and S3. This crucial stage requires the release manager to coordinate with the component teams, address any dependency issues, and ultimately merge the pull requests that involve version increments.

Code Complete and Feature Freeze

Coordinate with the Core and component teams to ensure that the code for this particular release version is fully prepared and that the corresponding branch has been included in the release version Input Manifest. Update Jenkins workflows that execute daily snapshot builds for both OpenSearch and OpenSearch Dashboards (Ref Increase the build frequency). Submit pull requests to incorporate each component into the respective version level Input Manifest along with the necessary checks. Sample PR. The Feature Freeze date and the Code Complete date is determined as the release date minus 4 days.

Release Candidate Creation and Testing

Distribution Build

Refer the Build Workflows section to get the details about the distribution build job for OpenSearch and OpenSearch Dashboards. This section covers how to sequentially handle the distribution build job during the release.

Workflow Trigger

Ensure the proper inputs are used to initiate the distribution. For instance, here's an example for OpenSearch. It's important that the OpenSearch Dashboard distribution aligns accordingly.

Alt Text

COMPONENT_NAME: To trigger a specific component, this includes standalone OpenSearch or specific plugin.

INPUT_MANIFEST: The release input manifest that drives the workflow.

TEST_MANIFEST: The release test input manifest that is used for the integ tests.

INTEG_TEST_JOB_NAME: The integ test job name. Default already added to this input integ-test for OpenSearch and integ-test-opensearch-dashboards for OpenSearch Dashboards.

BUILD_PLATFORM: The input used to build for a specific platform, followed by its own distributions within the platform.

BUILD_DISTRIBUTION: Input to build selected distribution related artifacts, choices include 'tar', 'rpm', 'deb', 'zip'. Can combine multiple distributions with space in between (docker is only available on tar).

BUILD_DOCKER: Input with a dropdown that has 3 options build_docker, build_docker_with_build_number_tag, do_not_build_docker, the release manager has to take a call with right inputs.

UPDATE_LATEST_URL: To update the /latest CFN URL, Visit latest-distribution-url for more details.

Order of Execution

Following is the order of execution of the distribution build to address the components dependencies.

Note: The execution order specified is necessary only for versions up to 1.3.x. For 2.x and above Maven dependencies are published through each component repository using the GH workflow. For more details check the META issue.

OpenSearch
OpenSearch 
OpenSearch + common-utils + job-scheduler 
OpenSearch + common-utils + job-scheduler + performance-analyzer 
OpenSearch + common-utils + job-scheduler + performance-analyzer + security 
OpenSearch + common-utils + job-scheduler + ml-commons + performance-analyzer + security 
All components (which are ready after completion of version increment)
OpenSearch Dashboards
OpenSearch Dashboards
All components (which are ready after completion of version increment)

Release Candidate

Now once all the version increment PRs are completed and all the components are part of the input manifest, now it's time to generate the RC. Use the Distribution Build to generate the release candidate. Use the following section as the reference to generate the RC, validate it and broadcast it for a given release. The process of Release Candidate Generation and Testing should commence at least 6 days prior to the release date.

Sample Build details

Following is the generated build number after triggering the Distribution Build workflow. The distribution build number denotes the RC, now with the example below the finalized RC’s are OS: 7848, OSD: 6126.

OpenSearch OpenSearch Dashboards
build_7848 build_6126

Integ Test TAR

Following are the TAR integ test jobs for a given RC based on the above section Sample Build details. The integ tests are executed for artifacts generated as part of the builds OS: 7848, OSD: 6126.

Integ Test Tar x64 arm64
OpenSearch x64 arm64
OpenSearch Dashboards x64 arm64

Integ Test RPM

Following are the RPM integ test jobs for a given RC based on the above section Sample Build details. The integ tests for rpm are executed for artifacts generated as part of the builds OS: 7848, OSD: 6126.

Integ Test RPM x64 arm64
OpenSearch x64 arm64
OpenSearch Dashboards x64 arm64

Docker Build and Scan

Following are the details for the docker image build and scan. The docker images are built using the TAR artifact generated as part of the Distribution Build (From the above example OS: 7848, OSD: 6126). The Distribution Build workflow with input BUILD_DOCKER (Ref Workflow Trigger) triggers the [docker-build] workflow as downstream.

Docker build scan
OpenSearch Build Scan
OpenSearch Dashboards Build Scan
Docker RC Freeze

This to ensure that check-for-build.jenkinsfile won't re-build periodically and override the docker, the RC docker is created with build number. This step can be skipped if the input BUILD_DOCKER: build_docker_with_build_number_tag (Ref Workflow Trigger used in the Distribution Build).

Docker Freeze copy
OpenSearch docker-copy
OpenSearch Dashboards docker-copy

Benchmark Tests

For running the benchmark tests, use the benchmark-test job part of the Build Workflows. For more details in running the benchmark tests refer Benchmarking Tests section part of the test workflow. This job offers multiple options to test the performance of a specific version cluster using various metrics. The benchmark performance results can be accessed via the OpenSearch Performance Benchmarks dashboard. Sample benchmark tests for 2.9.0 release.

Backwards Compatibility Tests

For more details in running the BWC tests refer Backwards Compatibility Tests section. On board the components/plugins to the test Test Manifest with the bwc-test setting. Example as follows:

  - name: index-management
    bwc-test:
      test-configs:
        - with-security

Windows Integration Test

Currently, the windows integration tests for a release is manual. The manually tested windows zip is being evaluated and approved by the plugin teams for sign off. In order to test the windows distribution, two instances need to be created: one with security features enabled and another without security. Afterward, API calls should be tested by launching the OpenSearch and Dashboard processes through direct execution of the .bat file.

Broadcast and Communication

Broadcast the release candidate in OpenSearch public slack workspace and the release GitHub issue using format sample broadcast message to gather votes.

As a release manager, it is essential to ensure the successful completion of all the above mentioned jobs. In the event of failures during integration tests or scans, the release manager should collaborate with the component teams and initiate a re-run to ensure that all jobs are executed successfully.

Post all the job related failures in the Release issue, Sample post.

Note: Sometimes the integ-test jobs are flaky and might not pass due to several reasons with the component code, in that case coordinate with the respective component team and get a manual sign off. Sample manual sign off. All the failed logs are in s3 accessed through the cloudfront. Sample link.

Release Candidate Lock

Stop builds for this version of OpenSearch and/or OpenSearch Dashboards in order to avoid accidental commits going in unknowingly. Restart only if necessary, else manually run the build workflow and declare new release candidate.

Once the RC is finalized, in order to exclude the release from running periodically, at this point it is necessary for the release manager to lock the input manifest and update the check-for-build.jenkins to remove it from the scheduled execution, sample PR.

Release

Pre-Release

Ensure that all pre-release activities listed below are completed before proceeding with the final release. These activities hold great significance as they determine whether the release should progress further and, if approved, they contribute to saving a substantial amount of time during the release window.

Release Labeled Issues

Verify all issues labeled with this release have been resolved. Coordinate with the core/components team to close the gaps in resolving the issues labeled with this release.

Go or No-Go

Get the Go / No-Go votes from project management committee (PMC) before staging the release artifacts for production publishing process,

Promote Repos

Repo OpenSearch (Sample Runs) OpenSearch Dashboards (Sample Runs)
YUM yum-os yum-osd
APT apt-os apt-osd

Promote Artifacts

Artifacts OpenSearch (Sample Runs) OpenSearch Dashboards (Sample Runs)
Windows os-windows-zip-x64 osd-windows-zip-x64
Debian os-deb-arm64, os-deb-x64 osd-deb-arm64, osd-deb-x64
TAR os-tar-arm64, os-tar-x64 osd-tar-arm64, osd-tar-x64
RPM os-rpm-arm64, os-rpm-x64 osd-rpm-arm64, osd-rpm-x64

Release Notes

Coordinate with the plugin teams and create a consolidates release notes. Sample PR. Release manager can check if a plugin team has created a release notes or not using the release notes tracker tool. Sample run.

Main Release

Release the artifacts to production distribution channels, update the website and inform the community of the release.

Maven Promotion

Promote OpenSearch to maven, trigger the publish-to-maven workflow (Ref Release Workflows), sample run.

Docker Promotion

Publish the images to docker and ECR, trigger the docker promotion workflow (Ref Release Workflows), sample run.

Collaboration with the Project Management Team

While not directly under the responsibility of a Release Manager, the following sections require coordination with the Project Management Team to ensure that the tasks are successfully fulfilled.

Website and Documentation Changes

Coordinate with the documentation website team to ensure the changes are in place and close the gaps in promoting the website changes to prod. At this point the maintainers of the repos documentation-website and project-website will handle this task.

Publish blog posts

Sample Blog Post

Advertise on Social Media

Coordinate with the project management team to ensure the social media advertisement is completed.

Release Validation

Use the validation workflow (Ref Release Workflows) to validate the published artifacts, sample validation workflow run.

Release Checklist

Please update the checklist either in the release issue body or as a new comment to the release issue. By following and updating the release checklist, we can ensure the success of the release. Sample release checklist for 2.9.0 release.

Post Release

Once the release is completed following are the activities that needs to be completed by the release manager:

Release Tags

Create release tags for each OpenSearch and Dashboard components. Sample OpenSearch, OpenSearch Dashboards workflow runs.

Input Manifest Update

Replace refs in input manifest with tags. The refs can be identified from the bundle or build manifest, sample PR that updates the input manifest.

OpenSearch Build Release notes

Generate distribution release notes for opensearch-build repository, sample 1.3.10 release details.

Decrease the build frequency

Lower the frequency of builds for the release version of OpenSearch and/or OpenSearch Dashboards, sample PR.

Retrospective Issue

Create an issue for a retrospective, solicit feedback, and publish a summary. Sample retro issue.

Helm and Ansible Playbook release

Update and release the Helm chart and ansible playbook with the new OpenSearch and Dashboard version. Sample helm chart PR and ansible PR.

Upcoming Release Preparation

The release manager for the current release should ensure that a release manager is assigned for the upcoming release. This can be achieved by coordinating in a Slack channel and @opensearch-project/engineering-effectiveness team should assist in assigning the user to the upcoming Release issue.

Communication Templates

Please utilize the provided communication templates to effectively coordinate with the teams involved in the release process. These templates are designed to assist you in communicating through the GitHub release issue and the public Slack #releases channel.

Release Announcement

We are starting the process of <RELEASE_VERSION> release.
Release Issue: <RELEASE_ISSUE>
Release Manager: <RELEASE_MANAGER>

Kindly review the following information provided below regarding the release dates:

Release Branch and Version Increment:  <REPLACE_RELEASE-minus-14-days>
Feature freeze: <REPLACE_RELEASE-minus-12-days>
Code Complete: <REPLACE_RELEASE-minus-10-days>
RC creation and : <REPLACE_RELEASE-minus-6-days>
Pre Release: <REPLACE_RELEASE-minus-1-days>
Release date: <RELEASE_DATE>
Post Release: <RELEASE_DATE>

Release Readiness

The version increment pull requests are still awaiting attention. We kindly request the plugin/component owners to review them and make an effort to merge them promptly.

OpenSearch: https://github.com/pulls?q=is%3Aopen+is%3Apr+org%3Aopensearch-project+increment+<RELEASE_VERSION>+in%3Atitle+

OpenSearch Dashboards: https://github.com/pulls?q=is%3Aopen+is%3Apr+org%3Aopensearch-project+increment+<RELEASE_VERSION>.0+in%3Atitle+

Release Candidate Announcement

Refer the sample rc announcement from past 1.3.10 release.

Release Complete

OpenSearch 2.8.0 version has been released to public :tada:!

Thanks everyone for the help to bring <RELEASE_VERSION> release out.
Component repo owners please create a github release based on the tags of <RELEASE_VERSION.0>.
Presenting the retrospective <RELEASE_RETRO_ISSUE_LINK> for the release. Please feel free to provide your valuable feedback for further improvements in the upcoming release.