This document describes steps to onboard a new plugin to release workflow for continuous integration and testing.
Add the new plugin to the opensearch-plugins meta, e.g. opensearch-plugins#97, which added cross-cluster-replication.
-
Update a manifest for a particular release to include your plugin. For example to be included in the 1.1.0 release, you would update opensearch-1.1.0.yml. We require your plugin name, repository URL, and git ref that should be used. For unreleased versions this should be a branch in your repository. Once a release is cut, these refs will be updated to build from a tag or specific commit hash.
-
Create a
scripts/build.sh
if you have specific requirements that are not covered by the default build.sh script and commit it to your repository. -
Ensure your
build.sh
reads and passes along both-Dbuild.snapshot=
and-Dopensearch.version=
flags. Snapshot builds should produce a -SNAPSHOT tagged artifact for exampleopensearch-plugin-1.1.0.0-SNAPSHOT.zip
where a release build of the same component would produceopensearch-plugin-1.1.0.0.zip
. -
Execute
./build.sh
to ensure your component builds and all artifacts are correctly placed into ./artifacts/ with correct output names. -
Execute
./assemble.sh
to ensure the full bundle is assembled and placed in to /bundles/*.tar.gz. Unpack the tarball to ensure all your components are placed in their correct locations. -
Publish a PR to this repo including the updated manifest and the names of the artifacts being added.
-
Update the test configuration file (use 1.3.0 as an example), opensearch-1.3.0-test.yml, for a particular release, to include your plugin. This test configuration defines full suite of tests -
integ
,bwc
, that can be run on the plugin. -
For integration testing, the
test-workflow
runs integration tests available in the plugin repository. You will need to addinteg-test
config for your plugin in opensearch-1.3.0-test.yml, example.-
It supports two test configs -
with-security
andwithout-security
, which runs test with security plugin enabled and disabled respectively. Choose one or both depending on what your plugin integration tests support. -
If your plugin is dependent on
job-scheduler
zip, you can define that inbuild-dependencies
in the config. Currently, the test workflow only supportsjob-scheduler
as build dependency. Please create an issue if your plugin needs more support.
-
-
For backward compatibility testing, the
test-workflow
runs backward compatibility tests available in the plugin repository, (see reference). Like integration test, it has a set of configurable options defined in opensearch-1.3.0-test.yml, example.- It supports two test configs -
with-security
andwithout-security
, which runs test with security plugin enabled and disabled respectively. Choose one or both depending on what your plugin integration tests support.
- It supports two test configs -
This document describes steps to onboard a new component to universal or 1-click release process.
See opensearch-project#1234 for details about end to end workflow.
- Please ensure that opensearch-ci-bot has the write access to your repository. If not, request by creating an issue in this repository.
- Add a webhook token as credentials to CI system using configuration as code. To generate the token, use command
head -30 /dev/urandom | sha512sum | awk '{print $1;}'
that generates random alpha-numeric string. - Create a Jenkins workflow that utilizes one of the build libraries to publish the artifacts to right platform. Please check the library requirements and retrieval methods before using it.
- For publishing to a new platform (other than ones specified above) a new library needs to be added. (ETA: 2 weeks)
- Release Drafter: Release drafter is a GitHub Action workflow that drafts a release that may or may not contain the release artifacts. The drafted release acts as a trigger to the Jenkins workflow. It also acts as a staging environment for release artifacts. This is to make sure the build environment remains the same even for release artifacts. Example
- 2 Person Review It is highly recommended to add 2 PR approval for any release workflow. In universal release process this can be added to release-drafter workflow as that is the starting point to trigger any release. See how to add the mechanism in the workflow. The mentioned solution creates an issue that notifies and assigns the reviewers. Example.
- Jenkins Workflow: Once the Jenkins workflow is added to the repository, onboard the workflow to publicly available CI system
- Create a
New Item
- Name it
<component-name>-release
- Select
Pipeline
as type of the project - Hit
Ok
and scroll to the bottom of the page - Select "Pipeline script from SCM" under Pipeline section
- SCM: GitHub repository link. eg: https://github.com/opensearch-project/opensearch-build
- Script Path: Relative path to jenkins file. eg: jenkins/check-for-build.jenkinsfile
- Run the workflow once in order to update the configuration of the Jenkins Workflow. You can abort once the workflow starts pulling the docker image.
- Create a
- GitHub Webhook: Add webhook to your GitHub repository by going to repository settings → Click
Webhooks
:Key Value Payload URL https://build.ci.opensearch.org/generic-webhook-trigger/invoke?token=tokenAddedInStep2 Content type application/json SSL verification Enable Which events would you like to trigger this webhook Releases. Please ensure to deselect the default option "Pushes" Active Enable - Once a webhook is added, it will send a ping to check the connectivity (✅). You can check the ping by going to repository settings → Webhooks → Click on
Recent Deliveries
tabs - Add
RELEASING.md
file to the repository documenting how to release the artifact. Example - Adding tests: Each library has a respective library tester associated with it that can be used to test you jenkins workflow. This tests can be used to verify that the workflow is making the calls. The build system used is gradle. For example, this PublishToNpm test uses PublishToNpmLibTester with expected parameter that can be unique to your workflow. The assertions makes sure that calls to npm registry is made which is mandatory to release an artifact.