Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OperatorHub Release CLI / Moving OperatorHub release to Buildkite #6281

Merged
merged 109 commits into from
Feb 27, 2023

Conversation

naemono
Copy link
Contributor

@naemono naemono commented Jan 5, 2023

This adds a Buildkite pipeline for release operations pertaining to OperatorHub/Red Hat.
This also augments our existing OpertaorHub CLI and adds new commands/sub-commands.

What's been manually tested:

Full set of operatorhub release operations with a BC build of Operator.

In progress:

  • Currently testing Buildkite integration (will fully test during actual release).

^^ All fully tested during release process for 2.6.x.

Adding vault option for easier usage.
move vault to container command
move single bundle command to push + publish commands
make create-pr separate command
adding prefight
additional fixes
code cleanup
un-export funcs
Adjust some comments
removal of 'all' command.
Removing preflight
move container prerune to persistent to ensure sub-commands also have it run
cleanup readme.
flags validation update.
attempting to hide root flags on sub-commands.
Cleanup readme
Make vault secrets default properly.
Update readme
Update some env var comments.
@naemono naemono added >enhancement Enhancement of existing functionality exclude-from-release-notes Exclude this PR from appearing in the release notes labels Jan 5, 2023
Copy link
Contributor

@thbkrkr thbkrkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I miss something but as discussed in #6281 (comment), I think we shouldn't duplicate what is defined in ./config.yaml (OHUB_CONF). This configuration file is the source of truth that lives in the codebase and is updated only through pull requests.
This means that all the following flags / env vars should disappear:

  • OHUB_SUPPORTED_OPENSHIFT_VERSIONS
  • OHUB_STACK_VERSION
  • OHUB_PREV_VERSION
  • OHUB_TAG
  • OHUB_BRANCH

It seems to me that having to adjust them is painful and above all prone to error.

.buildkite/pipeline-release-operatorhub.yml Outdated Show resolved Hide resolved
.buildkite/scripts/operatorhub/operations.sh Show resolved Hide resolved
.buildkite/pipeline-release-operatorhub.yml Show resolved Hide resolved
hack/operatorhub/internal/container/container.go Outdated Show resolved Hide resolved
.buildkite/scripts/operatorhub/operations.sh Outdated Show resolved Hide resolved
.buildkite/scripts/operatorhub/operations.sh Outdated Show resolved Hide resolved
Comment on lines 129 to 130
# If wanting to override newVersion|prevVersion|stackVersion variables in config.yaml
./bin/operatorhub generate-manifests -c config.yaml -y ../../config/crds.yaml -y ./../config/operator.yaml -s '8.6.0' -V '2.5.0' -t '2.6.0'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the use case that would require overriding the values defined in config.yaml?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I was wanting to ensure that the settings were absolutely correct when running in an automated fashion, but since the typical use case for this is to run this from a branch that already has config.yaml updated appropriately, then we can likely do without these, agreed. They've been removed.

cd hack/operatorhub
/usr/local/bin/operatorhub generate-manifests --prev-version="$OHUB_PREV_VERSION" --stack-version="$OHUB_STACK_VERSION" --yaml-manifest=../../config/crds.yaml --yaml-manifest=../../config/operator.yaml
/usr/local/bin/operatorhub bundle generate --dir="$(pwd)"
/usr/local/bin/operatorhub bundle create-pr --dir="$(pwd)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have kept these release commands in the pipeline and just use this script for the preflight commands.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured since I was cleaning up the pipeline, I might as well keep it as clean as possible, hence moving all of this to a script. Lmk if you need me to move it back.

cmd := &cobra.Command{
Use: "buildkite",
Short: "Start operatorhub release operation within Buildkite",
Long: "Start operatorhub release operation within Buildkite.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think

curl -XPOST "https://api.buildkite.com/v2/organizations/elastic/pipelines/cloud-on-k8s-operator-release-operatorhub/builds" \
-H "Authorization: Bearer $BK_TOKEN" \
-d '{
    "commit": "2.6.1",
    "branch": "2.6"
}'

could have been enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed this privately, and decided that opening #6408 would be beneficial.

naemono and others added 4 commits February 10, 2023 10:02
Try and chmod the binary prior to uploading as bk artifact.
Add headers to bash scripts.
Silence curl.
Adjust preflight command in script to actually call preflight.
Use io instead of ioutil.
Remove unused vars.
Use http.NewRequestWithContext instead of http.NewRequest.

Signed-off-by: Michael Montgomery <[email protected]>
Signed-off-by: Michael Montgomery <[email protected]>
ApiKey => APIKey per linter.
README updates.
Use io instead of ioutil.

Signed-off-by: Michael Montgomery <[email protected]>
@naemono
Copy link
Contributor Author

naemono commented Feb 13, 2023

Maybe I miss something but as discussed in #6281 (comment), I think we shouldn't duplicate what is defined in ./config.yaml (OHUB_CONF). This configuration file is the source of truth that lives in the codebase and is updated only through pull requests. This means that all the following flags / env vars should disappear:

  • OHUB_SUPPORTED_OPENSHIFT_VERSIONS
  • OHUB_STACK_VERSION
  • OHUB_PREV_VERSION
  • OHUB_TAG
  • OHUB_BRANCH

It seems to me that having to adjust them is painful and above all prone to error.

@thbkrkr I've removed prev/stack, but removing the others is a much bigger change as, for example, tag is used throughout the tool. I'll get these removed/updated soon.

Remove prev/stack version flags, and use values in config file.
Remove need for OHUB_BRANCH.

Signed-off-by: Michael Montgomery <[email protected]>
@naemono
Copy link
Contributor Author

naemono commented Feb 14, 2023

@thbkrkr changes made to remove need for

  • OHUB_STACK_VERSION
  • OHUB_PREV_VERSION
  • OHUB_TAG
  • OHUB_BRANCH

Unfortunately OHUB_SUPPORTED_OPENSHIFT_VERSIONS is required, as there's no setting for it in the configuration file. If you want, I can certainly add that as a new option there and remove the cli flag. LMK.

@thbkrkr
Copy link
Contributor

thbkrkr commented Feb 14, 2023

Unfortunately OHUB_SUPPORTED_OPENSHIFT_VERSIONS is required, as there's no setting for it in the configuration file. If you want, I can certainly add that as a new option there and remove the cli flag. LMK.

I think it's good to have all the configuration in the source files, so I feel like saying yes please.
Should it be more something like minSupportedOpenShiftVersion as we don't want the range here?

We already have the range in the template that we could half-generate from it or accept to have the info duplicated (we already have the minKubeVersion duplicated in the template).

* OpenShift 4.7-4.11

@naemono
Copy link
Contributor Author

naemono commented Feb 14, 2023

We already have the range in the template that we could half-generate from it or accept to have the info duplicated (we already have the minKubeVersion duplicated in the template).

I've kept it simple, and just added it to the configuration file, and updated the validation to ensure that it exists, and is not a range.

Copy link
Contributor

@thbkrkr thbkrkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

The vault bits could benefit from #6418 but we shouldn't block for that.

@naemono naemono merged commit 6e5d6e1 into elastic:main Feb 27, 2023
@naemono naemono deleted the 258-simpify-redhat-publishing branch February 27, 2023 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement Enhancement of existing functionality exclude-from-release-notes Exclude this PR from appearing in the release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants