Release Cryostat 2.1 on OperatorHub.io

[jan-law]

Now that the 2.1 upstream release is available (see github tags), Cryostat 2.1 can be released on OperatorHub. Similar to the 2.0 release, you'll need to complete the prerequisites and open a PR with the updated operator.

To speed up the approval process, you can also run the operator pre-submission tests as explained here. These tests are identical to the tests that will run when you open the PR. I think this script worked last time for 2.0:

# (assuming you have podman already installed)
# setup steps
sudo dnf install ansible
sudo dnf install kubernetes-kubeadm
sudo swapoff -a # [disable swap for kubelet to work](https://stackoverflow.com/a/52196985)
systemctl enable kubelet.service
systemctl start kubelet

# This command will run the test suite. Notice the container tool is Docker:
OPP_CONTAINER_TOOL=docker OPP_AUTO_PACKAGEMANIFEST_CLUSTER_VERSION_LABEL=1 OPP_PRODUCTION_TYPE=k8s \
bash <(curl -sL https://raw.githubusercontent.com/redhat-openshift-ecosystem/community-operators-pipeline/ci/latest/ci/scripts/opp.sh) \
all \
operators/cryostat-operator/2.0.0 \
jan-law/community-operators \
cryostat-operator

[jan-law]

If the new interns are interested, I was thinking it would be good for them to get involved with the OperatorHub release. @hareetd should also be available to help when he gets back on the 26th too

[ebaron]

2.1.1 has now been released. Any takers for this?

[jan-law]

2.1.1 has now been released. Any takers for this?

@tthvo @maxcao13 ?

[maxcao13]

Yeah, I want to try this, @tthvo do you want to do it as well? I think we are going to definitely need some guidance.

[tthvo]

yeh sounds great! we can both work on this! @maxcao13

[hareetd]

Here's the issue for the original release on OperatorHub.io that @jan-law and I worked on. The PR for the release can be found here.

The issue I've linked contains some context to get you started. Much of the discussion on the issue has to do with errors we encountered running Cryostat on Kubernetes (as opposed to OpenShift) that have since been resolved but there's still useful info regarding the packaging process for the release.

[maxcao13]

Me and @tthvo are having some trouble getting the tests through locally. The error we both see is the one shown here https://gist.github.com/maxcao13/a47e2e1519a1d86db55f93a863fc8718

EDIT: I changed /etc/hosts/ and added kind-registry as a localhost alias and got slightly different errors
https://gist.github.com/maxcao13/71b61de209739dfff389f007c120c3ff

@hareetd and @jan-law helped us through, but we couldn't get past it. Interestingly the tests passed on @hareetd Fedora 36 local with Podman 4, but we have are not sure why it does not work for us. We've tried restarting our computers and docker as well.

Things we have tried:

sudo dnf install ansible
sudo dnf install kubernetes-kubeadm
sudo swapoff -a
systemctl start docker
systemctl enable kubelet.service
systemctl start kubelet
minikube start --driver=kvm2

OPP_CONTAINER_TOOL=docker OPP_AUTO_PACKAGEMANIFEST_CLUSTER_VERSION_LABEL=1 OPP_PRODUCTION_TYPE=k8s bash <(curl -sL https://raw.githubusercontent.com/redhat-openshift-ecosystem/community-operators-pipeline/ci/latest/ci/scripts/opp.sh) \
all \ 
operators/cryostat-operator/2.1.1 \
maxcao13/community-operators \
cryostat-operator

My forked repo and branch is here
https://github.com/maxcao13/community-operators/tree/cryostat-operator

Fedora 35:
Podman version: 3.4.7
Docker version 20.10.17, build 100c701
Minikube version: v1.25.2

EDIT 2: We've managed to pass all tests by instead using kind v0.11.0. Credit to @tthvo for figuring this part out!

[tthvo]

It turned out the script skipped the task to create the registry if kind is not available on PATH, specifically in /usr/local/bin as set here.

https://github.com/redhat-openshift-ecosystem/operator-test-playbooks/blob/master/upstream/roles/reset_kind/tasks/reset.yml

Also, running kind v0.12.0 appears to cause:

TASK [reset_kind : Configuring Kind with registry] **********************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/tmp/kind-with-registry.sh", "delta": "0:00:31.701774", "end": "2022-06-20 20:06:53.879448", "msg": "non-zero return code", "rc": 1, "start": "2022-06-20 20:06:22.177674", "stderr": "Error: No such container: kind-registry\nCreating cluster \"operator-test\" ...\n • Ensuring node image (kindest/node:v1.21.1) 🖼  ...\n ✓ Ensuring node image (kindest/node:v1.21.1) 🖼\n • Preparing nodes 📦   ...\n ✗ Preparing nodes 📦 \nERROR: failed to create cluster: could not find a log line that matches \"Reached target .Multi-User System.|detected cgroup v1\"", "stderr_lines": ["Error: No such container: kind-registry", "Creating cluster \"operator-test\" ...", "

So, we downgraded kind to v0.11.0 as suggested here.

kubernetes-sigs/kind#2460

So, the test suite is passed now :D

[maxcao13]

@hareetd @jan-law Do we also have to run OpenShift tests outlined here?

EDIT: replaced k8s with ocp in the test-command and it passes regardless

[tthvo]

Our preview on OperatorHub :))

[tthvo]

Did we miss anyone?

---
updateGraph: semver-mode
reviewers:
 - ebaron
 - andrewazores
 - hareetd
 - jan-law
 - jiekang
 - maxcao13
 - tthvo

[jan-law]

@hareetd @jan-law Do we also have to run OpenShift tests outlined here?

This release for OperatorHub.io is just for kubernetes users I think. I just checked the OpenShift operator catalog and the 2.1.1 release is already there.

Did we miss anyone?

Looks good to me!

[jan-law]

Our preview on OperatorHub :))

@ebaron it looks like the example Flight Recording and Recording CRDs (custom resource definitions) are still visible on the preview. I thought those were deprecated?

[hareetd]

This release for OperatorHub.io is just for kubernetes users I think. I just checked the OpenShift operator catalog and the 2.1.1 release is already there.

Yeah, our normal Cryostat release process with QA takes care of releasing Cryostat on OpenShift already so this issue is just for the Kubernetes users on OperatorHub.io

[maxcao13]

Full steps for passing all OperatorHub tests

Environment

• Fedora 35
• kind v0.11.0 in PATH (specifically /usr/local/bin/kind)
• podman v3.4.7
• docker v.20.10.17
• ansible v.2.9.27
• kubernetes-kubeadm v1.22.7
• curl
• openssl
• git

To setup environment

# Installing dependencies
sudo dnf install git openssl curl # If not yet installed
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
sudo dnf install podman
sudo dnf install moby-engine docker-compose
sudo dnf install ansible
sudo dnf install kubernetes-kubeadm

# Create and add yourself to docker group
# This avoids typing sudo on docker command
# Might need to log out and in again for effect to take place.
sudo groupadd docker
sudo usermod -aG docker $USER

# Start docker
sudo systemctl start docker

# Start kubelet
sudo swapoff -a # disable swap for kubelet to work -> https://stackoverflow.com/a/52196985
systemctl enable kubelet.service # (Kubernetes v1.22.7)
systemctl start kubelet

Run test suites

OPP_CONTAINER_TOOL=docker OPP_AUTO_PACKAGEMANIFEST_CLUSTER_VERSION_LABEL=1 OPP_PRODUCTION_TYPE=k8s \
bash <(curl -sL https://raw.githubusercontent.com/redhat-openshift-ecosystem/community-operators-pipeline/ci/latest/ci/scripts/opp.sh) \
all \
operators/cryostat-operator/{CRYOSTAT_VERSION} \
{GITHUB_USERNAME}/community-operators \
cryostat-operator

Note:

• Please fork the community-operators and clone the fork repository to your local machine.
• Create/Checkout a new branch cryostat-operator. When done with creating a new release bundle, commit your changes (signed-off) git commit -s and push your branch to your fork.
• Replace GITHUB_USERNAME with your GitHub username and CRYOSTAT_VERSION with new release version.
• Run this script directly under community-operators directory.

Clean up after tests

bash <(curl -sL https://raw.githubusercontent.com/redhat-openshift-ecosystem/community-operators-pipeline/ci/latest/ci/scripts/opp.sh) \
clean

Troubleshooting
If OLM resource error occurs, test resources might not have been cleaned. Follow step above to clean resources or run

/tmp/operator-test/operator-sdk olm uninstall

[tthvo]

@jan-law @hareetd We have some questions while going through the PR template.

Example templates for all Custom Resource Definitions intended to be used

This question, we need to wait for Elliott as here right?

Is your new CSV referenced in the appropriate channel defined in the package.yaml or annotations.yaml ?

We are not sure we understand this and how to check if valid?

[ebaron]

Our preview on OperatorHub :))

@ebaron it looks like the example Flight Recording and Recording CRDs (custom resource definitions) are still visible on the preview. I thought those were deprecated?

That's right. They're deprecated and we use the operators.operatorframework.io/internal-objects annotation to hide them from view in the OpenShift Console. It appears that OperatorHub.io doesn't support this annotation yet though: k8s-operatorhub/operatorhub.io#14.

Users will see warnings about those types being deprecated when using them on the command line at least.

[ebaron]

Is your new CSV referenced in the appropriate channel defined in the package.yaml or annotations.yaml ?

We are not sure we understand this and how to check if valid?

All of our operator releases should use the stable channel going forward. This is because we only want to support the latest release. The head of the stable channel will always contain the latest released version of our operator.

package.yaml seems to refer to the old packagemanifests format, so you don't need to worry about that.

annotations.yaml is what we use with the bundle format. For the 2.1.1 release, that would be this file: https://github.com/cryostatio/cryostat-operator/blob/v2.1.1/bundle/metadata/annotations.yaml. The stable channel is listed as the only channel, and also the default:

cryostat-operator/bundle/metadata/annotations.yaml

Lines 7 to 8 in 88ee941

	operators.operatorframework.io.bundle.channels.v1: stable
	operators.operatorframework.io.bundle.channel.default.v1: stable

I think in our case, we just have to ensure that the channels listed in annotations.yaml are what we expect them to be.

[ebaron]

Full steps for passing all OperatorHub tests

Thanks @maxcao13! This will be a very useful resource when we need to do our next release.

[ebaron]

Did we miss anyone?

---
updateGraph: semver-mode
reviewers:
 - ebaron
 - andrewazores
 - hareetd
 - jan-law
 - jiekang
 - maxcao13
 - tthvo

Looks good!

[tthvo]

Thanks @ebaron we will open the PR in a few minutes and send the link here

[tthvo]

Our 2.1.1 release PR is up!

k8s-operatorhub/community-operators#1365

[ebaron]

It's showing up here now: https://operatorhub.io/operator/cryostat-operator

Thanks for doing this!