The following document will walk through the process of defining 5 custom resource definitions to create and manage an MCM Application. By defining these resources we can install, delete and update resources on the managed MCM clusters. When these MCM resources are created the changes are applied on the target MCM cluster via the MCM Klusterlet. For this example we will be walking through the steps needed to deploy an Nginx MCM Application in a targeted Cluster. A diagram of how the resources work together to make an MCM Application can be seen below:
The channel resource defines the location of a resource to be deployed. These resources can be a Helm repository, Kubernetes namespace, Object store or Git repository. For this example we will be using a GitHub resource. Lets take a look at the 00-channel.yaml file and see what each section is.
# Create a Namespace to hold the Channel
apiVersion: app.ibm.com/v1alpha1
kind: Channel
metadata:
name: nginx-channel
namespace: nginx-channel
spec:
type: GitHub
pathname: https://github.com/ldesrosi/cp4mcm-nginx-app.git
When creating a Channel Yaml file you wan to specify the following:
- apiVersion: Contains the MCM API version for the Channel resource.
- Required. Set the value to
app.ibm.com/v1alpha1
.
- Required. Set the value to
- kind: Specifies that this is an Channel resource type.
- Required. Set the value to
Channel
to indicate that the resource is a channel.
- Required. Set the value to
- metadata: Contains the resource name and and it's namespace.
- name: Required. The name of the channel, we chose to go with nginx-channel
- namespace: Required. The namespace for the channel. We want this channel in our nginx-channel namespace.
- spec.kind: Required. The channel type. The supported types are:
Namespace
,HelmRepo
,GitHub
, andObjectBucket
. We will be specifying this channel as aGitHub
Channel. - spec.pathname: Required for
HelmRepo
,GitHub
, andObjectBucket
channels. For aGitHub
channel, set the value to be the HTTPS URL for the GitHub repository. This is where the artifacts for the app can be found. The artifacts don't need to be in a deployable yaml, but can be discovered by the MCM cluster.
Normally, to deploy an application to multiple clusters, you would need to run the following steps on each cluster:
- Login to get the kubernetes context.
- Run the
helm install
command.
This does not seem like a lot of work for 2 or 3 clusters. But the more clusters you have to work with, the more credentials you have to remember and manage. Also, if you are using CI/CD pipelines to deploy your workloads, the more clusters you have, the more pipelines you have to write and embed those credentials to, which can lead to human errors. If those credentials were to change, you would have to update those credentials wherever they are being used, which can get hard to track.
By leveraging MCM, access to your clusters is managed from a central location. MCM expands on this capability with Placement Policies
by allowing you to deploy workloads through it by specifying cluster labels, which are much easier to remember and manage. By adopting MCM into your CI/CD pipelines, you only have to remember the credentials for the MCM Hub cluster. To deploy workloads to multiple clusters, you only have to provide the number of application replicas and the cluster labels that match the clusters that you would like to deploy them to.
Lets inspect our 01-placement-rule.yaml:
apiVersion: app.ibm.com/v1alpha1
kind: PlacementRule
metadata:
name: nginx-placement-rule
namespace: nginx-app
labels:
name: nginx-placement-rule
app: nginx-app
spec:
clusterReplicas: 1
clusterConditions:
- type: OK
clusterLabels:
matchLabels:
environment: Dev
-
apiVersion: Contains the MCM API version for PlacementPolicy.
- Required. Set the value to
app.ibm.com/v1alpha1
.
- Required. Set the value to
-
Kind: Specifies that this is an
PlacementPolicy
resource type.- Required. Set the value to
PlacementRule
to indicate that the resource is a placement rule.
- Required. Set the value to
-
metadata: Contains the resource name and some labels
- name: Required. The name for identifying the placement rule.
- namespace: Required. The namespace resource to use for the placement rule.
- Labels: Optional. The labels for the placement rule.
-
spec.clusterReplicas: Optional, contains the number of replicas (application instances) to deploy.
-
spec.clusterConditions: Optional. Define any conditions for the cluster.
- Specify that you only want to deploy this application to a cluster that that is healthy by adding type with value OK.
-
spec.clusterLabels.matchLabels: Contains the list of labels and values that clusters must have for MCM to deploy applications to them.
- Optional. The labels that must exist for the target clusters.
You can create and manage subscriptions to identify, retrieve, and deploy new and updated resources to managed clusters. By using subscriptions, you can improve the continuous delivery capabilities of your application management.
Subscription resources can point to a channel for identifying new and updated Helm charts or Kubernetes resources for deployment. The subscription operator then watches the channel for new and updated charts and deployables. When a new or updated Helm chart or Kubernetes resource is detected, the subscription operator downloads the Helm release version for the specified Helm chart version or the specified Kubernetes resource. The subscription operator can download these objects directly, or as deployables, from the storage location to target managed clusters without checking the Hub cluster first.
Lets inspect our 02-subscription.yaml file.
apiVersion: app.ibm.com/v1alpha1 kind: Subscription metadata: name: nginx-sub namespace: nginx-app labels: app: nginx-app spec: channel: nginx-channel/nginx-channel placement: placementRef: name: nginx-placement-rule kind: PlacementRule group: app.ibm.com
-
apiVersion: Contains the MCM API version for the Deployment resource.
- Required. Set the value to `app.ibm.com/v1alpha1'.
-
kind: Specifies that this is an Deployment resource type.
- Required. Set the value to
Subscription
to indicate that the resource is a subscription.
- Required. Set the value to
-
metadata: Contains the resource name and some labels.
- name: Required. The name for identifying the placement rule.
- namespace: Required. The namespace resource to use for the placement rule.
- Labels: Optional. The labels for the placement rule.
-
spec.channel: Optional. The NamespaceName ("Namespace/Name") that defines the channel for the subscription. Define either the
channel
, or thesource
, or thesourceNamespace
field. In general, use thechannel
field to point to the channel instead of using thesource
orsourceNamespace
fields. If more than one field is defined, the first field that is defined is used. -
spec.placement.placementRef: Optional. Defines the placement rule to use for the subscription. Use only one of
clusters
,clusterSelector
, orplacementRef
to define where your subscription is to be placed for a multi-cluster. If your cluster is a stand-alone cluster that is not your Hub cluster, you can also uselocal
.- name: Optional, but required for using a placement rule. The name of the placement rule for the subscription.
- kind: Optional, but required for using a placement rule. Set the value to
PlacementRule
to indicate that a placement rule is used for deployments with the subscription. - group: MISSING
In order for MCM to recognize your workloads as an MCM application, you have to create an
Application
resource. TheApplication
resource lets you specify criteria to associate Kubernetes resources to your application so that MCM can manage and monitor them. Lets look at your 03-app.yaml file.apiVersion: app.k8s.io/v1beta1 kind: Application metadata: name: nginx-app namespace: nginx-app labels: name: nginx-app app: nginx-app spec: componentKinds: - group: app.ibm.com/v1alpha1 kind: Subscription selector: matchLabels: app: nginx-app
-
apiVersion: Contains the Community SIG API version for Applications.
- Required. Set the value to
app.k8s.io/v1beta1
.
- Required. Set the value to
-
kind: Specifies that this is an
Application
resource type.- Required. Set the value to
Application
to indicate the resource is an application resource.
- Required. Set the value to
-
metadata: Contains the resource name and some labels.
- name: Required. The name for identifying the application resource.
- namespace: The namespace resource to use for the application.
- Labels: Optional. The labels for the deployable.
-
spec.selector.componentKinds: Contains a list of the kinds of resources to be associated with the application.
- Optional. The list of the kinds of resources to be associated with the application.
-
spec.selector.matchLabels.app
- MISSING
If you have IBM Container Image Security Enforcement enabled in your MCM cluster, you can limit where images are allowed to be pulled from by specifying the allowed repositories to use. Lets see our image-policy.yaml file.
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1 kind: ClusterImagePolicy metadata: name: nginx-app namespace: nginx-app spec: repositories: # allow all images - name: "*" policy:
- apiVersion: Contains the security enforcement API version for the Image Policy resource.
- Required: securityenforcement.admission.cloud.ibm.com/v1beta1. IBM Container Image Security Enforcement has to be enabled for your MCM cluster.
- kind: Specifies that this is an Image Security resource type.
- Required: The type of image
- metadata: Contains the resource name and some labels.
- name: Required. The name for identifying the application resource.
- namespace: The namespace resource to use for the application.
- spec.repositories
- name: name of the repository you would like to use. By setting the name as a wildcard "*" of the repository, you are allowing images to be pulled from any repository.
Now that we have covered our resources and know what each of them do, we can go ahead and deploy our Nginx MCM App on an Openshift Cluster by following the steps below:
-
Login to your OpenShift environment by copying your login credentials from the Openshift UI.
oc login --token=<your_token> --server=<server url>
-
Clone the github repo and open the resource directory.
git clone https://github.com/ldesrosi/cp4mcm-nginx-mgmt.git
cd resources
-
Create
nginx-app
andnginx-channel
namespaces which are going to be used to work on the artifacts.oc new-project nginx-app
oc new-project nginx-channel
-
If you have IBM Container Image Security Enforecement enabled and would like to allow all repos to be used run the following command.
oc apply -f ../policies/image-policy.yaml
-
Create your channel.
oc apply -f 00-channel.yaml
If you get any errors such as
Github is not a type of Enum
you must update the Custom Resource Definition. View the troubleshoot section. -
Crate your Placement Rule by running the following command.
oc apply -f 01-placement-rule.yaml
-
Create your subcription with the following command.
oc apply -f 02-subscription.yaml
-
Create your application resource with the following oc command.
oc apply -f 03-app.yaml
This will not deploy the application to a targeted cluster as the placement rule has to be met first. For our placement rule, there needs to be a healthy targeted cluster with the label environment
set to Dev
. Once a targeted cluster meets this criteria, the application will then be deployed to that cluster.
Note that this repository does NOT contain any NGINX deployment. Instead the 00-channel.yaml contains a GitHub repo that will be used by MCM to clone and instrument the standard NGINX deployment with the MCM instructions.
Depending on the version of MCM, the CRD might not have the type Github
, hence we must manually update this.
Copy and paste the following and name is as mcm-crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: channels.app.ibm.com
spec:
additionalPrinterColumns:
- JSONPath: .spec.type
description: type of the channel
name: Type
type: string
- JSONPath: .spec.pathname
description: pathname of the channel
name: PathName
type: string
- JSONPath: .metadata.creationTimestamp
name: Age
type: date
group: app.ibm.com
names:
kind: Channel
listKind: ChannelList
plural: channels
singular: channel
scope: Namespaced
subresources: {}
validation:
openAPIV3Schema:
description: Channel is the Schema for the channels API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: ChannelSpec defines the desired state of Channel
properties:
configRef:
description: ObjectReference contains enough information to let you
inspect or modify the referred object.
properties:
apiVersion:
description: API version of the referent.
type: string
fieldPath:
description: 'If referring to a piece of an object instead of an
entire object, this string should contain a valid JSON/Go field
access statement, such as desiredState.manifest.containers[2].
For example, if the object reference is to a container within
a pod, this would take on a value like: "spec.containers{name}"
(where "name" refers to the name of the container that triggered
the event) or if no container name is specified "spec.containers[2]"
(container with index 2 in this pod). This syntax is chosen only
to have some well-defined way of referencing a part of an object.
TODO: this design is not final and this field is subject to change
in the future.'
type: string
kind:
description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
type: string
name:
description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
type: string
namespace:
description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
type: string
resourceVersion:
description: 'Specific resourceVersion to which this reference is
made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
type: string
uid:
description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
type: string
type: object
gates:
description: ChannelGate defines criteria for promote to channel
properties:
annotations:
additionalProperties:
type: string
type: object
labelSelector:
description: A label selector is a label query over a set of resources.
The result of matchLabels and matchExpressions are ANDed. An empty
label selector matches all objects. A null label selector matches
no objects.
properties:
matchExpressions:
description: matchExpressions is a list of label selector requirements.
The requirements are ANDed.
items:
description: A label selector requirement is a selector that
contains values, a key, and an operator that relates the
key and values.
properties:
key:
description: key is the label key that the selector applies
to.
type: string
operator:
description: operator represents a key's relationship
to a set of values. Valid operators are In, NotIn, Exists
and DoesNotExist.
type: string
values:
description: values is an array of string values. If the
operator is In or NotIn, the values array must be non-empty.
If the operator is Exists or DoesNotExist, the values
array must be empty. This array is replaced during a
strategic merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: matchLabels is a map of {key,value} pairs. A single
{key,value} in the matchLabels map is equivalent to an element
of matchExpressions, whose key field is "key", the operator
is "In", and the values array contains only "value". The requirements
are ANDed.
type: object
type: object
name:
type: string
type: object
pathname:
type: string
secretRef:
description: ObjectReference contains enough information to let you
inspect or modify the referred object.
properties:
apiVersion:
description: API version of the referent.
type: string
fieldPath:
description: 'If referring to a piece of an object instead of an
entire object, this string should contain a valid JSON/Go field
access statement, such as desiredState.manifest.containers[2].
For example, if the object reference is to a container within
a pod, this would take on a value like: "spec.containers{name}"
(where "name" refers to the name of the container that triggered
the event) or if no container name is specified "spec.containers[2]"
(container with index 2 in this pod). This syntax is chosen only
to have some well-defined way of referencing a part of an object.
TODO: this design is not final and this field is subject to change
in the future.'
type: string
kind:
description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
type: string
name:
description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
type: string
namespace:
description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
type: string
resourceVersion:
description: 'Specific resourceVersion to which this reference is
made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
type: string
uid:
description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
type: string
type: object
sourceNamespaces:
items:
type: string
type: array
type:
description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
Important: Run "make" to regenerate code after modifying this file'
enum:
- Namespace
- HelmRepo
- ObjectBucket
- GitHub
- namespace
- helmrepo
- objectbucket
- github
type: string
required:
- pathname
- type
type: object
status:
description: ChannelStatus defines the observed state of Channel
type: object
type: object
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: true
Now you can update your CRD by running
oc apply -f ./mcm-crd.yaml