- QuarksStatefulSet
The QuarksStatefulSet component can be understood as the set of controllers responsible for translating the BOSH manifest instance_groups into Kubernetes resources.
The QuarksStatefulset component is a categorization of a set of controllers, under the same group. Inside the QuarksStatefulset component, we have a set of 2 controllers together with one separate reconciliation loop per controller.
Figure 1 illustrates a QuarksStatefulset component diagram that covers the set of controllers it uses.
Fig. 1: The QuarksStatefulset component
Fig. 2: The QuarksStatefulset controller
This controller will generate a Kubernetes statefulset for each instance_group defined in the BOSH manifest. This Statefulset will also include a set of Kubernetes services, so that each component can be accessed on specific ports.
QuarksStatefulset: CreationConfigmaps: UpdateSecrets: Update
Will generate versioned Statefulsets with the required data to make all jobs of the instance_group runnable.
Ability to set restrictions on how scaling can occur: min, max, odd replicas.
When an env value or mount changes due to a ConfigMap or Secret change, containers are restarted.
The operator watches all the ConfigMaps and Secrets referenced by the StatefulSet, and automatically performs the update, without extra workarounds.
Exposing quarksstatefulsets is similar to exposing statefulsets in kubernetes. A Kubernetes service makes use of labels to select the pods which should be in the service. We need to use two labels to group the pods of a single instance group.
quarks.cloudfoundry.org/instance-group-name: ((instanceGroupName))quarks.cloudfoundry.org/deployment-name: ((deploymentName))
Following is the example which creates a service with type ClusterIP for a single instance group named nats in deployment nats-deployment for exposing port 4222.
apiVersion: v1
kind: Service
metadata:
name: nats-service
spec:
type: ClusterIP
selector:
quarks.cloudfoundry.org/instance-group-name: nats
quarks.cloudfoundry.org/deployment-name: nats-deployment
ports:
- protocol: TCP
port: 80
targetPort: 4222Complete example can be found here.
Though, by default, quarks creates three services of type ClusterIP as defined here for any instance group.
For creating a service type LoadBalancer, we just need to change the .spec.type to LoadBalancer in the above example. The LoadBalancer Ingress is your public IP specified in the output of this command kubectl describe service nats-service.
Ingress doesn't use any labels but just sits on top of services and acts as a smart router. You can create services of different types based on the above examples and use them as values in the ingress Kubernetes spec. An example of Ingress can be found here
For more information about Kubernetes services, we recommend you to read this.
When an update needs to happen, a second StatefulSet for the new version is deployed, and both coexist until canary conditions are met. This also allows support for Blue/Green techniques.
Note:
This could make integration with Istio easier and (more) seamless.
Annotated with a version (auto-incremented on each update). The annotation key is quarks.cloudfoundry.org/version.
Ability to upgrade even though StatefulSet pods are not ready.
During upgrades, there is more than one StatefulSet version for an QuarksStatefulSet resource. The operator lists available versions and keeps track of which are running.
A running version means that at least one pod that belongs to a StatefulSet is running. When a version n is running, any version lower than n is deleted.
The controller continues to reconcile until there's only one version.
The zones key defines the availability zones the QuarksStatefulSet needs to span.
The zoneNodeLabel defines the node label that defines a node's zone.
The default value for zoneNodeLabel is failure-domain.beta.kubernetes.io/zone.
The example below defines an QuarksStatefulSet that should be deployed in two availability zones, us-central1-a and us-central1-b.
apiVersion: quarks.cloudfoundry.org/v1alpha1
kind: QuarksStatefulSet
metadata:
name: MyQuarksStatefulSet
spec:
zoneNodeLabel: "failure-domain.beta.kubernetes.io/zone"
zones: ["us-central1-a", "us-central1-b"]
...
template:
spec:
replicas: 2
...The QuarksStatefulSet controller creates one StatefulSet version for each availability zone, and adds affinity information to the pods of those StatefulSets:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "failure-domain.beta.kubernetes.io/zone"
operator: In
values: ["us-central1-a"]If zones are set for an QuarksStatefulSet, the following occurs:
-
The name of each created
StatefulSetis generated as<quarks statefulset name>-z<index of az>.myquarksstatefulset-z0 -
The
StatefulSetand itsPodsare labeled with the following:quarks.cloudfoundry.org/az-index: "0" quarks.cloudfoundry.org/az-name: "us-central1-a"
-
The
StatefulSetand itsPodsare annotated with an ordered JSON array of all the availability zones:quarks.cloudfoundry.org/zones: '["us-central1-a", "us-central1-b"]'
-
As defined above, each pod is modified to contain affinity rules.
-
Each container and init container of each pod have the following env vars set:
KUBE_AZ="zone name" BOSH_AZ="zone name" CF_OPERATOR_AZ="zone name" AZ_INDEX="zone index"
Taints and tolerations is a concept defined in kubernetes to repel pods from nodes link. Defining tolerations is same as defined in the kubernetes docs. Keep in mind the affinity rules added by the controller when az's are defined. An example is specified in the examples folder.
QuarksStatefulSets can be automatically updated when the environment/mounts have changed due to a referenced
ConfigMap or a Secret being updated. This behavior is controlled by the updateOnConfigChange flag which defaults to false.
StatefulSet: Creation/Update
It will delete statefulsets with old versions, only after the new statefulset version instances are up and running.
Fig. 3: The QuarksStatefulset active/passive controller
Active/passive model is application model that have multiple running instances, but only one instance is active and all other instances are passive (standby). If the active instance is down, one of the passive instances will be promoted to active immediately.
The activePassiveProbes key defines active probe to be performed on a container. The controller examines the active probe periodically to see if the active one is still active. If active pod is down or there isn’t an active pod, the first running pod will be promoted as active and label it as quarks.cloudfoundry.org/pod-active: active.
apiVersion: quarks.cloudfoundry.org/v1alpha1
kind: QuarksStatefulSet
metadata:
name: myquarksstatefulset
spec:
activePassiveProbes:
busybox:
# define a active probe on the container
exec:
command:
- /bin/sh
- -c
- /root/check-active.sh
template:
spec:
replicas: 2
template:
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
...The controller manages this active probing and provides pod designation label to the service's selectors. Any requests sent to the service will then only be sent to the active pod.
Fig. 4: Relationship with the BPM controller
Figure 4 illustrates the interaction of the BPM Controller with the QuarksStatefulSet Controller. Once the BPM controller consumes the data persisted in secrets from the QuarksJob Component, it will use that data to generate new QuarksStatefulset instances. When these resources are generated, the QuarksStatefulSet controller will be watching and trigger its reconciliation loop.
See https://github.com/cloudfoundry-incubator/cf-operator/tree/master/docs/examples/quarks-statefulset