crd.md

使用CRD扩展Kubernetes

1. 简介

• CRD简单的例子

  https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/

• 写一个controller处理CRD

  https://github.com/kubernetes/sample-controller

  sample-controller: 放在go/src/k8s.io目录下

  https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md

• 关于subresource(status/scale)及category

  https://blog.openshift.com/kubernetes-custom-resources-grow-up-in-v1-10/

  如果要生成status相关的方法，删除CRD数据结构上的下面tag即可：

  // +genclient:noStatus
  

  如果要生成scale相关的方法，需要在CRD数据结构上加上一些tag：

  Similar to how an UpdateStatus() method exists for the status subresource, we can generate the GetScale() and UpdateScale() methods for the scale subresource by adding the following tags on the Database type.
  
  // +genclient:method=GetScale,verb=get,subresource=scale,result=k8s.io/api/autoscaling/v1.Scale
  // +genclient:method=UpdateScale,verb=update,subresource=scale,input=k8s.io/api/autoscaling/v1.Scale,result=k8s.io/api/autoscaling/v1.Scale
  

  使用scale subresource可以很方便地实现自动扩缩容。

• CRD validation

  可以通过 OpenAPI v3 schema对custom resource(CR)进行校验，否则如果用户创建的CR包含有不合规则的字段，就会导致CRD相应的controller没法正常工作，比如：不能正常list这些CR（解析时会出错）。

  如果CRD里要校验的字段太多，可以考虑下面两种方式：

  • 使用validatingadmissionwebhook进行校验

  • 利用工具自动生成OpenAPI schema

    https://github.com/kubernetes-sigs/controller-tools/tree/master/cmd/crd

    https://github.com/ant31/crd-validation

2. 实现

• CRD registry

  相关的代码在staging/src/k8s.io/apiextensions-apiserver/pkg/registry下。

• ETCD存储

  CRD存储在etcd上的路径与deployment、pod等核心资源的存储位置不一样，它的路径为：root / resource.Group + "/" + resource.Resource（见staging/src/k8s.io/apiextensions-apiserver/pkg/apisever/customresource_handler.go#574，注：在project里搜索“ResourcePrefix”可以搜到其他resource的存储路径）。

  注： tapp为/registry/gaia/tapps/default/example-tapp。为了平滑升级，兼容以前的版本，我们需要修改tapp的存储路径。

3. 自动化工具

• operator-sdk

  This project is a component of the Operator Framework, an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way.

  使用operator-sdk时可参考operator-sdk get started。

• Kubebuilder

  Kubebuilder is a framework for building Kubernetes APIs using custom resource definitions (CRDs).

  Note: kubebuilder does not exist as an example to copy-paste, but instead provides powerful libraries and tools to simplify building and publishing Kubernetes APIs from scratch.

• metacontroller

  Metacontroller is an add-on for Kubernetes that makes it easy to write and deploy custom controllers in the form of simple scripts.

4. 一些有趣的CRD/Operator

Awesome Operators in the Wild: https://github.com/operator-framework/awesome-operators

Operator hub: https://operatorhub.io

5. 参考资料

• Extend the Kubernetes API with CustomResourceDefinitions: https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/
• Building an operator for Kubernetes with the sample-controller: https://itnext.io/building-an-operator-for-kubernetes-with-the-sample-controller-b4204be9ad56
• Building an operator for Kubernetes with kubebuilder: https://itnext.io/building-an-operator-for-kubernetes-with-kubebuilder-17cbd3f07761
• Building an operator for Kubernetes with operator-sdk: https://itnext.io/building-an-operator-for-kubernetes-with-operator-sdk-40a029ea056
• Kubernetes Operators Best Practices: https://blog.openshift.com/kubernetes-operators-best-practices/