Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update deployment page #139

Merged
merged 1 commit into from
Apr 9, 2019
Merged

update deployment page #139

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 78 additions & 79 deletions book/src/deploying.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,101 +1,102 @@
# Deploying CSI Driver on Kubernetes

> ## *This page is out-of-date and under active development.*

This page describes to CSI driver developers how to deploy their driver onto a Kubernetes cluster.

## Overview
There are three components plus the kubelet that enable CSI drivers to provide storage to Kubernetes. These components are sidecar containers which are responsible for communication with both Kubernetes and the CSI driver, making the appropriate CSI calls for their respectful Kubernetes events.

## Sidecar Containers
[![sidecar-container](images/sidecar-container.png)](https://docs.google.com/a/greatdanedata.com/drawings/d/1JExJ_98dt0NAsJ7iI0_9loeTn2rbLeEcpOMEvKrF-9w/edit?usp=sharing)
A CSI driver is typically deployed in Kubernetes as two components:
a controller component and a per-node component.

Sidecar containers manage Kubernetes events and make the appropriate calls to the CSI driver. These are the _external attacher_, _external provisioner_, _external snapshotter_ and the _driver registrar_.
### Controller Plugin

### External Attacher
[external-attacher](https://github.com/kubernetes-csi/external-attacher) is a sidecar container that watches Kubernetes _VolumeAttachment_ objects and triggers CSI _ControllerPublish_ and _ControllerUnpublish_ operations against a driver endpoint. As of this writing, the external attacher does not support leader election and therefore there can be only one running per CSI driver. For more information please read [_Attaching and Detaching_](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md#attaching-and-detaching).
The controller component can be deployed as a Deployment or StatefulSet on
any node in the cluster. It consists of the CSI driver that implements the
CSI Controller service and one or more
[sidecar containers](sidecar-containers.md). These controller
sidecar containers typically interact with Kubernetes objects and make calls
to the driver's CSI Controller service.

Note, even though this is called the _external attacher_, its function is to call the CSI API calls _ControllerPublish_ and _ControllerUnpublish_. These calls most likely will occur in a node which is _not_ the one that will mount the volume. For this reason, many CSI drivers do not support these calls, instead doing the attach/detach and mount/unmount both in the CSI _NodePublish_ and _NodeUnpublish_ calls done by the kubelet at the node which is supposed to mount.
It generally does not need direct access to the host and can perform all its
operations through the Kubernetes API and external control plane services.
Multiple copies of the controller component can be deployed for HA, however
it is recommended to use leader election to ensure there is only one active
controller at a time.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worthwhile to document how to do leader election/HA for our sidecars at some point?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#138 to track that work.


### External Provisioner
[external-provisioner](https://github.com/kubernetes-csi/external-provisioner) is a Sidecar container that watches Kubernetes _PersistentVolumeClaim_ objects and triggers CSI _CreateVolume_ and _DeleteVolume_ operations against a driver endpoint. For more information please read [_Provisioning and Deleting_](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md#provisioning-and-deleting).
Controller sidecars include the external-provisioner, external-attacher,
external-snapshotter, and external-resizer. Including a sidecar in the
deployment may be optional. See each sidecar's page for more details.

### External Snapshotter
[external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter) is a Sidecar container that watches Kubernetes _VolumeSnapshot_ objects and triggers CSI _CreateSnapshot_ and _DeleteSnapshot_ operations against a driver endpoint. For more information please read [_Snapshot Design Proposal_](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-snapshot.md).
#### Communication with Sidecars
[![sidecar-container](images/sidecar-container.png)](https://docs.google.com/a/greatdanedata.com/drawings/d/1JExJ_98dt0NAsJ7iI0_9loeTn2rbLeEcpOMEvKrF-9w/edit?usp=sharing)

### Driver Registrar
[driver-registrar](https://github.com/kubernetes-csi/driver-registrar) is a sidecar container that registers the CSI driver with kubelet, and adds the drivers custom NodeId to a label on the Kubernetes Node API Object. It does this by communicating with the _Identity_ service on the CSI driver and also calling the CSI _GetNodeId_ operation. The driver registrar must have the Kubernetes name for the node set through the environment variable `KUBE_NODE_NAME` as follows:
Sidecar containers manage Kubernetes events and make the appropriate
calls to the CSI driver. The calls are made by sharing a UNIX domain socket
through an emptyDir volume between the sidecars and CSI Driver.

```yaml
- name: csi-driver-registrar
imagePullPolicy: Always
image: quay.io/k8scsi/driver-registrar:v0.2.0
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /csi/csi.sock
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: socket-dir
mountPath: /csi
```
#### RBAC Rules

Most controller sidecars interact with Kubernetes objects and therefore need
to set RBAC policies. Each sidecar repository contains example RBAC
configurations.

### Node Plugin

The node component should be deployed on every node in the cluster through a
DaemonSet. It consists of the CSI driver that implements the CSI Node service and the
[node-driver-registrar](node-driver-registrar) sidecar container.

#### Communication with Kubelet

### Kubelet
[![kubelet](images/kubelet.png)](https://docs.google.com/a/greatdanedata.com/drawings/d/1NXaVNDh3mSDhog7Q3Y9eELyEF24F8Z-Kk0ujR3pyOes/edit?usp=sharing)

The Kubernetes kubelet runs on every node and is responsible for making the CSI calls _NodePublish_ and _NodeUnpublish_. These calls mount and unmount the storage volume from the storage system, making it available to the Pod to consume. As shown in the _external-attacher_, most CSI drivers choose to implement both their attach/detach and mount/unmount calls in the _NodePublish_ and _NodeUnpublish_ calls. They do this because the kubelet makes the request on the node which is to consume the volume.
The Kubernetes kubelet runs on every node and is responsible for making the CSI
Node service calls. These calls mount and unmount the storage volume from the
storage system, making it available to the Pod to consume. Kubelet makes calls
to the CSI driver through a UNIX domain socket shared on the host via a HostPath
volume. There is also a second UNIX domain socket that the node-driver-registrar
uses to register the CSI driver to kubelet.

#### Driver Volume Mounts
The node plugin needs direct access to the host for making block devices and/or
filesystem mounts available to the Kubernetes kubelet.

### Mount point
The mount point used by the CSI driver must be set to _Bidirectional_. See the example below:
The mount point used by the CSI driver must be set to _Bidirectional_ to allow Kubelet
on the host to see mounts created by the CSI driver container. See the example below:

```yaml
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: mountpoint-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
volumes:
containers:
- name: my-csi-driver
...
volumeMounts:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-hostpath
type: DirectoryOrCreate
mountPath: /csi
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: Directory
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: node-driver-registrar
...
volumeMounts:
- name: registration-dir
mountPath: /registration
volumes:
# This volume is where the socket for kubelet->driver communication is done
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/<driver-name>
type: DirectoryOrCreate
# This volume is where the driver mounts volumes
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: Directory
# This volume is where the node-driver-registrar registers the plugin
# with kubelet
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
```

### RBAC Rules
Side car containers need the appropriate permissions to be able to access and manipulate Kubernetes objects. Here are the RBAC rules needed:

```yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: csi-hostpath-role
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["create", "delete", "get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
```

## Deploying
Deploying a CSI driver onto Kubernetes is highlighted in detail in [_Recommended Mechanism for Deploying CSI Drivers on Kubernetes_](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md#recommended-mechanism-for-deploying-csi-drivers-on-kubernetes).
Expand All @@ -121,10 +122,8 @@ Another feature that CSI depends on is mount propagation. It allows the sharing

### Examples

- Simple deployment example using a single pod for all components: see the [hostpath example](Example.html).
- Full deployment example using a _DaemonSet_ for the node plugin and _StatefulSet_ for the controller plugin: check the [NFS driver deployment files][nfs-driver].
- Simple deployment example using a single pod for all components: see the [hostpath example](example.html).
- Full deployment example using a _DaemonSet_ for the node plugin and _StatefulSet_ for the controller plugin: TODO

## More information
For more information, please read [_CSI Volume Plugins in Kubernetes Design Doc_](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md).

[nfs-driver]: https://github.com/kubernetes-csi/drivers/tree/master/pkg/nfs/deploy/kubernetes