diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..79784286c --- /dev/null +++ b/docs/README.md @@ -0,0 +1,24 @@ +# DirectPV +DirectPV is a CSI driver to provision local volumes to Pods using [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage). It comes with two components. +1. `DirectPV plugin` - To be installed on local machine, to manage DirectPV CSI driver. +2. `DirectPV CSI driver` - To be installed on kubernetes cluster, to provision local volumes. + +## For new users +Refer below documentation +* [Installation guide](./installation.md) +* [Drive management guide](./drive-management.md) +* [Volume provisioning guide](./volume-provisioning.md) +* [Volume management guide](./volume-management.md) +* [Command reference guide](./command-reference.md) +* [Monitoring guide](./monitoring.md) + +## For existing users +Refer below documentation +* [Upgrade guide](./upgrade.md) +* [Node management guide](./node-management.md) + +## Further references +* [Volume scheduling guide](./volume-scheduling.md) +* [Issue reporting](./issue-reporting.md) +* [DirectPV CSI driver specification](./specification.md) +* [Troubleshooting guide](./troubleshooting.md) diff --git a/docs/architecture.md b/docs/architecture.md index c2765fb84..26ce044e8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,138 +1,96 @@ ---- -title: Architecture ---- - -Architecture -------------- - -### Components - -DirectPV is made up of 5 components: - -| Component | Description | -|-------------------|---------------------------------------------------------------------------------------| -| CSI Driver | Performs mounting, unmounting of provisioned volumes | -| CSI Controller | Responsible for scheduling and detaching volumes on the nodes | -| Drive Controller | Formats and manages drive lifecycle | -| Volume Controller | Manages volume lifecycle | -| Drive Discovery | Discovers and monitors the drives and their states on the nodes | - -The 4 components run as two different pods. - -| Name | Components | Description | -|-------------------------------|-----------------------------------------------------------------------|------------------------------------| -| DirectPV Node Driver | CSI Driver, Driver Controller, Volume Controller, Drive Discovery | runs on every node as a DaemonSet | -| DirectPV Central Controller | CSI Controller | runs as a deployment | - - -### Scalability - -Since the node driver runs on every node, the load on it is constrained to operations specific to that node. - -The central controller needs to be scaled up as the number of drives managed by DirectPV is increased. By default, 3 replicas of central controller are run. As a rule of thumb, having as many central controller instances as etcd nodes is a good working solution for achieving high scale. - -### Availability - -If node driver is down, then volume mounting, unmounting, formatting and cleanup will not proceed for volumes and drives on that node. In order to restore operations, bring node driver to running status. - -In central controller is down, then volume scheduling and deletion will not proceed for all volumes and drives in the DirectPV cluster. In order to restore operations, bring the central controller to running status. - -Security is covered [here](./security.md) - -### Node Driver - -This runs on every node as a Daemonset in `directpv` namespace. Each pod consists of four containers - -#### Node driver registrar - -This is a kubernetes csi side-car container which registers the `directpv` CSI driver with kubelet. This registration is necessary for kubelet to issue CSI RPC calls like `NodeGetInfo`, `NodeStageVolume`, `NodePublishVolume` to the corresponding nodes. - -For more details, please refer [node-driver-registrar](https://github.com/kubernetes-csi/node-driver-registrar). - -#### Livenessprobe - -This is a kubernetes csi side-car container which exposes an HTTP `/healthz` endpoint as a liveness hook. This endpoint will be used by kubernetes for csi-driver liveness checks. - -For more details. please refer [livenessprobe](https://github.com/kubernetes-csi/livenessprobe) - -#### Dynamic drive discovery - -This container uses `directpv` binary with `--dynamic-drive-handler` flag enabled. This container is responsible for discovering and managing the drives in the node. - -The devices will be discovered from `/run/data/udev/` directory and dynamically listens for udev events for any add, change and remove uevents. Apart from dynamically listening, there is a periodic 30sec sync which checks and syncs the drive states. - -For any change, the directcsidrive object will be synced to match the local state. A new directcsidrive object will be created when a new device is detected during sync or when an "Add" uevent occurs. If an inuse/ready drive gets corrupted or lost, it will be tagged with a error condition on the drive. If an Available/Unavailable drive is lost, it will be deleted. - -#### DirectPV - -This container acts as a node plugin and implements the following node service RPCs. - -- [NodeGetInfo](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetinfo) -- [NodeGetCapabilities](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetinfoNodeGetCapabilities) -- [NodeGetVolumeStats](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetvolumestats) -- [NodeStageVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodestagevolume) -- [NodePublishVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodepublishvolume) -- [NodeUnstageVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodeunstagevolume) -- [NodeUnpublishVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodeunpublishvolume) - -This container is responsible for bind-mounting and umounting volumes on the responding nodes. Monitoring volumes is a WIP and will be added soon. Please refer [csi spec](https://github.com/container-storage-interface/spec) for more details on the CSI volume lifecycle. - -Apart from this, there are also drive and volume controllers in place. - -#### Drive Controller - -Drive controller manages the `directpvdrives` object lifecycle. This actively listens for drive object (post-hook) events like Add, Update and Delete. The drive controller is responsible for the following - -_Formatting a drive_ :- - -If `.Spec.RequestedFormat` is set on the drive object, it indicates the `kubectl directpv drives format` was called on it and this drive will be formatted. - -_Releasing a drive_ :- - -`kubectl directpv drives release` is a special command to release a "Ready" drive in directpv cluster by umounting the drives and making it "Available". If `.Status.DriveStatus` is set to "Released", it indicates that `kubectl directpv drives release` was called on the drive and it will be released. - -_Checking the primary mount of the drive_ :- - -Drive controller also checks for the primary drive mounts. If an "InUse" or "Ready" drive is not mounted or if it has unexpected mount options set, the drive will be remounted with correct mountpoint and mount options. - -_Tagging the lost drives_ :- - -If a drive is not found on the host, it will be tagged as "lost" with an error message attached to the drive object and its respective volume objects. - -Overall, drive controller validates and tries to sync the host state of the drive to match the expected state of the drive. For example, it mounts the "Ready" and "InUse" drives if their primary mount is not present in host. - -For more details on the drive states, please refer [Drive States](./drive-states.md). - -#### Volume Controller - -Volume controller manages the `directpvvolumes` object lifecycle. This actively listens for volume object (post-hook) events like Add, Update and Delete. The volume controller is responsible for the following - -_Releasing/Purging deleted volumes and free-ing up its space on the drive_ :- - -When a volume is deleted (PVC deletion) or purged (using `kubectl directpv drives purge` command), the corresponding volume object will be in terminating state (with deletion timestamp set on it). The volume controller will look for such deleted volume objects and releases them by freeing up the disk space and unsetting the finalizers. - - -### Central Controller - -This runs as a deployment in `directpv` namespace with default replica count 3. - -(Note: The central controller does not do any device level interactions in the host) - -Each pod consist of two continers - -#### CSI Provisioner - -This is a kubernetes csi side-car container which is responsible for sending volume provisioning (CreateVolume) and volume deletion (DeleteVolume) requests to csi drivers. - -For more details, please refer [external-provisioner](https://github.com/kubernetes-csi/external-provisioner). - -#### DirectPV - -This container acts as a central controller and implements the following RPCs - -- [CreateVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#createvolume) -- [DeleteVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#deletevolume) - -This container is responsible for selecting a suitable drive for a volume scheduling request. The selection algorithm looks for range and topology specifications provided in the CreateVolume request and selects a drive based on its free capacity. - -(Note: kube-scheduler is responsible for selecting a node for a pod, central controller will just select a suitable drive in the requested node based on the specifications provided in the create volume request) +# DirectPV CSI driver architecture + +DirectPV is implemented as per [CSI specification](https://github.com/container-storage-interface/spec/blob/master/spec.md). It comes with below components run as Pods in Kubernetes. +* `Controller` +* `Node server` + +When DirectPV contains legacy volumes from `DirectCSI`, below additional components run as Pods +* `Legacy controller ` +* `Legacy node server` + +## Controller +Controller runs as `Deployment` Pods named `controller` which is three replicas in any Kubernetes nodes not limiting to where node server is running. In the three replicas, one instance is elected to serve requests. Each pod contains below running containers +* `CSI provisioner` - Bridges volume creation and deletion requests from `Persistent Volume Claim` to CSI controller. +* `Controller` - Controller server which honors CSI requests to create, delete and expand volumes. +* `CSI resizer` - Bridges volume expansion requests from `Persistent Volume Claim` to CSI controller. + +### Controller server +Controller server runs as container `controller` in a `controller` `Deployment` Pod. It handles below requests +* `Create volume` - Controller server creates new `DirectPVVolume` CRD after reversing requested storage space on suitable `DirectPVDrive` CRD. For more information, refer [Volume scheduling guide](./volume-scheduling.md) +* `Delete volume` - Controller server deletes `DirectPVVolume` CRD for unbound volumes after releasing previously reserved space in `DirectPVDrive` CRD. +* `Expand volume` - Controller server expands `DirectPVVolume` CRD after reversing requested storage space in `DirectPVDrive` CRD. + +Below is a workflow diagram +``` +┌────────────┐ ┌────────────┐ +│ │ Create Event ┌─────────────┐ CreateVolume API │ │ ┌────────────────────┐ +│ │------------->│ CSI │----------------->│ │-->│ DirectPVDrive CRD │ +│ Persistent │ Delete Event │ │ DeleteVolume API │ │ └────────────────────┘ +│ Volume │------------->│ Provisioner │----------------->│ Controller │ +│ Claim │ └─────────────┘ │ Server │ +│ (PVC) │ │ │ ┌────────────────────┐ +│ │ Update Event ┌─────────────┐ ExpandVolume API │ │-->│ DirectPVVolume CRD │ +│ │------------->│ CSI Resizer │----------------->│ │ └────────────────────┘ +│ │ └─────────────┘ └────────────┘ +└────────────┘ +``` + +## Legacy controller +Legacy controller runs as `Deployment` Pods named `legacy-controller` which is three replicas in any Kubernetes nodes not limiting to where node server is running. In the three replicas, one instance is elected to serve requests. Each pod contains below running containers +* `CSI provisioner` - Bridges legacy volume creation and deletion requests from `Persistent Volume Claim` to CSI controller. +* `Controller` - Honors CSI requests to delete and expand volumes. Create volume request is prohibited i.e. this controller works only for legacy volumes previously created in `DirectCSI`. +* `CSI resizer` - Bridges legacy volume expansion requests from `Persistent Volume Claim` to CSI controller. + +### Legacy controller server +Legacy controller server runs as container `controller` in a `legacy-controller` `Deployment` Pod. It handles below requests +* `Create volume` - Controller server errors out for this request. +* `Delete volume` - Controller server deletes `DirectPVVolume` CRD for unbound volumes after releasing previously reserved space in `DirectPVDrive` CRD. +* `Expand volume` - Controller server expands `DirectPVVolume` CRD after reversing requested storage space in `DirectPVDrive` CRD. + +Below is a workflow diagram +``` +┌─────────────────────┐ ┌────────────┐ +│ │ Delete Event ┌─────────────────┐ DeleteVolume API │ │ ┌────────────────────┐ +│ Persistent Volume │------------->│ CSI Provisioner │----------------->│ │-->│ DirectPVDrive CRD │ +│ Claim (PVC) with │ └─────────────────┘ │ Controller │ └────────────────────┘ +│ direct-csi-min-io │ │ Server │ ┌────────────────────┐ +│ Storage Class │ Update Event ┌─────────────────┐ ExpandVolume API │ │-->│ DirectPVVolume CRD │ +│ │------------->│ CSI Resizer │----------------->│ │ └────────────────────┘ +│ │ └─────────────────┘ └────────────┘ +└─────────────────────┘ +``` +## Node server +Node server runs as `DaemonSet` Pods named `node-server` in all or selected Kubernetes nodes. Each node server Pod runs on a node independently. Each pod contains below running containers +* `Node driver registrar` - Registers node server to kubelet to get CSI RPC calls. +* `Node server` - Honors stage, unstage, publish, unpublish and expand volume RPC requests. +* `Node controller` - Honors `DirectPVDrive`, `DirectPVVolume`, `DirectPVNode` and `DirectPVInitRequest` CRDs events. +* `Liveness probe` - Exposes `/healthz` endpoint to check node server liveness by Kubernetes. + +Below is a workflow diagram +``` +┌─────────┐ ┌────────┐ ┌──────────────────────────────────┐ ┌────────────────────┐ +│ │ StageVolume RPC │ │ StageVolume │ * Create data directory │ │ │ +│ │------------------->│ │---------------->│ * Set xfs quota │<-->│ │ +│ │ │ │ │ * Bind mount staging target path │ │ │ +│ │ │ │ └──────────────────────────────────┘ │ │ +│ │ PublishVolume RPC │ │ PublishVolume ┌──────────────────────────────────┐ │ │ +│ │------------------->│ │---------------->│ * Bind mount target path │<-->│ │ +│ Kubelet │ │ Node │ └──────────────────────────────────┘ │ DirectPVDrive CRD │ +│ │UnpublishVolume RPC │ Server │ UnpublishVolume ┌──────────────────────────────────┐ │ DirectPVVolume CRD │ +│ │------------------->│ │---------------->│ * Unmount target path │<-->│ │ +│ │ │ │ └──────────────────────────────────┘ │ │ +│ │ UnstageVolume RPC │ │ UnstageVolume ┌──────────────────────────────────┐ │ │ +│ │------------------->│ │---------------->│ * Unmount staging target path │<-->│ │ +│ │ │ │ └──────────────────────────────────┘ │ │ +│ │ ExpandVolume RPC │ │ ExpandVolume ┌──────────────────────────────────┐ │ │ +│ │------------------->│ │---------------->│ * Set xfs quota │<-->│ │ +└─────────┘ └────────┘ └──────────────────────────────────┘ └────────────────────┘ +``` + +## Legacy node server +Legacy node server runs as `DaemonSet` Pods named `legacy-node-server` in all or selected Kubernetes nodes. Each legacy node server Pod runs on a node independently. Each pod contains below running containers +* `Node driver registrar` - Registers legacy node server to kubelet to get CSI RPC calls. +* `Node server` - Honors stage, unstage, publish, unpublish and expand volume RPC requests. +* `Liveness probe` - Exposes `/healthz` endpoint to check legacy node server liveness by Kubernetes. + +Workflow diagram is same as in [node server](#node-server). diff --git a/docs/cli.md b/docs/command-reference.md similarity index 75% rename from docs/cli.md rename to docs/command-reference.md index 485ac8f38..e898729c6 100644 --- a/docs/cli.md +++ b/docs/command-reference.md @@ -1,17 +1,50 @@ -### Install Kubectl plugin - -The `directpv` kubectl plugin can be used to manage the lifecycle of volumes and drives in the kubernetes cluster - -```sh -$ kubectl krew install directpv +# Command reference + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer [installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer [installation guide](./installation.md#directpv-csi-driver-installation). + +## Note +DirectPV plugin command is referred as `kubectl directpv` in this document. For direct binary users, replace with `kubectl-directpv`. + +## Command changes from DirectCSI +| DirectCSI command | DirectPV command | +|:---------------------------------|:----------------------------------------------------------| +| `kubectl directcsi drives list` | `kubectl directpv list drives` | +| `kubectl directcsi volumes list` | `kubectl directpv list volumes` | +| `kubectl directcsi format` | `kubectl directpv discover`, then `kubectl directpv init` | + +## Global flags +Any plugin command has below global flags. + +| Flag | Argument | Description | +|:---------------|:---------|:----------------------------------------------------| +| `--kubeconfig` | _string_ | Path to the kubeconfig file to use for CLI requests | +| `--quiet` | - | Suppress printing error messages | +| `-h`, `--help` | - | help for directpv | +| `--version` | - | version for directpv | + +## Commands +List of subcommands are below + +| Subcommand | Description | +|:------------|:----------------------------------------------------------------------------------| +| `install` | Install DirectPV in Kubernetes | +| `discover` | Discover new drives | +| `init` | Initialize the drives | +| `info` | Show information about DirectPV installation | +| `list` | List drives and volumes | +| `label` | Set labels to drives and volumes | +| `cordon` | Mark drives as unschedulable | +| `uncordon` | Mark drives as schedulable | +| `migrate` | Migrate drives and volumes from legacy DirectCSI | +| `move` | Move volumes excluding data from source drive to destination drive on a same node | +| `clean` | Cleanup stale volumes | +| `remove` | Remove unused drives from DirectPV | +| `uninstall` | Uninstall DirectPV in Kubernetes | + +## `install` command ``` - -### Install DirectPV - -Install DirectPV in your kubernetes cluster - -```sh -$ kubectl directpv install --help Install DirectPV in Kubernetes USAGE: @@ -22,13 +55,14 @@ FLAGS: --tolerations strings Set toleration labels on the storage nodes (KEY[=VALUE]:EFFECT,..) --registry string Name of container registry (default "quay.io") --org string Organization name in the registry (default "minio") - --image string Name of the DirectPV image (default "directpv:0.0.0-dev") + --image string Name of the DirectPV image (default "directpv:v4.0.6") --image-pull-secrets strings Image pull secrets for DirectPV images (SECRET1,..) --apparmor-profile string Set path to Apparmor profile --seccomp-profile string Set path to Seccomp profile -o, --output string Generate installation manifest. One of: yaml|json --kube-version string Select the kubernetes version for manifest generation (default "1.27.0") --legacy Enable legacy mode (Used with '-o') + --openshift Use OpenShift specific installation -h, --help help for install GLOBAL FLAGS: @@ -56,15 +90,10 @@ EXAMPLES: 7. Install DirectPV with seccomp profile $ kubectl directpv install --seccomp-profile profiles/seccomp.json - ``` -### Discover drives - -Discover the block devices present in the cluster - -```sh -$ kubectl directpv discover --help +## `discover` command +``` Discover new drives USAGE: @@ -97,17 +126,10 @@ EXAMPLES: 5. Discover specific drives from specific nodes $ kubectl directpv discover --nodes=node{1...4} --drives=sd{a...f} - ``` -### Initialize the available drives present in the cluster - -Initializing the drives will format the selected drives with XFS filesystem and mount them to `/var/lib/directpv/mnt/`. DirectPV can then use the initialized drives for provisioning Persistent Volumes in respones to PVC with the `directpv-min-io` storage class. - -**Warning**: This command will completely and irreversibly erase the data (mkfs) in the selected disks by formatting them - -```sh -$ kubectl directpv init --help +## `init` command +``` Initialize the drives USAGE: @@ -115,6 +137,7 @@ USAGE: FLAGS: --timeout duration specify timeout for the initialization process (default 2m0s) + --dangerous Perform initialization of drives which will permanently erase existing data -h, --help help for init GLOBAL FLAGS: @@ -124,13 +147,10 @@ GLOBAL FLAGS: EXAMPLES: 1. Initialize the drives $ kubectl directpv init drives.yaml - ``` -### Show overall information about the DirectPV installation in the cluster - -```sh -$ kubectl directpv info --help +## `info` command +``` Show information about DirectPV installation USAGE: @@ -142,13 +162,35 @@ FLAGS: GLOBAL FLAGS: --kubeconfig string Path to the kubeconfig file to use for CLI requests --quiet Suppress printing error messages +``` +## `list` command ``` +List drives and volumes + +USAGE: + directpv list [command] + +FLAGS: + -n, --nodes strings Filter output by nodes; supports ellipses pattern e.g. node{1...10} + -d, --drives strings Filter output by drive names; supports ellipses pattern e.g. sd{a...z} + -o, --output string Output format. One of: json|yaml|wide + --no-headers When using the default or custom-column output format, don't print headers (default print headers) + -h, --help help for list + +GLOBAL FLAGS: + --kubeconfig string Path to the kubeconfig file to use for CLI requests + --quiet Suppress printing error messages -### List the drives initialized and managed by DirectPV +AVAILABLE COMMANDS: + drives List drives + volumes List volumes -```sh -$ kubectl directpv list drives --help +Use "directpv list [command] --help" for more information about this command. +``` + +### `drives` command +``` List drives USAGE: @@ -196,13 +238,10 @@ EXAMPLES: 8. List drives filtered by labels $ kubectl directpv list drives --labels tier=hot - ``` -### List the volumes provisioned and managed by DirectPV - -```sh -$ kubectl directpv list volumes --help +### `volumes` command +``` List volumes USAGE: @@ -260,13 +299,35 @@ EXAMPLES: 10. List volumes filtered by labels $ kubectl directpv list volumes --labels tier=hot +``` +## `label` command ``` +Set labels to drives and volumes + +USAGE: + directpv label [command] + +FLAGS: + -n, --nodes strings If present, filter objects from given nodes; supports ellipses pattern e.g. node{1...10} + -d, --drives strings If present, filter objects by given drive names; supports ellipses pattern e.g. sd{a...z} + --all If present, select all objects + --dry-run Run in dry run mode + -h, --help help for label -### Set lables on the drives managed by DirectPV +GLOBAL FLAGS: + --kubeconfig string Path to the kubeconfig file to use for CLI requests + --quiet Suppress printing error messages + +AVAILABLE COMMANDS: + drives Set labels to drives + volumes Set labels to volumes + +Use "directpv label [command] --help" for more information about this command. +``` -```sh -$ kubectl directpv label drives --help +### `drives` command +``` Set labels to drives USAGE: @@ -298,13 +359,10 @@ EXAMPLES: 3. Remove 'tier: hot' label from all drives in all nodes $ kubectl directpv label drives tier- --all - ``` -### Set labels on the volumes managed by DirectPV - -```sh -$ kubectl directpv label volumes --help +### `volumes` command +``` Set labels to volumes USAGE: @@ -339,13 +397,10 @@ EXAMPLES: 3. Remove 'tier: hot' label from all volumes in all nodes $ kubectl directpv label volumes tier- --all - ``` -### Cordon the drives to make them unschedulable - -```sh -$ kubectl directpv cordon --help +## `cordon` command +``` Mark drives as unschedulable USAGE: @@ -378,13 +433,10 @@ EXAMPLES: 5. Cordon drives which are in 'error' status $ kubectl directpv cordon --status=error - ``` -### Uncordon the cordoned drives - -```sh -$ kubectl directpv uncordon --help +## `uncordon` command +``` Mark drives as schedulable USAGE: @@ -420,13 +472,30 @@ EXAMPLES: 6. Uncordon drives which are in 'error' status $ kubectl directpv uncordon --status=error +``` +## `migrate` command ``` +Migrate drives and volumes from legacy DirectCSI + +USAGE: + directpv migrate [flags] -### Move volumes from one drive to another drive within a node (excluding data) +FLAGS: + --dry-run Run in dry run mode + -h, --help help for migrate + +GLOBAL FLAGS: + --kubeconfig string Path to the kubeconfig file to use for CLI requests + --quiet Suppress printing error messages -```sh -$ kubectl directpv move --help +EXAMPLES: +1. Migrate drives and volumes from legacy DirectCSI + $ kubectl directpv migrate +``` + +## `move` command +``` Move volumes excluding data from source drive to destination drive on a same node USAGE: @@ -445,13 +514,10 @@ GLOBAL FLAGS: EXAMPLES: 1. Move volumes from drive af3b8b4c-73b4-4a74-84b7-1ec30492a6f0 to drive 834e8f4c-14f4-49b9-9b77-e8ac854108d5 $ kubectl directpv drives move af3b8b4c-73b4-4a74-84b7-1ec30492a6f0 834e8f4c-14f4-49b9-9b77-e8ac854108d5 - ``` -### Cleanup stale (released|deleted) volumes - -```sh -$ kubectl directpv clean --help +## `clean` command +``` Cleanup stale volumes USAGE: @@ -492,13 +558,10 @@ EXAMPLES: 7. Clean volumes by pod namespace $ kubectl directpv clean --pod-namespaces=tenant-{1...3} - ``` -### Remove unused drives from DirectPV - -```sh -$ kubectl directpv remove --help +## `remove` command +``` Remove unused drives from DirectPV USAGE: @@ -531,13 +594,10 @@ EXAMPLES: 5. Remove drives are in 'error' status $ kubectl directpv remove --status=error - ``` -### Uninstall DirectPV from the kubernetes cluster - -```sh -$ kubectl directpv uninstall --help +## `uninstall` command +``` Uninstall DirectPV in Kubernetes USAGE: @@ -549,5 +609,4 @@ FLAGS: GLOBAL FLAGS: --kubeconfig string Path to the kubeconfig file to use for CLI requests --quiet Suppress printing error messages - ``` diff --git a/docs/developer.md b/docs/developer.md new file mode 100644 index 000000000..8fe9710ae --- /dev/null +++ b/docs/developer.md @@ -0,0 +1,17 @@ +# Development and Testing +We welcome any contribution to DirectPV. All your pull requests are welcomed. + +## Building DirectPV from source +1. Get source code from [DirectPV Github repository](https://github.com/minio/directpv) +2. Run `./build.sh`. You will get `directpv` CSI driver binary and `kubectl-directpv` plugin binary. + +## Building DirectPV container image +1. Run `podman build -t quay.io/minio/directpv:$(./kubectl-directpv --version | awk '{ print $NF }') .` + +## Pushing container image to kubernetes nodes +1. Run `podman save --output directpv.tar quay.io/minio/directpv:$(./kubectl-directpv --version | awk '{ print $NF }')` to get image tarball. +2. Copy `directpv.tar` to all your kubernetes nodes. +3. Import `directpv.tar` by running `/usr/local/bin/ctr images import --digests --base-name quay.io/minio/directpv directpv.tar` on each node. This step works for `K3S` cluster. + +## Installing DirectPV build +1. Run `./kubectl-directpv install` diff --git a/docs/development-and-testing.md b/docs/development-and-testing.md deleted file mode 100644 index 18a2efd2e..000000000 --- a/docs/development-and-testing.md +++ /dev/null @@ -1,154 +0,0 @@ -## Development and Testing -1. Have `quay.io` account, running docker and kubernetes. You could use any registry account and replace `quay.io` with such registry name in below steps. -2. Docker login to your `quay.io` account. -```bash -$ docker login --username https://quay.io -``` -3. Create `csi-provisioner`, `livenessprobe` and `csi-node-driver-registrar` repositories by pull/tag/push respective images to your `quay.io` account. -```bash -$ docker pull quay.io/minio/csi-provisioner:v3.4.0 && \ -docker tag quay.io/minio/csi-provisioner:v3.4.0 quay.io//csi-provisioner:v3.4.0 && \ -docker push quay.io//csi-provisioner:v3.4.0 - -$ docker pull quay.io/minio/livenessprobe:v2.9.0 && \ -docker tag quay.io/minio/livenessprobe:v2.9.0 quay.io//livenessprobe:v2.9.0 && \ -docker push quay.io//livenessprobe:v2.9.0 - -$ docker pull quay.io/minio/csi-node-driver-registrar:v2.6.3 && \ -docker tag quay.io/minio/csi-node-driver-registrar:v2.6.3 quay.io//csi-node-driver-registrar:v2.6.3 && \ -docker push quay.io//csi-node-driver-registrar:v2.6.3 - -$ docker pull quay.io/minio/csi-resizer:v1.7.0 && \ -docker tag quay.io/minio/csi-resizer:v1.7.0 quay.io//csi-resizer:v1.7.0 && \ -docker push quay.io//csi-resizer:v1.7.0 -``` -4. Make sure `csi-provisioner`, `livenessprobe`, `csi-node-driver-registrar` and `csi-resizer` repositories are `public` in your `quay.io` account. -5. Go to your DirectPV project root. -```bash -$ cd $GOPATH/src/github.com/minio/directpv -``` -6. Hack, hack, hack... -7. Run ./build.sh -```bash -$ ./build.sh -``` -8. Run docker build to tag image. -```bash -$ docker build -t quay.io//directpv: . -``` -9. Push newly created image to your `quay.io` account. -```bash -$ docker push quay.io//directpv: -``` -10. Make sure `directpv` repository is `public` in your `quay.io` account. -11. Install DirectPV. -```bash -$ ./kubectl-directpv --kubeconfig install \ ---image directpv: --org --registry quay.io -``` -12. Check running DirectPV -```bash -$ ./kubectl-directpv --kubeconfig info - -$ ./kubectl-directpv --kubeconfig list drives -``` - -## Testing with minikube - -1. Setup LVs - -The following script will create 4 LVs backed up by 4 loopback devices - -```bash -sudo truncate --size=1G /tmp/disk-{1..4}.img -for disk in /tmp/disk-{1..4}.img; do sudo losetup --find $disk; done -devices=( $(for disk in /tmp/disk-{1..4}.img; do sudo losetup --noheadings --output NAME --associated $disk; done) ) -sudo pvcreate "${devices[@]}" -vgname="vg0" -sudo vgcreate "$vgname" "${devices[@]}" -for lvname in lv-{0..3}; do sudo lvcreate --name="$lvname" --size=800MiB "$vgname"; done -``` - -2. Start minikube - -```bash -minikube start --driver=none -``` - -3. Install DirectPV - -Install the freshly built version - -```bash -./kubectl-directpv install --image directpv: --org --registry quay.io -``` - -4. Discover and initialize the drives - -```bash -./kubectl-directpv discover --output-file drives.yaml -./kubectl-directpv init drives.yaml -``` - -5. Check if the drives are showing up - -```bash -./kubectl-directpv list drives -``` - -6. Apply the minio.yaml file - -Download and apply a sample MinIO deployment file available [here](https://github.com/minio/directpv/blob/master/functests/minio.yaml) - -```bash -kubectl apply -f minio.yaml -``` - -7. Check if the pods are up and running - -```bash -kubectl get pods -``` - -8. Check the volumes - -```bash -./kubectl-directpv list volumes -``` - -9. Check the drives contain volumes - -```bash -./kubectl-directpv list drives -``` - -10. Uninstall the MinIO deployment - -```bash -kubectl delete -f minio.yaml -``` - -11. Delete the PVCs - -```bash -kubectl delete pvc --all -``` - -After deleting the PVCs, check if the drives are freed up. - -12. Release freed drives - -```bash -./kubectl-directpv remove --all -``` - -13. Cleanup LV setup - -```sh -sudo lvremove vg0 -y -sudo vgremove vg0 -y -sudo pvremove /dev/loop /dev/loop /dev/loop /dev/loop # n can be replaced with the loopbacks created -sudo losetup --detach-all -``` - -Please refer [here](./troubleshooting.md) for any trouble shooting guidelines. diff --git a/docs/drain-node.md b/docs/drain-node.md deleted file mode 100644 index 21ee8a1ec..000000000 --- a/docs/drain-node.md +++ /dev/null @@ -1,27 +0,0 @@ -Drain a node -------------- - -Draining will forcefully remove the DirectPV resources from a node. This [bash script](./tools/drain.sh) can be used for draining and has to be executed cautiously as it is an irreversible operation and may incur data loss. - -You can consider draining a node in the following circumstances - -#### When a node is detached from kubernetes - -If a node which was used by DirectPV is detached from kubernetes, the DirectPV resources from that node will remain intact until the resources are drained. - -#### When DirectPV is unselected to run on a specific node - -If a node which was used by DirectPV is decided to be a "non-storage" node. - -For example, If DirectPV is decided to not run on a specific node by changing the node-selectors like the following example - -```sh -$ kubectl directpv uninstall -$ kubectl directpv install --node-selector node-label-key=node-label-value -``` - -the resources from the detached node can then be cleaned up by - -```sh -./drain.sh -``` diff --git a/docs/drive-management.md b/docs/drive-management.md new file mode 100644 index 000000000..2adb59747 --- /dev/null +++ b/docs/drive-management.md @@ -0,0 +1,109 @@ +# Drive management + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer [installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer [installation guide](./installation.md#directpv-csi-driver-installation). + +## Add drives +Drives are added to DirectPV to provision volumes. This involves two steps process. Below are the steps. + +1. Run `discover` command - `discover` command probes eligible drives from DirectPV nodes and stores drive information in YAML file. You should carefully examine the YAML file and set `select` field to `yes` or `no` value to indicate drive selection. `select` field is set to `yes` value by default. Below is an example of `discover` command +```sh +# Probe and save drive information to drives.yaml file. +$ kubectl directpv discover + + Discovered node 'master' ✔ + Discovered node 'node1' ✔ + +┌─────────────────────┬────────┬───────┬─────────┬────────────┬──────┬───────────┬─────────────┐ +│ ID │ NODE │ DRIVE │ SIZE │ FILESYSTEM │ MAKE │ AVAILABLE │ DESCRIPTION │ +├─────────────────────┼────────┼───────┼─────────┼────────────┼──────┼───────────┼─────────────┤ +│ 252:16$ud8mwCjPT... │ master │ vdb │ 512 MiB │ - │ - │ YES │ - │ +│ 252:16$gGz4UIuBj... │ node1 │ vdb │ 512 MiB │ - │ - │ YES │ - │ +└─────────────────────┴────────┴───────┴─────────┴────────────┴──────┴───────────┴─────────────┘ + +Generated 'drives.yaml' successfully. + +# Show generated drives.yaml file. +$ cat drives.yaml +version: v1 +nodes: + - name: master + drives: + - id: 252:16$ud8mwCjPTH8147TysmiQ2GGLpffqUht6bz7NtHqReJo= + name: vdb + size: 536870912 + make: "" + select: "yes" + - name: node1 + drives: + - id: 252:16$gGz4UIuBjQlO1KibOv7bZ+kEDk3UCeBneN/UJdqdQl4= + name: vdb + size: 536870912 + make: "" + select: "yes" + +``` + +2. Run `init` command - `init` command creates a request to add selected drives in the YAML file generated using `discover` command. As this process wipes out all data on the selected drives, wrong drive selection will lead to permanent data loss. Below is an example of `init` command + +```sh +$ kubectl directpv init drives.yaml + + ███████████████████████████████████████████████████████████████████████████ 100% + + Processed initialization request 'c24e22f5-d582-49ba-a883-2ce56909904e' for node 'master' ✔ + Processed initialization request '7e38a453-88ed-412c-b146-03eef37b23bf' for node 'node1' ✔ + +┌──────────────────────────────────────┬────────┬───────┬─────────┐ +│ REQUEST_ID │ NODE │ DRIVE │ MESSAGE │ +├──────────────────────────────────────┼────────┼───────┼─────────┤ +│ c24e22f5-d582-49ba-a883-2ce56909904e │ master │ vdb │ Success │ +│ 7e38a453-88ed-412c-b146-03eef37b23bf │ node1 │ vdb │ Success │ +└──────────────────────────────────────┴────────┴───────┴─────────┘ +``` + +Refer [discover command](./command-reference.md#discover-command) and [init command](./command-reference.md#init-command) for more information. + +## List drives +To get information of drives from DirectPV, just run `list drives` command. Below is an example + +```sh +$ kubectl directpv list drives +┌────────┬──────┬──────┬─────────┬─────────┬─────────┬────────┐ +│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ +├────────┼──────┼──────┼─────────┼─────────┼─────────┼────────┤ +│ master │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +│ node1 │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +└────────┴──────┴──────┴─────────┴─────────┴─────────┴────────┘ +``` + +Refer [list drives command](./command-reference.md#drives-command) for more information. + +## Label drives +Drives are labeled to set custom tagging which can be used in volume provisioning. Below is an example +```sh +# Set label 'tier' key to 'hot' value. +$ kubectl directpv label drives tiet=hot + +# Remove label 'tier'. +$ kubectl directpv label drives tier- +``` + +Refer [label drives command](./command-reference.md#drives-command-1) for more information. + +## Replace drive +Faulty drive can be replaced with a new drive on a same node. In this process, all volumes in the faulty drive are moved to the new drive then faulty drive is removed from DirectPV. Currently DirectPV does not support moving volume data to the new drive. Use [replace.sh](./tools/replace.sh) script to perform drive replacement. Below is an example +```sh +# Replace 'sdd' drive by 'sdf' drive on 'node1' node +$ replace.sh sdd sdf node1 +``` + +## Remove drives +Drives they are not part of any volume can be removed. Below is an example +```sh +# Remove drive 'vdb' from 'node1' node +$ kubectl directpv remove --drives=vdb --nodes=node1 +``` + +Refer [remove command](./command-reference.md#remove-command) for more information. diff --git a/docs/drive-replacement.md b/docs/drive-replacement.md deleted file mode 100644 index 593d92090..000000000 --- a/docs/drive-replacement.md +++ /dev/null @@ -1,114 +0,0 @@ -Drive replacement (applies for versions > v3.2.2) -------------- - -These are the steps to move the volumes from one drive to another drive from the **same node**. These steps will be useful during drive replacement. - -(NOTE: This will just move the volume references from source to the destination drive without moving the actual data) - -## STEP-BY-STEP Guide for drive replacement - -### STEP 1: Add new drive - -Once the new drive has been attached to the node as a replacement of an older drive, Discover and initialize the newly added drive by executing the following steps - -```sh -$ kubectl directpv discover --drives /dev/ --nodes -``` - -This command will generate `drives.yaml` which will be used later for initialization. Please refer [Discover CLI](./cli.md#discover-drives) for more helpers on discovery. - -Now, initialize the drive using the `drives.yaml` file that was created by the previous command - -```sh -$ kubectl directpv init drives.yaml -``` - -On success, you should see the new drive listed in `kubectl directpv list drives --drives /dev/` - -### STEP 2: Get source and destination drive IDs - -```sh -$ kubectl directpv list drives --drives , --nodes -o wide -``` - -### STEP 3: Cordon source and destination drives - -By cordoning, we mark the drives as unschedulable so that no new volumes will be scheduled on these drives - -```sh -$ kubectl directpv cordon -``` - -### STEP 4: Get the pods used by the source drive - -```sh -$ kubectl directpv list volumes --drives /dev/ --nodes -``` - -### STEP 5: Cordon the corresponding kubernetes node and delete the pods used by the source drive - -Cordon the corresponding k8s node and delete the corresponding workload pods used by the source drive, this will take the workloads offline and also doesn't schedule any new pods on this node. - -```sh -$ kubectl cordon -``` - -(NOTE: Avoid force deleting here as force deletion might not unpublish the volumes) - -```sh -$ kubectl delete pod/ -n -``` - -This pod will fall in "pending" state after deleting as the node is cordoned - -### STEP 6: Wait for associated DirectPV volumes to be unbounded - -There shouldn't be any volumes in "Bounded" state - -```sh -$ kubectl directpv list volumes --pod-names= --pod-namespaces= -``` - -### STEP 7: Run move command - -```sh -$ kubectl directpv move -``` - -After moving, list the drives to see if the volumes got moved to destination - -``` -$ kubectl directpv list drives --drives , --nodes -o wide -``` - -### STEP 8: Uncordon the destination drive - -```sh -$ kubectl directpv uncordon -``` - -### STEP 9: Uncordon the kubernetes node - -```sh -$ kubectl uncordon -``` - -Now, the corresponding workload pods should be coming up as the node is uncordoned. This pod will be using the new drive. - -### STEP 10 (Optional): You can remove the old drive which was replaced - -This will release the drive as it is not used anymore - -```sh -$ kubectl directpv remove -``` - -## Bash script for drive replacement - -Alternatively, after the "STEP 2" in the previous section, you can use this [bash script](./tools/replace.sh) to move the drives. - -#### Example Usage - -```sh -./replace.sh -``` diff --git a/docs/faq.md b/docs/faq.md deleted file mode 100644 index e3b6ab562..000000000 --- a/docs/faq.md +++ /dev/null @@ -1,72 +0,0 @@ -FAQ ----- - -#### What type of disks are recommended for DirectPV? - -DirectPV is specifically meant for [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage) such as hard drives, solid-state drives and JBODs etc. - -Avoid using DirectPV with SAN and NAS based storage options, as they inherently involve extra network hops in the data path. This leads to poor performance and increased complexity. - -#### How DirectPV is different from LocalPV and HostPath? - -Hostpath volumes are ephemeral and are tied to the lifecyle of pods. Hostpath volumes are lost when a pod is restarted or deleted, resulting in the loss of any stored data. - -DirectPV volumes are persistent through node and pod reboots. The lifecycle of a DirectPV volume is managed by the associated [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). - -LocalPVs are statically provisioned from a local storage resource on the nodes where persistent volumes are required, and must be created prior to the workloads using them. User management is required down to creating the actual object resource for the local PV. - -DirectPV also creates statically provisioned volume resources, but does not require user intervention for the creation of the object resource. Instead, DirectPV dynamically provisions the persistent volume in response to a PVC requesting a DirectPV storage volume. This significantly reduces complexity of management. - -#### How are the disks selected for a Pod? - -DirectPV operates only on those disks which it [explicitly manages](./cli.md#initialize-the-available-drives-present-in-the-cluster) - - 1. DirectPV selects managed disks local to the node where the pod is scheduled. This provides direct access for the pods to the disks. - - 2. DirectPV runs a selection algorithm to choose a disk for a volume creation request - - 3. DirectPV then creates a sub-directory for the volume with quota set on the sub-directory for the requested volume size - - 4. DirectPV publishes the volume to the pod. - - To know more on the selection algorithm, please refer [here](./volume-scheduling.md). - -#### What does drive initialization do? - -DirectPV command `kubectl directpv init` command will prepare the drives by formatting them with XFS filesystem and mounting them in a desired path (/var/lib/directpv/mnt/). Upon success, these drives will be ready for volumes to be scheduled and you can see the initialized drives in `kubectl directpv list drives`. - -#### What are the conditions that a drive must satisfy to be used for DirectPV? - -A drive must meet the following requirements for it to be used for initialization - -- The size of the drive should not be less then 512MiB. -- The drive must not be hidden. (Check /sys/class/block//hidden, It should not be "1") -- The drive must not be a read-only. -- The drive must not be partitioned. -- The drive must not be held by other devices. (Check /sys/class/block//holders, It should not have any entries) -- The drive must not be mounted. -- The drive must not have swap-on enabled. -- The drive must not be a CDROM. - -DirectPV can support any storage resource presenting itself as a block device. While this includes LVMs, LUKs, LUNs, and similar virtualized drives, we do not recommend using DirectPV with those in production environments. - -#### What does error "no drive found for requested topology" mean? - -This error can be seen in PVC and pod description when - -- Kubernetes scheduled the pod on a node where DirectPV doesn't have any drives initialized. This can be fixed by using necessary selectors in the workload to make k8s schedule the pods on the correct set of nodes. Please refer [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) to know more on assigning the pod to specific nodes. -- The requested topology specifications like size, zone etc cannot be satisified by any of initialized drives in the selected node. This can be fixed by changing the topology parameter in the workloads to suitable value. - -#### Is `direct-csi-min-io` storage class still supported? - -No, the support for `direct-csi-min-io` is removed from DirectPV version v4.0.0. No new volumes will be provisioned using `direct-csi-min-io` storage class. However, the existing volumes which are already provisioned by `direct-csi-min-io` storage class will still be managed by the latest DirectPV versions. - -Alternatively, you can use `directpv-min-io` storage class for provisioning new volumes. And there are no functional or behavioral differences between 'direct-csi-min-io' and 'directpv-min-io'. - -#### Does DirectPV support volume snapshotting? - -No, DirectPV doesn't support CSI snapshotting. DirectPV is specifically meant for use cases like MinIO where the data availability and resiliency is taken care by the application itself. Additionally, with the AWS S3 versioning APIs and internal healing, snapshots isn't required. - -#### Does DirectPV support _ReadWriteMany_? - -No, DirectPV doesn't support _ReadWriteMany_. The workloads using DirectPV run local to the node and are provisioned from local disks in the node. This makes the workloads to direcly access the data path without any additional network hops unlike remote volumes (network PVs). The additional network hops may lead to poor performance and increases the complexity. So, DirectPV doesn't support _ReadWriteMany_ and only supports _ReadWriteOnce_. diff --git a/docs/index.md b/docs/index.md deleted file mode 100644 index 73e9a0d3b..000000000 --- a/docs/index.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Documentation ---- - -Documentation -------------- - -### Setup and Usage - - [Installation](./installation.md) - - [CLI reference](./cli.md) - - [Scheduling](./scheduling.md) - - [Version upgrade](./upgrade.md) - -### Advanced - - [Architecture](./architecture.md) - - [Security](./security.md) - - [CSI Specification](./specification.md) - - [Metrics](./metrics.md) - - [Troubleshooting](./troubleshooting.md) - - [Drive States](./drive-states.md) - - - -### References - - [Kubernetes CSI](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) diff --git a/docs/installation.md b/docs/installation.md index 965b6de9e..69d9db4a8 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,150 +1,103 @@ ---- -title: Installation ---- +# Installation +DirectPV comes with two components. +1. DirectPV plugin - installed on client machine. +2. DirectPV CSI driver - installed on Kubernetes cluster. -Installation -------------- +## DirectPV plugin installation +The plugin needs to be installed to manage DirectPV CSI driver in Kubernetes. ### Prerequisites +* Access to Kubernetes cluster. -| Name | Version | -| -------------|----------| -| kubectl | v1.19+ | -| kubernetes | v1.19+ | - - -### Plugin Installation - -DirectPV plugin can be installed from kubectl krew index using: - +### Installation using `Krew` +The latest DirectPV plugin is available in `Krew` repository. Use below steps to install the plugin in your system. ```sh -kubectl krew install directpv -``` - -After running this installation: - - - The `directpv` plugin will be installed in your krew installation directory (default: `$HOME/.krew`) - - Run `kubectl directpv --version` to verify that the installation worked - - If the error `Error: unknown command "directpv" for "kubectl"` is shown, try adding `$HOME/.krew/bin` to your `$PATH` - -### Driver Installation - -For installation in production grade environments, ensure that all criteria in the [Production Readiness Checklist](#production-readiness-checklist) are satisfied. - -(NOTE: Please refer the [CLI Guide](./cli.md) for more helpers on the following commands.) +# Update the plugin list. +$ kubectl krew update -#### 1. Install the driver - -```sh -kubectl directpv install +# Install DirectPV plugin. +$ kubectl krew install directpv ``` -This will install `directpv-min-io` CSI driver in the kubernetes cluster. - -**Notes:** - - - DirectPV components are installed in the namespace `directpv` - - alternate kubeconfig path can be specified using `kubectl directpv --kubeconfig /path/to/kubeconfig` - - the rbac requirements for the driver is [here](./specification.md#driver-rbac) - - the driver runs in `privileged` mode, which is required for mounting, unmounting and formatting drives - - The deamonset used by directpv will be using the ports `10443` for metrics and `30443` for readiness handlers. Make sure these ports are open on the nodes. - -#### 2. List discovered drives - +### Installation of release binary +The plugin binary name starts by `kubectl-directpv` and is available at https://github.com/minio/directpv/releases/latest. Download the binary as per your operating system and architecture. Below is an example for `GNU/Linux` on `amd64` architecture. ```sh -kubectl directpv discover -``` +# Download DirectPV plugin. +$ release=$(curl -sfL "https://api.github.com/repos/minio/directpv/releases/latest" | awk '/tag_name/ { print substr($2, 3, length($2)-4) }') +$ curl -fLo kubectl-directpv https://github.com/minio/directpv/releases/download/v${release}/kubectl-directpv_${release}_linux_amd64 -This will list all available drives in the kubernetes cluster and will generate an init config file (default: drives.yaml) to initialize these drives. +# Make the binary executable. +$ chmod a+x kubectl-directpv +``` -#### 3. Intitialize the drives +## DirectPV CSI driver installation +Before starting the installation, it is required to have DirectPV plugin installed on your system. For plugin installation refer [this documentation](#directpv-plugin-installation). If you are not using `krew`, replace `kubectl directpv` by `kubectl-directpv` in below steps. +### Prerequisites +* Kubernetes >= v1.18 on GNU/Linux on amd64, arm64 or ppc64le. +* If you use private registry, below images must be pushed into your registry. You could use [this helper script](tools/push-images.sh) to do that. + - quay.io/minio/csi-node-driver-registrar:v2.6.3 + - quay.io/minio/csi-provisioner:v3.4.0 _(for Kubernetes >= v1.20)_ + - quay.io/minio/csi-provisioner:v2.2.0-go1.18 _(for kubernetes < v1.20)_ + - quay.io/minio/livenessprobe:v2.9.0 + - quay.io/minio/csi-resizer:v1.7.0 + - quay.io/minio/directpv:latest +* If `seccomp` is enabled, load [DirectPV seccomp profile](../seccomp.json) on nodes where you want to install DirectPV and use `--seccomp-profile` flag to `kubectl directpv install` command. For more information, refer Kubernetes documentation [here](https://kubernetes.io/docs/tutorials/clusters/seccomp/) +* If `apparmor` is enabled, load [DirectPV apparmor profile](../apparmor.profile) on nodes where you want to install DirectPV and use `--apparmor-profile` flag to `kubectl directpv install` command. For more information, refer Kubernetes documentation [here](https://kubernetes.io/docs/tutorials/clusters/apparmor/) +* Enabled `ExpandCSIVolumes` [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) for [volume expansion](https://kubernetes-csi.github.io/docs/volume-expansion.html) feature. +* Review [driver specification](./specification.md) +* For Red Hat Openshift users, refer [Openshift documentation](./openshift.md) for configuration prior to install DirectPV. + +### Default installation +To install DirectPV in all Kubernetes nodes, just run ```sh -kubectl directpv init drives.yaml +$ kubectl directpv install ``` -This will initialize selected drives in drives.yaml - -**Notes:** - - - initialization will erase all data on the drives. Double check to make sure that only intended drives are specified - -#### 4. Verify installation +### Customized installation +To install DirectPV on selected Kubernetes nodes and/or with tolerations and/or with non-standard `kubelet` directory, use below steps accordingly. +#### Installing on selected nodes +To install DirectPV on selected nodes, use `--node-selector` flag to `install` command. Below is an example ```sh -kubectl directpv info +# Install DirectPV on nodes having label 'group-name' key and 'bigdata' value +$ kubectl directpv install --node-selector group-name=bigdata ``` -This will show information about the drives formatted and added to DirectPV. - -After running this installation: - - - storage class named `directpv-min-io` is created - - `directpv-min-io` can be specified as the `storageClassName` in in `PodSpec.VolumeClaimTemplates` to provision DirectPV volumes - - example statefulset using `directpv-min-io` can be found [here](../minio.yaml#L61) - - optional: view the [driver specification](./specification.md) - - -## Air-gapped Installation (Private Registry) - -Push the following images to your private registry - - - quay.io/minio/csi-node-driver-registrar:v2.6.3 - - quay.io/minio/csi-provisioner:v3.4.0 - - quay.io/minio/livenessprobe:v2.9.0 - - quay.io/minio/csi-resizer:v1.7.0 - - quay.io/minio/directpv:${latest_tag_name} - - **Notes:** - - - If your kubernetes version is less than v1.20, you need push `quay.io/minio/csi-provisioner:v2.2.0-go1.18` - -Here is a shell script to Copy-Paste into your terminal to do the above steps: +#### Installing on tainted nodes +To install DirectPV on tainted nodes, use `--toleration` flag to `install` command. Below is an example ```sh -/bin/bash -e - -# set this to private registry URL (the URL should NOT include http or https) -if [ -z $PRIVATE_REGISTRY_URL ]; then "PRIVATE_REGISTRY_URL env var should be set"; fi - -images[0]=quay.io/minio/csi-node-driver-registrar:v2.6.3 -images[1]=quay.io/minio/csi-provisioner:v3.4.0 -images[2]=quay.io/minio/livenessprobe:v2.9.0 -images[3]=quay.io/minio/csi-resizer:v1.7.0 -images[4]=quay.io/minio/directpv:$(curl -s "https://api.github.com/repos/minio/directpv/releases/latest" | grep tag_name | sed -E 's/.*"([^"]+)".*/\1/') +# Install DirectPV on tainted nodes by tolerating 'key1' key, 'Equal' operator for 'value1' value with 'NoSchedule' effect +$ kubectl directpv install --tolerations key1=value1:NoSchedule -function privatize(){ echo $1 | sed "s#quay.io#${PRIVATE_REGISTRY_URL}#g"; } -function pull_tag_push(){ docker pull $1 && docker tag $1 $2 && docker push $2; } -for image in ${images[*]}; do pull_tag_push $image $(privatize $image); done +# Install DirectPV on tainted nodes by tolerating 'key2' key, 'Exists' operator with 'NoExecute' effect +$ kubectl directpv install --tolerations key2:NoExecute ``` -## Custom Installation - -If any other customization is desired, - -Step 1: Generate the specification for installing DirectPV +#### Installing on non-standard `kubelet` directory +To install on non-standard `kubelet` directory, set `KUBELET_DIR_PATH` environment variable and start the installation. Below is an example ```sh -$ kubectl directpv install -o yaml > directpv-install.yaml +$ export KUBELET_DIR_PATH=/path/to/my/kubelet/dir +$ kubectl directpv install ``` -Step 2: Make appropriate changes to the resources -``` -$ emacs directpv-install.yaml +#### Installing on Openshift +To install DirectPV on Openshift with specific configuration, use `--openshift` flag. Below is an example +```sh +$ kubectl directpv install --openshift ``` -Step 3: Install DirectPV -``` -$ kubectl create -f directpv-install.yaml +#### Installing by generating DirectPV manifests +To install using generated manifests file, run below command +```sh +$ curl -sfL https://github.com/minio/directpv/raw/master/docs/tools/install.sh | sh - apply ``` -**Notes:** - -- Client-side upgrade functionality will not be available for custom installations. You need to execute `kubectl directpv migrate` to migrate the old resources. - -## Production Readiness Checklist - -Make sure the following check-boxes are ticked before production deployment +## What's next +* [Add drives](./drive-management.md#add-drives) +* [Provision volumes](./volume-provisioning.md) - - [ ] If using a private registry, all the images listed in [air-gapped installation](#air-gapped-installation-private-registry) should be available in the private registry - - [ ] If seccomp is enabled in the system, DirectPV [seccomp policy](../seccomp.json) should be loaded on all nodes. Instructions available [here](https://kubernetes.io/docs/tutorials/clusters/seccomp/) - - [ ] If apparmor is enabled in the system, DirectPV [apparmor profile](../apparmor.profile) should be loaded on all nodes. Instructions available [here](https://kubernetes.io/docs/tutorials/clusters/apparmor/) - - [ ] Review and Sign-off the [Security Checklist](../security-checklist.md) for providing elevated privileges to DirectPV. +## Further reads +* [Drive management guide](./drive-mangement.md) +* [Volume management guide](./volume-management.md) +* [Troubleshooting guide](./troubleshooting.md) diff --git a/docs/security.md b/docs/issue-reporting.md similarity index 79% rename from docs/security.md rename to docs/issue-reporting.md index 5d24e9744..83dc891b8 100644 --- a/docs/security.md +++ b/docs/issue-reporting.md @@ -1,23 +1,13 @@ ---- -title: Security ---- +# Issue reporting -Security ---------- +## Reporting bugs +Please open Github issue at https://github.com/minio/directpv/issues -**Important**: Report security issues to security@min.io. Please do not report security issues here. +## Reporting security issues -## Supported Versions +The [latest release](https://github.com/minio/minio/releases/latest) of DirectPV comes with security and vulnerability fixes in previous releases. It is strongly recommended to use the latest version. Before reporting security/vulnerability issues, make sure those issues are not fixed in the latest release. -We always provide security updates for the [latest release](https://github.com/minio/minio/releases/latest). -Whenever there is a security update you just need to upgrade to the latest version. - -## Reporting a Vulnerability - -All security bugs in [minio/directpv](https://github,com/minio/directpv) (or other minio/* repositories) -should be reported by email to security@min.io. Your email will be acknowledged within 48 hours, -and you'll receive a more detailed response to your email within 72 hours indicating the next steps -in handling your report. +All security issues should be reported by email to security@min.io. Your email will be acknowledged within 48 hours, and you'll receive a more detailed response to your email within 72 hours indicating the next steps in handling your report. DO NOT OPEN GITHUB ISSUES for security bugs. Please, provide a detailed explanation of the issue. In particular, outline the type of the security issue (DoS, authentication bypass, information disclose, ...) and the assumptions you're making (e.g. do @@ -47,15 +37,14 @@ This process can take some time, especially when coordination is required with m Every effort will be made to handle the bug in as timely a manner as possible, however it's important that we follow the process described above to ensure that disclosures are handled consistently. -## Vulnerability Management Policy -------------------------------------- +### Vulnerability Management Policy This document formally describes the process of addressing and managing a reported vulnerability that has been found in the MinIO server code base, any directly connected ecosystem component or a direct / indirect dependency of the code base. -### Scope +#### Scope The vulnerability management policy described in this document covers the process of investigating, assessing and resolving a vulnerability report @@ -64,7 +53,7 @@ opened by a MinIO employee or an external third party. Therefore, it lists pre-conditions and actions that should be performed to resolve and fix a reported vulnerability. -### Vulnerability Management Process +#### Vulnerability Management Process The vulnerability management process requires that the vulnerability report contains the following information: diff --git a/docs/metrics.md b/docs/monitoring.md similarity index 57% rename from docs/metrics.md rename to docs/monitoring.md index b9bbfb35e..6090c3120 100644 --- a/docs/metrics.md +++ b/docs/monitoring.md @@ -1,22 +1,34 @@ ---- -title: Metrics ---- +# Monitoring using Prometheus -Monitoring guidelines ----------------------- +DirectPV nodes export Prometheus compatible metrics data via port `10443`. The metrics data includes +* directpv_stats_bytes_used +* directpv_stats_bytes_total +and categorized by labels `tenant`, `volumeID` and `node`. -DirectPV nodes export Prometheus compatible metrics data by exposing a metrics endpoint at /directpv/metrics. Users looking to monitor their tenants can point Prometheus configuration to scrape data from this endpoint. +To scrape data in Prometheus, each node must be accessible by port `10443`. A simple example is below -DirectPV node server exports the following metrics +1. Make node server metrics port accessible by localhost:8080 +``` +$ kubectl -n directpv port-forward node-server-4nd6q 8080:10443 +``` -- directpv_stats_bytes_used -- directpv_stats_bytes_total +2. Add below YAML configuration into Prometheus configuration. +```yaml +scrape_configs: + - job_name: 'directpv-monitor' + # Override the global default and scrape targets from this job every 5 seconds. + scrape_interval: 5s + static_configs: + - targets: ['localhost:8080'] + labels: + group: 'production' +``` -These metrics are categorized by labels ['tenant', 'volumeID', 'node']. These metrics will be representing the volume stats of the published volumes. +3. Run `directpv_stats_bytes_total{node="node-3"}` promQL in Prometheus web interface. -Please apply the following Prometheus config to scrape the metrics exposed. +Below is an example comprehensive YAML configuration. -``` +```yaml global: scrape_interval: 15s external_labels: @@ -67,17 +79,3 @@ scrape_configs: action: replace target_label: kubernetes_name ``` - -For example, use the following promQL to query the volume stats. - -- To filter out the volumes scheduled in `node-3` node :- - -``` -directpv_stats_bytes_total{node="node-3"} -``` - -- To filter out the volumes of tenant `tenant-1` scheduled in `node-5` node :- - -``` -directpv_stats_bytes_used{tenant="tenant-1", node="node-5"} -``` diff --git a/docs/node-management.md b/docs/node-management.md new file mode 100644 index 000000000..a05c78d1e --- /dev/null +++ b/docs/node-management.md @@ -0,0 +1,33 @@ +# Node management + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer [installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer [installation guide](./installation.md#directpv-csi-driver-installation). + +## Add node +After Adding a node into DirectPV DaemonSet, run DirectPV plugin `discover` command. Below is an example +```sh +$ kubectl directpv discover +``` + +## List node +Run DirectPV plugin `info` command to get list of nodes. Below is an example +```sh +$ kubectl directpv info +┌──────────┬──────────┬───────────┬─────────┬────────┐ +│ NODE │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │ +├──────────┼──────────┼───────────┼─────────┼────────┤ +│ • master │ 512 MiB │ 32 MiB │ 2 │ 1 │ +│ • node1 │ 512 MiB │ 32 MiB │ 2 │ 1 │ +└──────────┴──────────┴───────────┴─────────┴────────┘ + +64 MiB/1.0 GiB used, 4 volumes, 2 drives + +``` + +Refer [info command](./command-reference.md#info-command) for more information. + +## Delete node +***CAUTION: THIS IS DANGEROUS OPERATION WHICH LEADS TO DATA LOSS*** + +Before removing a node make sure no volumes or drives on the node are in us, then remove the node from DirectPV DaemonSet and run [remove-node.sh](./tools/remove-node.sh) script. diff --git a/docs/openshift.md b/docs/openshift.md new file mode 100644 index 000000000..97e19eb69 --- /dev/null +++ b/docs/openshift.md @@ -0,0 +1,9 @@ +# DirectPV on Red Hat OpenShift + +## Settings required +* Create new project like command `$ oc new-project my-directpv-installation --description="My DirectPV installation for local volume provisioning" --display-name="DirectPV"` +* Add privileges to `directpv` namespace and DirectPV service account by adding `system:serviceaccount:directpv:directpv-min-io` to users by command `$ oc edit scc privileged` + +## Limitations +* DirectPV does not support volume snapshot feature as per CSI specification. DirectPV is specifically meant for use cases like MinIO where the data availability and resiliency is taken care by the application itself. Additionally, with the AWS S3 versioning APIs and internal healing, snapshots is not a requirement. +* DirectPV does not support `ReadWriteMany` volume access mode. The workloads using DirectPV run local to the node and are provisioned from local storage drives in the node. This makes the workloads to directly access data without any additional network hops unlike remote volumes, network PVs, etc. The additional network hops may lead to poor performance and increases the complexity. With `ReadWriteOnce` access mode, DirectPV provides high performance storage for Pods. diff --git a/docs/scheduling.md b/docs/scheduling.md deleted file mode 100644 index 3ff0cd6ca..000000000 --- a/docs/scheduling.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Scheduling ---- - -Scheduling guidelines -------------- - -### Volume scheduling based on drive labels - -In addition to scheduling based on resource constraints (available space) and node topology (affinity/anti-affinity etc.), it is possible to further influence the scheduling of workloads to specific volumes based on drive labels. The DirectPV drives can be labeled based on its classification and such labels can be used in the storage class parameters to control scheduling to pick up the chosen drives for volumes. - -By default, the DirectPV drives will not have any user defined labels set on them. Use `kubectl directpv label drives` command to set user defined labels to DirectPV drives. - -**Notes:** - -- This applies only for creating new volumes as this is a schedule time process. -- In the following example, please replace the place holders ``,`` and `` with appropriate values based on the classification you chose - -#### Step 1: Set the label on the DirectPV drive(s) - -```sh -kubectl directpv label drives = --drives /dev/ -``` - -To Verify if the labels are properly set, list the drives with `--show-labels` flag - -```sh -kubectl directpv list drives --drives /dev/ --show-labels -``` - -#### Step 2: Set the 'directpv-min-io/: ' parameter in storage class definition - -Create a storage class with the following parameter set - -```yaml -parameters: - directpv.min.io/: -``` - -For example, create a new storage class with the `parameters` section - -(NOTE: Please refer the exiting storage class `kubectl get storageclass directpv-min-io -n directpv -o yaml` to compare and check if all the fields are present on the new storage class) - -```yaml -allowVolumeExpansion: false -allowedTopologies: -- matchLabelExpressions: - - key: directpv.min.io/identity - values: - - directpv-min-io -apiVersion: storage.k8s.io/v1 -kind: StorageClass -metadata: - finalizers: - - foregroundDeletion - labels: - application-name: directpv.min.io - application-type: CSIDriver - directpv.min.io/created-by: kubectl-directpv - directpv.min.io/version: v1beta1 - name: directpv-min-io-new # Please choose any storage class name of your choice - resourceVersion: "511457" - uid: e93d8dab-b182-482f-b8eb-c69d4a1ec62d -parameters: - fstype: xfs - directpv.min.io/: -provisioner: directpv-min-io -reclaimPolicy: Delete -volumeBindingMode: WaitForFirstConsumer -``` - -#### Step 3: Deploy the workload with the new storage class name set - -You will see volumes placed on the labeled drives only. You can verify this by the following command - -```sh -kubectl directpv list drives --labels : -kubectl directpv list volumes -``` diff --git a/docs/specification.md b/docs/specification.md index 6205a8eb3..b060fe911 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -1,10 +1,6 @@ ---- -title: Driver Specification ---- +# DirectPV CSI Driver Specification -## Driver Specification - -### CSIDriver +## CSIDriver | Key | Value | |---------------------|---------------------------| @@ -16,55 +12,55 @@ title: Driver Specification | `storageCapacity` | `false` | | `modes` | `Persistent`, `Ephemeral` | -### StorageClass +## StorageClass -| Key | Value | -|------------------------|--------------------------| -| `name` | `directpv-min-io` | -| `provisioner` | `directpv-min-io` | -| `reclaimPolicy` | `Delete` | -| `allowVolumeExpansion` | `false` | -| `volumeBindingMode` | `WaitForFirstConsumer` | +| Key | Value | +|------------------------|------------------------| +| `name` | `directpv-min-io` | +| `provisioner` | `directpv-min-io` | +| `reclaimPolicy` | `Delete` | +| `allowVolumeExpansion` | `true` | +| `volumeBindingMode` | `WaitForFirstConsumer` | -### DirectPVDrives CRD +## DirectPVDrives CRD -| Key | Value | -| -------------|-----------------------| -| `name` | `directpvdrives` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|-------------------| +| `name` | `directpvdrives` | +| `apigroup` | `directpv.min.io` | -### DirectPVVolumes CRD +## DirectPVVolumes CRD -| Key | Value | -| -------------|-----------------------| -| `name` | `directpvvolumes` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|-------------------| +| `name` | `directpvvolumes` | +| `apigroup` | `directpv.min.io` | -### DirectPVNodes CRD +## DirectPVNodes CRD -| Key | Value | -| -------------|-----------------------| -| `name` | `directpvnodes` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|-------------------| +| `name` | `directpvnodes` | +| `apigroup` | `directpv.min.io` | -### DirectPVInitRequests CRD +## DirectPVInitRequests CRD -| Key | Value | -| -------------|------------------------| -| `name` | `directpvinitrequests` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|------------------------| +| `name` | `directpvinitrequests` | +| `apigroup` | `directpv.min.io` | -### Driver RBAC +## Driver RBAC -| apiGroup | Resources | Verbs | -| --------------------------|-----------------------------|------------------------------------------------------| -| (core) | `endpoints` | `get`, `list`, `watch`, `create`, `update`, `delete` | -| (core) | `events` | `list`, `watch`, `create`, `update`, `patch` | -| (core) | `nodes` | `get`, `list`, `watch` | -| (core) | `persistentvolumes` | `get`, `list`, `watch`, `create`, `delete` | -| (core) | `persistentvolumeclaims` | `get`, `list`, `watch`, `update` | -| (core) | `pods,pod` | `get`, `list`, `watch` | -| `policy` | `podsecuritypolicies` | `use` | +| apiGroup | Resources | Verbs | +|---------------------------|-----------------------------|------------------------------------------------------| +| (core) | `endpoints` | `get`, `list`, `watch`, `create`, `update`, `delete` | +| (core) | `events` | `list`, `watch`, `create`, `update`, `patch` | +| (core) | `nodes` | `get`, `list`, `watch` | +| (core) | `persistentvolumes` | `get`, `list`, `watch`, `create`, `delete` | +| (core) | `persistentvolumeclaims` | `get`, `list`, `watch`, `update` | +| (core) | `pods,pod` | `get`, `list`, `watch` | +| `policy` | `podsecuritypolicies` | `use` | | `apiextensions.k8s.io` | `customresourcedefinitions` | `get`, `list`, `watch`, `create`, `update`, `delete` | | `coordination.k8s.io` | `leases` | `get`, `list`, `watch`, `update`, `delete`, `create` | | `directpv.min.io` | `directpvdrives` | `get`, `list`, `watch`, `create`, `update`, `delete` | @@ -77,5 +73,4 @@ title: Driver Specification | `storage.k8s.io` | `storageclasses` | `get`, `list`, `watch` | | `storage.k8s.io` | `volumeattachments` | `get`, `list`, `watch` | - -The service account binded to the above clusterrole is `directpv-min-io` in `directpv` namespace and the corresponding clusterrolebinding is `directpv-min-io` +The service account binded to the above clusterrole is `directpv-min-io` in `directpv` namespace and the corresponding clusterrolebinding is `directpv-min-io`. diff --git a/docs/tools/create-storage-class.sh b/docs/tools/create-storage-class.sh new file mode 100755 index 000000000..fc4739c96 --- /dev/null +++ b/docs/tools/create-storage-class.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# +# This file is part of MinIO DirectPV +# Copyright (c) 2023 MinIO, Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +set -e -C -o pipefail + +declare NAME DRIVE_LABEL + +function init() { + if [[ $# -ne 2 ]]; then + cat < + +ARGUMENTS: + NAME new storage class name. + DRIVE-LABEL drive labels to be attached. + +EXAMPLE: + # Create new storage class 'fast-tier-storage' with drive labels 'directpv.min.io/tier: fast' + $ create-storage-class.sh fast-tier-storage 'directpv.min.io/tier: fast' +EOF + exit 255 + fi + + NAME="$1" + DRIVE_LABEL="$2" + + if ! which kubectl >/dev/null 2>&1; then + echo "kubectl not found; please install" + exit 255 + fi +} + +function main() { + kubectl apply -f - <. - -# -# This script drains the DirectPV resources from a selected node -# -# **CAUTION** -# -# This operation is irreversible and may incur data loss if not used cautiously. -# - -set -e -C -o pipefail - - -function drain() { - selector="directpv.min.io/node=${2}" - - # unset the finalizers - kubectl get "${1}" --selector="${selector}" -o custom-columns=NAME:.metadata.name --no-headers | while read -r resource_name; do - kubectl patch "${1}" "${resource_name}" -p '{"metadata":{"finalizers":null}}' --type=merge - done - - # delete the objects - kubectl delete "${1}" --selector="${selector}" --ignore-not-found=true -} - -function init() { - - if [[ $# -ne 1 ]]; then - echo "usage: drain.sh " - echo - echo "This script forcefully removes all the DirectPV resources from the node" - echo "This operation is irreversible and may incur data loss if not used cautiously." - exit 255 - fi - - if ! which kubectl >/dev/null 2>&1; then - echo "kubectl not found; please install" - exit 255 - fi - - if kubectl get csinode "${1}" -o go-template="{{range .spec.drivers}}{{if eq .name \"directpv-min-io\"}}{{.name}}{{end}}{{end}}" --ignore-not-found | grep -q .; then - echo "the node is still under use by DirectPV CSI Driver; please remove DirectPV installation from the node to drain" - exit 255 - fi -} - -function main() { - node="$1" - - drain "directpvvolumes" "${node}" - drain "directpvdrives" "${node}" - drain "directpvinitrequests" "${node}" - kubectl delete directpvnode "${node}" --ignore-not-found=true -} - -init "$@" -main "$@" diff --git a/docs/tools/push-images.sh b/docs/tools/push-images.sh new file mode 100755 index 000000000..baa8bc517 --- /dev/null +++ b/docs/tools/push-images.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# +# This file is part of MinIO DirectPV +# Copyright (c) 2023 MinIO, Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +# +# This script pushes DirectPV and its sidecar images to private registry. +# + +set -o errexit +set -o nounset +set -o pipefail + +declare registry podman + +function init() { + if [ "$#" -ne 1 ]; then + cat < + +ARGUMENT: + Image registry without 'http' prefix. + +EXAMPLE: +$ push-images.sh example.org/myuser +EOF + exit 255 + fi + registry="$1" + + if which podman >/dev/null 2>&1; then + podman=podman + elif which docker >/dev/null 2>&1; then + podman=docker + else + echo "no podman or docker found; please install" + exit 255 + fi +} + +# usage: push_image +function push_image() { + image="$1" + private_image="${image/quay.io\/minio/$registry}" + echo "Pushing image ${image}" + "${podman}" pull --quiet "${image}" + "${podman}" tag "${image}" "${private_image}" + "${podman}" push --quiet "${private_image}" +} + +function main() { + push_image "quay.io/minio/csi-node-driver-registrar:v2.6.3" + push_image "quay.io/minio/csi-provisioner:v3.4.0" + push_image "quay.io/minio/csi-provisioner:v2.2.0-go1.18" + push_image "quay.io/minio/livenessprobe:v2.9.0" + push_image "quay.io/minio/csi-resizer:v1.7.0" + release=$(curl -sfL "https://api.github.com/repos/minio/directpv/releases/latest" | awk '/tag_name/ { print substr($2, 3, length($2)-4) }') + push_image "quay.io/minio/directpv:v${release}" +} + +init "$@" +main "$@" diff --git a/docs/tools/remove-node.sh b/docs/tools/remove-node.sh new file mode 100755 index 000000000..4880180bd --- /dev/null +++ b/docs/tools/remove-node.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# +# This file is part of MinIO DirectPV +# Copyright (c) 2023 MinIO, Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +set -e -C -o pipefail + +declare NODE + +function delete_resource() { + resource="$1" + selector="directpv.min.io/node=${NODE}" + + # unset the finalizers + kubectl get "${resource}" --selector="${selector}" -o custom-columns=NAME:.metadata.name --no-headers | while read -r name; do + kubectl patch "${resource}" "${name}" -p '{"metadata":{"finalizers":null}}' --type=merge + done + + # delete the objects + kubectl delete "${resource}" --selector="${selector}" --ignore-not-found=true +} + +function init() { + if [[ $# -ne 1 ]]; then + cat < + +This script forcefully removes all the DirectPV resources from the node. +CAUTION: Remove operation is irreversible and may incur data loss if not used cautiously. +EOF + exit 255 + fi + + if ! which kubectl >/dev/null 2>&1; then + echo "kubectl not found; please install" + exit 255 + fi + + NODE="$1" + + if kubectl get --ignore-not-found=true csinode "${NODE}" -o go-template='{{range .spec.drivers}}{{if eq .name "directpv-min-io"}}{{.name}}{{break}}{{end}}{{end}}' | grep -q .; then + echo "node ${NODE} is still in use; remove node ${NODE} from DirectPV DaemonSet and try again" + exit 255 + fi +} + +function main() { + delete_resource directpvvolumes + delete_resource directpvdrives + delete_resource directpvinitrequests + kubectl delete directpvnode "${NODE}" --ignore-not-found=true +} + +init "$@" +main "$@" diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 000000000..a3d77485c --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,67 @@ +# Frequently Asked Questions +* [DirectPV installation fails in my Kubernetes. Why?](#directpv-installation-fails-in-my-kubernetes-why) +* [After upgrading DirectPV to v4.x.x, I do not find `direct-csi-min-io` storage class. Why?](#after-upgrading-directpv-to-v4xx-i-do-not-find-direct-csi-min-io-storage-class-why) +* [In the YAML output of `discover` command, I do not find my storage drive(s). Why?](#in-the-yaml-output-of-discover-command-i-do-not-find-my-storage-drives-why) +* [Do you support SAN, NAS, iSCSI, network drives etc.,?](#do-you-support-san-nas-iscsi-network-drives-etc) +* [Do you support LVM, Linux RAID, Hardware RAID, Software RAID etc.,?](#do-you-support-lvm-linux-raid-hardware-raid-software-raid-etc) +* [Is LUKS device supported?](#is-luks-device-supported) +* [I am already using Local Persistent Volumes (Local PV) for storage. Why do I need DirectPV?](#i-am-already-using-local-persistent-volumes-local-pv-for-storage-why-do-i-need-directpv) +* [I see `no drive found ...` error message in my Persistent Volume Claim. Why?](#i-see-no-drive-found--error-message-in-my-persistent-volume-claim-why) +* [I see Persistent Volume Claim is created, but respective DirectPV volume is not created. Why?](#i-see-persistent-volume-claim-is-created-but-respective-directpv-volume-is-not-created-why) +* [I see volume consuming Pod still in `Pending` state. Why?](#i-see-volume-consuming-pod-still-in-pending-state-why) +* [I see `volume XXXXX is not yet staged, but requested with YYYYY` error. Why?](#i-see-volume-xxxxx-is-not-yet-staged-but-requested-with-yyyyy-error-why) +* [I see ```unable to find device by FSUUID xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; either device is removed or run command `sudo udevadm control --reload-rules && sudo udevadm trigger` on the host to reload``` error. Why?](#i-see-unable-to-find-device-by-fsuuid-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-either-device-is-removed-or-run-command-sudo-udevadm-control---reload-rules--sudo-udevadm-trigger-on-the-host-to-reload-error-why) + +### DirectPV installation fails in my Kubernetes. Why? +You need to have necessary privileges and permissions to perform installation. Go though [specifications](./specifications.md) here. For Red Hat OpenShift, refer [this](./openshift.md). + +### After upgrading DirectPV to v4.x.x, I do not find `direct-csi-min-io` storage class. Why? +Legacy DirectCSI is deprecated including storage class `direct-csi-min-io` and it is no longer supported, but previously created volumes are continued to work normally. For new volume requests, please use `directpv-min-io` storage class. + +### In the YAML output of `discover` command, I do not find my storage drive(s). Why? +DirectPV ignores certain drives those met any one of the below conditions +* The size of the drive is less than 512MiB. +* The drive is hidden. +* The drive is read-only. +* The drive is parititioned. +* The drive is held by other devices. +* The drive is mounted or in use by DirectPV already. +* The drive is in-use swap partition. +* The drive is a CDROM. + +Check last column of `discover --all` command output to what conditions are met. Resolve the conditions and retry again. + +### Do you support SAN, NAS, iSCSI, network drives etc.,? +DirectPV is meant for high performance local volumes with Direct Attached Storage. We do not recommend any remote drives which may lead to poor performance. + +### Do you support LVM, Linux RAID, Hardware RAID, Software RAID etc.,? +It works, but we strongly recommend to use raw devices for better performance. + +### Is LUKS device supported? +Yes + +### I am already using Local Persistent Volumes (Local PV) for storage. Why do I need DirectPV? +Local Persistent Volumes are ephemeral and are tied to the lifecyle of pods. These volumes are lost when a pod is restarted or deleted; this may lead to data loss. Additionally Local Persistent Volumes lifecycle requires administrative skills. Whereas DirectPV dynamically provisions volumes on-demand; they are persistent through pod/node restarts. The lifecycle of DirectPV volumes are managed by associated Persistent Volume Claims (PVCs) which simplifies volume management. + +### I see `no drive found ...` error message in my Persistent Volume Claim. Why? +Below is the reason and solution +| Reason | Solution | +|:-------------------------------------------------------------|:---------------------------------------------------| +| Volume claim is made without adding any drive into DirectPV. | Please add drives. | +| No drive has free space for requested size. | 1. Please add new drives. 2. Remove stale volumes. | +| Requested topology is not met. | Please modify your Persistent Volume Claim. | +| Requested drive is not found on the requested node. | Please modify Persistent Volume Claim. | +| Requested node is not DirectPV node. | Please modify Persistent Volume Claim. | + +### I see Persistent Volume Claim is created, but respective DirectPV volume is not created. Why? +DirectPV comes with [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) volume binding mode i.e. Pod consuming volume must be scheduled first. + +### I see volume consuming Pod still in `Pending` state. Why? +* If you haven't created respective Persistent Volume Claim, create it. +* You may be facing Kubernetes scheduling problem. Please refer Kubernetes documentation [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) + +### I see `volume XXXXX is not yet staged, but requested with YYYYY` error. Why? +According to CSI specification, `Kubelet` should call `StageVolume` RPC first, then `PublishVolume` RPC next. In a rare event, `StageVolume` RPC is not fired/called, but `PublishVolume` RPC is called. Please restart your Kubelet and report this issue to your Kubernetes provider. + +### I see ```unable to find device by FSUUID xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; either device is removed or run command `sudo udevadm control --reload-rules && sudo udevadm trigger` on the host to reload``` error. Why? +In a rare event, `Udev` in your system missed updating `/dev` directory. Please run command `sudo udevadm control --reload-rules && sudo udevadm trigger` and report this issue to your OS vendor. diff --git a/docs/upgrade.md b/docs/upgrade.md index 6c2d355c5..aeb983c7f 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -1,41 +1,42 @@ ---- -title: Upgrade ---- +# Upgrade DirectPV installation -Version Upgrade ---------------- +## Upgrade DirectPV CSI driver -### Guidelines to upgrade to the latest DirectPV version - -DirectPV version upgrades are seameless and transparent. The resources will be upgraded automatically when you run the latest version over the existing resources. The latest version of DirectPV should be available in [krew](https://github.com/kubernetes-sigs/krew-index). For more details on the installation, Please refer the [Installation guidelines](./installation.md). - -NOTE: For the users who don't prefer krew, Please find the latest images in [releases](https://github.com/minio/directpv/releases). - - -#### Upgrade from versions < v3.2.x - -If you are on DirectPV version < 3.2.x, it is recommended to upgrade to v3.2.2 and then to the latest - -Please follow https://github.com/minio/directpv/blob/master_old/docs/upgrade.md#upgrade-to-v300 for the upgrade steps from legacy versions - - -#### Upgrade from v3.2.x - -In the latest version of DirectPV, the CSI sidecar images have been updated. If private registry is used for images, please make sure the following images are available in your registry before upgrade. +### Upgrade DirectPV CSI driver v4.x.x +#### Offline upgrade +Below steps to be followed for offline upgrade +1. Uninstall DirectPV CSI driver. +```sh +$ kubectl directpv uninstall ``` -quay.io/minio/csi-node-driver-registrar:v2.6.3 -quay.io/minio/csi-provisioner:v3.4.0 -quay.io/minio/livenessprobe:v2.9.0 -quay.io/minio/csi-resizer:v1.7.0 +2. Upgrade DirectPV plugin by [this documentation](#upgrade-directpv-plugin). +3. Install the latest DirectPV CSI driver by [this documentation](./installation.md#directpv-csi-driver-installation). + +#### In-place upgrade +Below steps to be followed for in-place upgrade +1. Upgrade DirectPV plugin by [this documentation](#upgrade-directpv-plugin). +2. Run install script with appropriate node-selector, tolerations and KUBELET_DIR_PATH. Below is an example +```sh +$ curl -sfL https://github.com/minio/directpv/raw/master/docs/tools/install.sh | sh - apply ``` -**Notes:** - -- If your kubernetes version is less than v1.20, you need push `quay.io/minio/csi-provisioner:v2.2.0-go1.18` +### Upgrade legacy DirectCSI CSI driver v3.x.x and v2.0.9 +Below steps to be followed to upgrade to the latest DirectPV CSI driver. +1. Uninstall legacy DirectCSI CSI driver. +```sh +$ kubectl directcsi uninstall +``` +2. Install the latest DirectPV plugin by [this documentation](./installation.md#directpv-plugin-installation) or upgrade existing DirectPV plugin by [this documentation](#upgrade-directpv-plugin). +3. Install the latest DirectPV CSI driver by [this documentation](./installation.md#directpv-csi-driver-installation). -- If you are on DirectPV versions < v4.0.0 and if you are using any custom storage classes for controlling volume scheduling based on access-tiers as explained [here](https://github.com/minio/directpv/blob/master_old/docs/scheduling.md), you need to make the following change to these custom storage classes. +## Upgrade DirectPV plugin -You need to change `direct.csi.min.io/access-tier: ` to `directpv.min.io/access-tier: ` in the respective storage class parameters section. +### Upgrade using `Krew` +To upgrade the plugin, just run below command +```sh +$ kubectl krew upgrade directpv +``` -- The older CRDs (directcsidrives and directcsivolumes) are deprecated and not used in versions > v4.0.0, it can be removed after upgrading. Please use the [bash script](./tools/remove-directcsi.sh) to remove the older objects after upgrading to latest. +### Upgrade of release binary +Refer binary installation documentation [here](./installation.md#installation-of-release-binary). diff --git a/docs/volume-expansion.md b/docs/volume-expansion.md deleted file mode 100644 index 038fc8c8e..000000000 --- a/docs/volume-expansion.md +++ /dev/null @@ -1,24 +0,0 @@ -Volume Expansion (applies for versions > v3.2.2) ----------------- - -DirectPV supports volume expansion [CSI feature](https://kubernetes-csi.github.io/docs/volume-expansion.html) from versions above v3.2.2. With this support, the DirectPV provisioned volumes can be expanded to the requested size claimed by the PVC. DirectPV supports "online" volume expansion where the workloads will not have any downtimes during the expansion process. - -Volume expansion requires `ExpandCSIVolumes` feature gate to be enabled (enabled by default in k8s v1.16 and above). For more details on the feature gates, please check [here](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/). - -To expand the volume, edit the storage size in the corresponding PVC spec - -```yaml -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: # Set the new size here - storageClassName: directpv-min-io - volumeMode: Filesystem - volumeName: pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -``` - -DirectPV would expand the volume and reflect the new size in `kubectl directpv list volumes pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -o wide`. The corresponding PVC would remain in "Bounded" state with the expanded size. - -(NOTE: As DirectPV supports "online" volume expansion, workload pods doesn't require restarts) diff --git a/docs/volume-management.md b/docs/volume-management.md new file mode 100644 index 000000000..9058d2e78 --- /dev/null +++ b/docs/volume-management.md @@ -0,0 +1,150 @@ +# Volume management + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer [installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer [installation guide](./installation.md#directpv-csi-driver-installation). +* Added drives in DirectPV. Refer [drive management guide](./drive-management.md) + +## Add volume +Refer [volume provisioning guide](./volume-provisioning.md) + +## List volume +To get information of volumes from DirectPV, just run `list volumes` command. Below is an example + +```sh +$ kubectl directpv list drives +┌────────┬──────┬──────┬─────────┬─────────┬─────────┬────────┐ +│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ +├────────┼──────┼──────┼─────────┼─────────┼─────────┼────────┤ +│ master │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +│ node1 │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +└────────┴──────┴──────┴─────────┴─────────┴─────────┴────────┘ +``` + +Refer [list volumes command](./command-reference.md#volumes-command) for more information. + +## Expand volume +DirectPV supports online volume expansion which does not require restart of pods using those volumes. This is automatically done after setting expanded size to `Persistent Volume Claim`. Below is an example. +```sh +# Get 'minio-data-1-minio-0' Persistent volume claim. +$ kubectl get pvc minio-data-1-minio-0 -o yaml > minio-data-1-minio-0.yaml + +$ cat minio-data-1-minio-0.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + annotations: + pv.kubernetes.io/bind-completed: "yes" + pv.kubernetes.io/bound-by-controller: "yes" + volume.beta.kubernetes.io/storage-provisioner: directpv-min-io + volume.kubernetes.io/selected-node: master + volume.kubernetes.io/storage-provisioner: directpv-min-io + creationTimestamp: "2023-06-08T04:46:02Z" + finalizers: + - kubernetes.io/pvc-protection + labels: + app: minio + name: minio-data-1-minio-0 + namespace: default + resourceVersion: "76360" + uid: d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 16Mi + storageClassName: directpv-min-io + volumeMode: Filesystem + volumeName: pvc-d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +status: + accessModes: + - ReadWriteOnce + capacity: + storage: 16Mi + phase: Bound + +# Edit 'minio-data-1-minio-0' PVC to increase the size from 16MiB to 64MiB. +$ cat minio-data-1-minio-0.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + annotations: + pv.kubernetes.io/bind-completed: "yes" + pv.kubernetes.io/bound-by-controller: "yes" + volume.beta.kubernetes.io/storage-provisioner: directpv-min-io + volume.kubernetes.io/selected-node: master + volume.kubernetes.io/storage-provisioner: directpv-min-io + creationTimestamp: "2023-06-08T04:46:02Z" + finalizers: + - kubernetes.io/pvc-protection + labels: + app: minio + name: minio-data-1-minio-0 + namespace: default + resourceVersion: "76360" + uid: d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 64Mi # <--- increase size here + storageClassName: directpv-min-io + volumeMode: Filesystem + volumeName: pvc-d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +status: + accessModes: + - ReadWriteOnce + capacity: + storage: 16Mi + phase: Bound + +# Apply changes +$ kubectl apply -f minio-data-1-minio-0.yaml + +# After successful expansion, you will see updated YAML +$ kubectl get pvc minio-data-1-minio-0 -o yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + annotations: + pv.kubernetes.io/bind-completed: "yes" + pv.kubernetes.io/bound-by-controller: "yes" + volume.beta.kubernetes.io/storage-provisioner: directpv-min-io + volume.kubernetes.io/selected-node: master + volume.kubernetes.io/storage-provisioner: directpv-min-io + creationTimestamp: "2023-06-08T04:46:02Z" + finalizers: + - kubernetes.io/pvc-protection + labels: + app: minio + name: minio-data-1-minio-0 + namespace: default + resourceVersion: "76651" + uid: d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 64Mi + storageClassName: directpv-min-io + volumeMode: Filesystem + volumeName: pvc-d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +status: + accessModes: + - ReadWriteOnce + capacity: + storage: 64Mi # <--- increased size here + phase: Bound +``` + +## Delete volume +***CAUTION: THIS IS DANGEROUS OPERATION WHICH LEADS TO DATA LOSS*** + +Volume can be deleted only if it is in `Ready` state i.e. no pod is using it. Run `kubectl delete pvc` command which triggers DirectPV volume deletion. As removing a volume leads to data loss, you have to double check on what volume you are deleting. Below is an example +```sh +# Delete `sleep-pvc` volume +kubectl delete pvc sleep-pvc +``` diff --git a/docs/volume-provisioning.md b/docs/volume-provisioning.md new file mode 100644 index 000000000..3b215ff68 --- /dev/null +++ b/docs/volume-provisioning.md @@ -0,0 +1,131 @@ +# Volume provisioning +Volume provisioning involves making a `PersistentVolumeClaim` using default `directpv-min-io` storage class or custom storage class having `directpv-min-io` provisioner. As the storage classes come with volume binding mode [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode); this mode delays volume binding and provisioning of a `PersistentVolume` until a `Pod` using the `PersistentVolumeClaim` is created. PersistentVolumes will be selected or provisioned conforming to the topology that is specified by the Pod's scheduling constraints. These include, but are not limited to, resource requirements, node selectors, pod affinity and anti-affinity, and taints and tolerations. Pods consuming volumes are scheduled to nodes where volumes were scheduled. This ensures high performance data access to the pod. + +DirectPV volume in `Ready` state indicates that the volume is ready for binding to the pod. After binding, `Bound` state is set to the volume. + +## Making Persistent volume claim +PV claim must be defined with specific parameters in `PersistentVolumeClaim` specification. These parameters are + +| Parameter | Value | +|:-------------------|:---------------------------------------------------------------------------------| +| `volumeMode` | `Filesystem` | +| `storageClassName` | `directpv-min-io` or any storage class name having `directpv-min-io` provisioner | +| `accessModes` | `[ "ReadWriteOnce" ]` | + +Below is an example claiming `8MiB` storage from `directpv-min-io` storage class for `sleep-pvc` PVC. +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: sleep-pvc +spec: + volumeMode: Filesystem + storageClassName: directpv-min-io + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 8Mi +``` + +For `WaitForFirstConsumer` volume binding mode, a pod consuming `sleep-pvc` must be defined. Below is an example which uses `sleep-volume` mounted on `/mnt` +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sleep-pod +spec: + volumes: + - name: sleep-volume + persistentVolumeClaim: + claimName: sleep-pvc + containers: + - name: sleep-container + image: example.org/test/sleep:v0.0.1 + volumeMounts: + - mountPath: "/mnt" + name: sleep-volume +``` + +## Making Persistent volume claim in StatefulSet +PV claim must be defined with specific parameters in `volumeClaimTemplates` specification. These parameters are + +| Parameter | Value | +|:-------------------|:---------------------------------------------------------------------------------| +| `storageClassName` | `directpv-min-io` or any storage class name having `directpv-min-io` provisioner | +| `accessModes` | `[ "ReadWriteOnce" ]` | + +Below is an example claiming two `16MiB` storage from `directpv-min-io` storage class for `minio-data-1` and `minio-data-2` PVC to two `minio` pods. + +```yaml +kind: Service +apiVersion: v1 +metadata: + name: minio + labels: + app: minio +spec: + selector: + app: minio + ports: + - name: minio + port: 9000 + +--- + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: minio + labels: + app: minio +spec: + serviceName: "minio" + replicas: 2 + selector: + matchLabels: + app: minio + template: + metadata: + labels: + app: minio + directpv.min.io/organization: minio + directpv.min.io/app: minio-example + directpv.min.io/tenant: tenant-1 + spec: + containers: + - name: minio + image: minio/minio + env: + - name: MINIO_ACCESS_KEY + value: minio + - name: MINIO_SECRET_KEY + value: minio123 + volumeMounts: + - name: minio-data-1 + mountPath: /data1 + - name: minio-data-2 + mountPath: /data2 + args: + - "server" + - "http://minio-{0...1}.minio.default.svc.cluster.local:9000/data{1...2}" + volumeClaimTemplates: + - metadata: + name: minio-data-1 + spec: + storageClassName: directpv-min-io + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 16Mi + - metadata: + name: minio-data-2 + spec: + storageClassName: directpv-min-io + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 16Mi +``` + +## Further reads +* [volume scheduling guide](./volume-scheduling.md) diff --git a/docs/volume-scheduling.md b/docs/volume-scheduling.md index 858d7b71f..9d91b13fd 100644 --- a/docs/volume-scheduling.md +++ b/docs/volume-scheduling.md @@ -1,8 +1,8 @@ # Volume Scheduling -`DirectPV` comes with storage class named `directpv-min-io` with volume binding mode [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode). This mode delays volume binding and provisioning of a `PersistentVolume` until a `Pod` using the `PersistentVolumeClaim` is created. PersistentVolumes will be selected or provisioned conforming to the topology that is specified by the Pod's scheduling constraints. These include, but are not limited to, resource requirements, node selectors, pod affinity and anti-affinity, and taints and tolerations. +`DirectPV` comes with default storage class `directpv-min-io` or custom storage class having `directpv-min-io` provisioner with volume binding mode [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode). This volume binding mode delays volume binding and provisioning of a `PersistentVolume` until a `Pod` using the `PersistentVolumeClaim` is created. PersistentVolumes will be selected or provisioned conforming to the topology that is specified by the Pod's scheduling constraints. These include, but are not limited to, resource requirements, node selectors, pod affinity and anti-affinity, and taints and tolerations. -## Drive selection +## Drive selection algorithm DirectPV CSI controller selects suitable drive for `CreateVolume` request like below 1. Filesystem type and/or access-tier in the request is validated. DirectPV supports `xfs` filesystem only. @@ -19,44 +19,76 @@ DirectPV CSI controller selects suitable drive for `CreateVolume` request like b 9. In case of parallel requests and the same drive is selected, step (6) succeeds for any one of the request and fails for rest of the requests by Kubernetes. ```text - +--------------+ - No | Is Valid | - +------------| CreateVolume | - | | request? | - | +==============+ - | | Yes - | +=======V=======+ Loop In - | | Loop each |-----------------+ - | | DirectPVDrive | | - | | CRD |<--+ +-----V-----+ - | +===============+ | | Is | +-------------+ - | | Loop Out | | requested | Yes | Return this | -+----V---+ +------V------+ | | volume |----->| drive | -| Return | Yes | Is no drive | | | present? | +-------------+ -| error |<-------| matched? | | +===========+ -+--------+ +=============+ | | No - | No | +-----V-----+ -+-----------+ +------V-------+ | No | Match by | -| Return | No | Is more than | |<------| capacity? | -| the first |<----| one drive | | +===========+ -| drive | | matched? | | | Yes -+-----^-----+ +==============+ | +-------V-------+ - | | Yes | No | Match by | - | +------V--------+ |<----| access-tier | - | | Filter drives | | | if requested? | - | | by maximum | | +===============+ - | | free capacity | | | Yes - | +---------------+ | +-------V-------+ - | | | | Match by | - | +------V-------+ | No | topology | - | No | Is more than | |<----| constraints | - +-----------| one drive | | | if requested? | - | matched? | | +===============+ - +==============+ | | Yes - | Yes | +-----V-----+ - +------V-----+ | | Append to | - | Randomly | +<-----| matched | - | select | | drives | - | a drive | +-----------+ - +------------+ + ╭╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╮ + No │ Is Valid │ + +-----------│ CreateVolume │ + | │ request? │ + | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ + | | Yes + | ╒═══════V═══════╕ Loop In + | │ Loop each │-----------------+ + | │ DirectPVDrive │ | + | │ CRD │<--+ ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ ┌─────────────┐ + | ╘═══════════════╛ | │ Is requested │ Yes │ Return this │ + | | Loop Out | │volume present?│----->│ drive │ +┌─────V─────┐ ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ └─────────────┘ +│ Return │ Yes │ Is no drive │ | | No +│ error │<----│ matched? │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ +└───────────┘ ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ | No │ Match by │ + | No |<----│ capacity? │ +┌───────────┐ ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ +│ Return │ No │ Is more than │ | | Yes +│ the first │<----│ one drive │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ +│ drive │ │ matched? │ | No │ Match by │ +└─────^─────┘ ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ |<----│ access-tier │ + | | Yes | │ if requested? │ + | ┌───────V───────┐ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ + | │ Filter drives │ | | Yes + | │ by maximum │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ + | │ free capacity │ | │ Match by │ + | └───────────────┘ | No │ topology │ + | | |<----│ constraints │ + | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | │ if requested? │ + | No │ Is more than │ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ + +-----------│ one drive │ | | Yes + │ matched? │ | ┌───────V───────┐ + ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ | │ Append to │ + | Yes +<----│ matched drives│ + ┌───────V───────┐ └───────────────┘ + │ Return │ + │ Randomly │ + │ selected drive│ + └───────────────┘ +``` + +## Customizing drive selection +Apart from controlling drive selection based on node selectors, pod affinity and anti-affinity, and taints and tolerations, drive labels are used to instruct DirectPV to pick up specific drives with custom storage class for volume scheduling. Below steps are involved for this process. + +* Label selected drives by [label drives](./command-reference.md#drives-command-1) command. Below is an example +```sh +# Label 'nvme1n1' drive in all nodes as 'fast' value to 'tier' key. +$ kubectl directpv label drives --drives=nvme1n1 tier=fast +``` + +* Create new storage class with drive labels using [create-storage-class.sh script](../tools/create-storage-class.sh). Below is an example +```sh +# Create new storage class 'fast-tier-storage' with drive labels 'directpv.min.io/tier: fast' +$ create-storage-class.sh fast-tier-storage 'directpv.min.io/tier: fast' +``` + +* Use newly created storage class in [volume provisioning](./volume-provisioning.md). Below is an example +```sh +$ kubectl apply -f - <