diff --git a/README.md b/README.md index b41aa5d70..85c5523c7 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,12 @@ -DirectPV ----------- +# DirectPV -DirectPV is a CSI driver for [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage). In a simpler sense, it is a distributed persistent volume manager, and not a storage system like SAN or NAS. It is useful to *discover, format, mount, schedule and monitor* drives across servers. Since Kubernetes `hostPath` and `local` PVs are statically provisioned and limited in functionality, DirectPV was created to address this limitation. +[DirectPV](https://github.com/minio/directpv) is a [CSI](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) driver for [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage). In a simpler sense, it is a distributed persistent volume manager, and not a storage system like SAN or NAS. It is useful to *discover, format, mount, schedule and monitor* drives across servers. Distributed data stores such as object storage, databases and message queues are designed for direct attached storage, and they handle high availability and data durability by themselves. Running them on traditional SAN or NAS based CSI drivers (Network PV) adds yet another layer of replication/erasure coding and extra network hops in the data path. This additional layer of disaggregation results in increased-complexity and poor performance. ![Architecture Diagram](https://github.com/minio/directpv/blob/master/docs/images/architecture.png?raw=true) -### Quickstart guide +## Quickstart 1. Install DirectPV Krew plugin ```sh @@ -24,86 +23,25 @@ $ kubectl directpv install $ kubectl directpv info ``` -4. Discover and add drives for volume scheduling. +4. Add drives ```sh -# Discover drives to check the available devices in the cluster to initialize -# The following command will create an init config file (default: drives.yaml) which will be used for initialization +# Probe and save drive information to drives.yaml file. $ kubectl directpv discover -# Review the drives.yaml for drive selections and initialize those drives +# Initialize selected drives. $ kubectl directpv init drives.yaml ``` -(NOTE: XFS is the filesystem used for formatting the drives here) - -5. Get list of added drives -```sh -$ kubectl directpv list drives -``` - -6. Deploy a demo MinIO server - -DirectPV enforces node constraints where it allocates storage based on the worker node where a pod deploys. If the pod deploys to a worker node with no or insufficient DirectPV-managed drives, DirectPV cannot allocate storage to that pod. DirectPV does not allocate storage from one node to a pod on another node. - -Modify the YAML to reflect the node and storage distribution of your Kubernetes cluster. - +5. Deploy a demo MinIO server ```sh -# This should create MinIO pods and PVCs using the `directpv-min-io` storage class -$ kubectl apply -f functests/minio.yaml +$ curl -sfL https://github.com/minio/directpv/raw/master/functests/minio.yaml | kubectl apply -f - ``` -For air-gapped setups and advanced installations, please refer to the [Installation Guide](./docs/installation.md). - -### Upgrade - -Firstly, it is required to uninstall older version of DirectPV. Once it is uninstalled, follow [Installation instructions](#Installation) to install the latest DirectPV. In this process, all existing drives and volumes will be migrated automatically. - -#### Upgrade using krew - -Refer the following steps for upgrading DirectPV using krew - -```sh -# Uninstall existing DirectPV installation -$ kubectl directpv uninstall - -# Upgrade directpv plugin via krew -$ kubectl krew upgrade directpv - -# Install latest DirectPV -$ kubectl directpv install -``` - -#### Upgrade from DirectPV v3.2.x - -For migrating from older versions < v3.2.0, Please refer the [Upgrade Guide](./docs/upgrade.md) - -### Security - -Please review the [security checklist](./security-checklist.md) before deploying to production. - -**Important**: Report security issues to security@min.io. Please do not report security issues here. - -### Additional Resources - -- [Installation Guide](./docs/installation.md) -- [Upgrade Guide](./docs/upgrade.md) -- [CLI Guide](./docs/cli.md) -- [Security Guide](./docs/security.md) -- [Scheduling Guide](./docs/scheduling.md) -- [Drive Replacement Guide](./docs/drive-replacement.md) -- [Volume Expansion](./docs/volume-expansion.md) -- [Drain a node](./docs/drain-node.md) -- [Driver Specification](./docs/specification.md) -- [Monitoring & Metrics](./docs/metrics.md) -- [Developer Guide](./docs/development-and-testing.md) -- [FAQ](./docs/faq.md) - -### Join Community - -DirectPV is a MinIO project. You can contact the authors over the slack channel: - -- [MinIO Slack](https://slack.min.io/) +## Further information +Refer [detailed documentation](./docs/README.md) -### License +## Join Community +DirectPV is a MinIO project. You can contact the authors over the [slack channel](https://slack.min.io/) -DirectPV is released under GNU AGPLv3 license. Please refer to the LICENSE document for a complete copy of the license. +## License +DirectPV is released under GNU AGPLv3 license. Refer the [LICENSE document](https://github.com/minio/directpv/blob/master/LICENSE) for a complete copy of the license. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..6cc9de95d --- /dev/null +++ b/docs/README.md @@ -0,0 +1,24 @@ +# DirectPV +DirectPV is a CSI driver to provision local volumes to Pods using [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage). It comes with two components. +1. `DirectPV plugin` - To be installed on local machine, to manage DirectPV CSI driver. +2. `DirectPV CSI driver` - To be installed on kubernetes cluster, to provision local volumes. + +## For new users +Refer below documentation +* [Installation guide](./installation.md) +* [Drive management guide](./drive-management.md) +* [Volume provisioning guide](./volume-provisioning.md) +* [Volume management guide](./volume-management.md) +* [Command reference guide](./command-reference.md) +* [Monitoring guide](./monitoring.md) + +## For existing users +Refer below documentation +* [Upgrade guide](./upgrade.md) +* [Node management guide](./node-management.md) + +## Further references +* [Volume scheduling guide](./volume-scheduling.md) +* [Issue reporting](./issue-reporting.md) +* [DirectPV CSI driver specification](./specification.md) +* [Troubleshooting guide](./faq.md) diff --git a/docs/architecture.md b/docs/architecture.md index c2765fb84..e3fde9564 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,138 +1,97 @@ ---- -title: Architecture ---- - -Architecture -------------- - -### Components - -DirectPV is made up of 5 components: - -| Component | Description | -|-------------------|---------------------------------------------------------------------------------------| -| CSI Driver | Performs mounting, unmounting of provisioned volumes | -| CSI Controller | Responsible for scheduling and detaching volumes on the nodes | -| Drive Controller | Formats and manages drive lifecycle | -| Volume Controller | Manages volume lifecycle | -| Drive Discovery | Discovers and monitors the drives and their states on the nodes | - -The 4 components run as two different pods. - -| Name | Components | Description | -|-------------------------------|-----------------------------------------------------------------------|------------------------------------| -| DirectPV Node Driver | CSI Driver, Driver Controller, Volume Controller, Drive Discovery | runs on every node as a DaemonSet | -| DirectPV Central Controller | CSI Controller | runs as a deployment | - - -### Scalability - -Since the node driver runs on every node, the load on it is constrained to operations specific to that node. - -The central controller needs to be scaled up as the number of drives managed by DirectPV is increased. By default, 3 replicas of central controller are run. As a rule of thumb, having as many central controller instances as etcd nodes is a good working solution for achieving high scale. - -### Availability - -If node driver is down, then volume mounting, unmounting, formatting and cleanup will not proceed for volumes and drives on that node. In order to restore operations, bring node driver to running status. - -In central controller is down, then volume scheduling and deletion will not proceed for all volumes and drives in the DirectPV cluster. In order to restore operations, bring the central controller to running status. - -Security is covered [here](./security.md) - -### Node Driver - -This runs on every node as a Daemonset in `directpv` namespace. Each pod consists of four containers - -#### Node driver registrar - -This is a kubernetes csi side-car container which registers the `directpv` CSI driver with kubelet. This registration is necessary for kubelet to issue CSI RPC calls like `NodeGetInfo`, `NodeStageVolume`, `NodePublishVolume` to the corresponding nodes. - -For more details, please refer [node-driver-registrar](https://github.com/kubernetes-csi/node-driver-registrar). - -#### Livenessprobe - -This is a kubernetes csi side-car container which exposes an HTTP `/healthz` endpoint as a liveness hook. This endpoint will be used by kubernetes for csi-driver liveness checks. - -For more details. please refer [livenessprobe](https://github.com/kubernetes-csi/livenessprobe) - -#### Dynamic drive discovery - -This container uses `directpv` binary with `--dynamic-drive-handler` flag enabled. This container is responsible for discovering and managing the drives in the node. - -The devices will be discovered from `/run/data/udev/` directory and dynamically listens for udev events for any add, change and remove uevents. Apart from dynamically listening, there is a periodic 30sec sync which checks and syncs the drive states. - -For any change, the directcsidrive object will be synced to match the local state. A new directcsidrive object will be created when a new device is detected during sync or when an "Add" uevent occurs. If an inuse/ready drive gets corrupted or lost, it will be tagged with a error condition on the drive. If an Available/Unavailable drive is lost, it will be deleted. - -#### DirectPV - -This container acts as a node plugin and implements the following node service RPCs. - -- [NodeGetInfo](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetinfo) -- [NodeGetCapabilities](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetinfoNodeGetCapabilities) -- [NodeGetVolumeStats](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodegetvolumestats) -- [NodeStageVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodestagevolume) -- [NodePublishVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodepublishvolume) -- [NodeUnstageVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodeunstagevolume) -- [NodeUnpublishVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#nodeunpublishvolume) - -This container is responsible for bind-mounting and umounting volumes on the responding nodes. Monitoring volumes is a WIP and will be added soon. Please refer [csi spec](https://github.com/container-storage-interface/spec) for more details on the CSI volume lifecycle. - -Apart from this, there are also drive and volume controllers in place. - -#### Drive Controller - -Drive controller manages the `directpvdrives` object lifecycle. This actively listens for drive object (post-hook) events like Add, Update and Delete. The drive controller is responsible for the following - -_Formatting a drive_ :- - -If `.Spec.RequestedFormat` is set on the drive object, it indicates the `kubectl directpv drives format` was called on it and this drive will be formatted. - -_Releasing a drive_ :- - -`kubectl directpv drives release` is a special command to release a "Ready" drive in directpv cluster by umounting the drives and making it "Available". If `.Status.DriveStatus` is set to "Released", it indicates that `kubectl directpv drives release` was called on the drive and it will be released. - -_Checking the primary mount of the drive_ :- - -Drive controller also checks for the primary drive mounts. If an "InUse" or "Ready" drive is not mounted or if it has unexpected mount options set, the drive will be remounted with correct mountpoint and mount options. - -_Tagging the lost drives_ :- - -If a drive is not found on the host, it will be tagged as "lost" with an error message attached to the drive object and its respective volume objects. - -Overall, drive controller validates and tries to sync the host state of the drive to match the expected state of the drive. For example, it mounts the "Ready" and "InUse" drives if their primary mount is not present in host. - -For more details on the drive states, please refer [Drive States](./drive-states.md). - -#### Volume Controller - -Volume controller manages the `directpvvolumes` object lifecycle. This actively listens for volume object (post-hook) events like Add, Update and Delete. The volume controller is responsible for the following - -_Releasing/Purging deleted volumes and free-ing up its space on the drive_ :- - -When a volume is deleted (PVC deletion) or purged (using `kubectl directpv drives purge` command), the corresponding volume object will be in terminating state (with deletion timestamp set on it). The volume controller will look for such deleted volume objects and releases them by freeing up the disk space and unsetting the finalizers. - - -### Central Controller - -This runs as a deployment in `directpv` namespace with default replica count 3. - -(Note: The central controller does not do any device level interactions in the host) - -Each pod consist of two continers - -#### CSI Provisioner - -This is a kubernetes csi side-car container which is responsible for sending volume provisioning (CreateVolume) and volume deletion (DeleteVolume) requests to csi drivers. - -For more details, please refer [external-provisioner](https://github.com/kubernetes-csi/external-provisioner). - -#### DirectPV - -This container acts as a central controller and implements the following RPCs - -- [CreateVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#createvolume) -- [DeleteVolume](https://github.com/container-storage-interface/spec/blob/master/spec.md#deletevolume) - -This container is responsible for selecting a suitable drive for a volume scheduling request. The selection algorithm looks for range and topology specifications provided in the CreateVolume request and selects a drive based on its free capacity. - -(Note: kube-scheduler is responsible for selecting a node for a pod, central controller will just select a suitable drive in the requested node based on the specifications provided in the create volume request) +# DirectPV CSI driver architecture + +DirectPV is implemented as per the [CSI specification](https://github.com/container-storage-interface/spec/blob/master/spec.md). It comes with the below components run as Pods in Kubernetes. +* `Controller` +* `Node server` + +When DirectPV contains legacy volumes from `DirectCSI`, the below additional components also run as Pods. +* `Legacy controller ` +* `Legacy node server` + +## Controller +The Controller runs as `Deployment` Pods named `controller`, which are three replicas located in any Kubernetes nodes. In the three replicas, one instance is elected to serve requests. Each pod contains below running containers: +* `CSI provisioner` - Bridges volume creation and deletion requests from `Persistent Volume Claim` to CSI controller. +* `Controller` - Controller server which honors CSI requests to create, delete and expand volumes. +* `CSI resizer` - Bridges volume expansion requests from `Persistent Volume Claim` to CSI controller. + +### Controller server +Controller server runs as container `controller` in a `controller` `Deployment` Pod. It handles below requests: +* `Create volume` - Controller server creates new `DirectPVVolume` CRD after reversing requested storage space on suitable `DirectPVDrive` CRD. For more information, refer to the [Volume scheduling guide](./volume-scheduling.md) +* `Delete volume` - Controller server deletes `DirectPVVolume` CRD for unbound volumes after releasing previously reserved space in `DirectPVDrive` CRD. +* `Expand volume` - Controller server expands `DirectPVVolume` CRD after reversing requested storage space in `DirectPVDrive` CRD. + +Below is a workflow diagram +``` +┌────────────┐ ┌────────────┐ +│ │ Create Event ┌─────────────┐ CreateVolume API │ │ ┌────────────────────┐ +│ │------------->│ CSI │----------------->│ │-->│ DirectPVDrive CRD │ +│ Persistent │ Delete Event │ Provisioner │ DeleteVolume API │ │ └────────────────────┘ +│ Volume │------------->│ │----------------->│ Controller │ +│ Claim │ └─────────────┘ │ Server │ +│ (PVC) │ │ │ ┌────────────────────┐ +│ │ Update Event ┌─────────────┐ ExpandVolume API │ │-->│ DirectPVVolume CRD │ +│ │------------->│ CSI Resizer │----------------->│ │ └────────────────────┘ +│ │ └─────────────┘ └────────────┘ +└────────────┘ +``` + +## Legacy controller +Legacy controller runs as `Deployment` Pods named `legacy-controller`, which are three replicas located in any Kubernetes nodes. In the three replicas, one instance is elected to serve requests. Each pod contains below running containers: +* `CSI provisioner` - Bridges legacy volume creation and deletion requests from `Persistent Volume Claim` to CSI controller. +* `Controller` - Honors CSI requests to delete and expand volumes. Create volume request is prohibited i.e. this controller works only for legacy volumes previously created in `DirectCSI`. +* `CSI resizer` - Bridges legacy volume expansion requests from `Persistent Volume Claim` to CSI controller. + +### Legacy controller server +Legacy controller server runs as container `controller` in a `legacy-controller` `Deployment` Pod. It handles below requests: +* `Create volume` - Controller server errors out for this request. +* `Delete volume` - Controller server deletes `DirectPVVolume` CRD for unbound volumes after releasing previously reserved space in `DirectPVDrive` CRD. +* `Expand volume` - Controller server expands `DirectPVVolume` CRD after reversing requested storage space in `DirectPVDrive` CRD. + +Below is a workflow diagram +``` +┌─────────────────────┐ ┌────────────┐ +│ │ Delete Event ┌─────────────────┐ DeleteVolume API │ │ ┌────────────────────┐ +│ Persistent Volume │------------->│ CSI Provisioner │----------------->│ │-->│ DirectPVDrive CRD │ +│ Claim (PVC) with │ └─────────────────┘ │ Controller │ └────────────────────┘ +│ direct-csi-min-io │ │ Server │ ┌────────────────────┐ +│ Storage Class │ Update Event ┌─────────────────┐ ExpandVolume API │ │-->│ DirectPVVolume CRD │ +│ │------------->│ CSI Resizer │----------------->│ │ └────────────────────┘ +│ │ └─────────────────┘ └────────────┘ +└─────────────────────┘ +``` + +## Node server +Node server runs as `DaemonSet` Pods named `node-server` in all or selected Kubernetes nodes. Each node server Pod runs on a node independently. Each pod contains below running containers: +* `Node driver registrar` - Registers node server to kubelet to get CSI RPC calls. +* `Node server` - Honors stage, unstage, publish, unpublish and expand volume RPC requests. +* `Node controller` - Honors CRD events from `DirectPVDrive`, `DirectPVVolume`, `DirectPVNode` and `DirectPVInitRequest`. +* `Liveness probe` - Exposes `/healthz` endpoint to check node server liveness by Kubernetes. + +Below is a workflow diagram +``` +┌─────────┐ ┌────────┐ ┌──────────────────────────────────┐ ┌────────────────────┐ +│ │ StageVolume RPC │ │ StageVolume │ * Create data directory │ │ │ +│ │------------------->│ │---------------->│ * Set xfs quota │<-->│ │ +│ │ │ │ │ * Bind mount staging target path │ │ │ +│ │ │ │ └──────────────────────────────────┘ │ │ +│ │ PublishVolume RPC │ │ PublishVolume ┌──────────────────────────────────┐ │ │ +│ │------------------->│ │---------------->│ * Bind mount target path │<-->│ │ +│ Kubelet │ │ Node │ └──────────────────────────────────┘ │ DirectPVDrive CRD │ +│ │UnpublishVolume RPC │ Server │ UnpublishVolume ┌──────────────────────────────────┐ │ DirectPVVolume CRD │ +│ │------------------->│ │---------------->│ * Unmount target path │<-->│ │ +│ │ │ │ └──────────────────────────────────┘ │ │ +│ │ UnstageVolume RPC │ │ UnstageVolume ┌──────────────────────────────────┐ │ │ +│ │------------------->│ │---------------->│ * Unmount staging target path │<-->│ │ +│ │ │ │ └──────────────────────────────────┘ │ │ +│ │ ExpandVolume RPC │ │ ExpandVolume ┌──────────────────────────────────┐ │ │ +│ │------------------->│ │---------------->│ * Set xfs quota │<-->│ │ +└─────────┘ └────────┘ └──────────────────────────────────┘ └────────────────────┘ +``` + +## Legacy node server +Legacy node server runs as `DaemonSet` Pods named `legacy-node-server` in all or selected Kubernetes nodes. Each legacy node server Pod runs on a node independently. Each pod contains the below running containers: +* `Node driver registrar` - Registers legacy node server to kubelet to get CSI RPC calls. +* `Node server` - Honors stage, unstage, publish, unpublish and expand volume RPC requests. +* `Liveness probe` - Exposes `/healthz` endpoint to check legacy node server liveness by Kubernetes. + +Workflow diagram is same as in [node server](#node-server). diff --git a/docs/cli.md b/docs/command-reference.md similarity index 74% rename from docs/cli.md rename to docs/command-reference.md index 485ac8f38..533482794 100644 --- a/docs/cli.md +++ b/docs/command-reference.md @@ -1,17 +1,50 @@ -### Install Kubectl plugin - -The `directpv` kubectl plugin can be used to manage the lifecycle of volumes and drives in the kubernetes cluster - -```sh -$ kubectl krew install directpv +# Command reference + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer to the [plugin installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer to the [driver installation guide](./installation.md#directpv-csi-driver-installation). + +## Note +The DirectPV plugin command is referred to as `kubectl directpv` in this document. If you are using the direct binary instead, replace with `kubectl-directpv` throughout. + +## Command changes from DirectCSI +| DirectCSI command | DirectPV command | +|:---------------------------------|:----------------------------------------------------------| +| `kubectl directcsi drives list` | `kubectl directpv list drives` | +| `kubectl directcsi volumes list` | `kubectl directpv list volumes` | +| `kubectl directcsi format` | `kubectl directpv discover`, then `kubectl directpv init` | + +## Global flags +All of the plugin commands can use the following global flags. + +| Flag | Argument | Description | +|:---------------|:---------|:----------------------------------------------------| +| `--kubeconfig` | _string_ | Path to the kubeconfig file to use for CLI requests | +| `--quiet` | - | Suppress printing error messages | +| `-h`, `--help` | - | help for directpv | +| `--version` | - | version for directpv | + +## Commands +List of subcommands are below + +| Subcommand | Description | +|:------------|:----------------------------------------------------------------------------------| +| `install` | Install DirectPV in Kubernetes | +| `discover` | Discover new drives | +| `init` | Initialize the drives | +| `info` | Show information about DirectPV installation | +| `list` | List drives and volumes | +| `label` | Set labels to drives and volumes | +| `cordon` | Mark drives as unschedulable | +| `uncordon` | Mark drives as schedulable | +| `migrate` | Migrate drives and volumes from legacy DirectCSI | +| `move` | Move volumes excluding data from source drive to destination drive on a same node | +| `clean` | Cleanup stale volumes | +| `remove` | Remove unused drives from DirectPV | +| `uninstall` | Uninstall DirectPV in Kubernetes | + +## `install` command ``` - -### Install DirectPV - -Install DirectPV in your kubernetes cluster - -```sh -$ kubectl directpv install --help Install DirectPV in Kubernetes USAGE: @@ -22,13 +55,14 @@ FLAGS: --tolerations strings Set toleration labels on the storage nodes (KEY[=VALUE]:EFFECT,..) --registry string Name of container registry (default "quay.io") --org string Organization name in the registry (default "minio") - --image string Name of the DirectPV image (default "directpv:0.0.0-dev") + --image string Name of the DirectPV image (default "directpv:v4.0.6") --image-pull-secrets strings Image pull secrets for DirectPV images (SECRET1,..) --apparmor-profile string Set path to Apparmor profile --seccomp-profile string Set path to Seccomp profile -o, --output string Generate installation manifest. One of: yaml|json --kube-version string Select the kubernetes version for manifest generation (default "1.27.0") --legacy Enable legacy mode (Used with '-o') + --openshift Use OpenShift specific installation -h, --help help for install GLOBAL FLAGS: @@ -56,15 +90,10 @@ EXAMPLES: 7. Install DirectPV with seccomp profile $ kubectl directpv install --seccomp-profile profiles/seccomp.json - ``` -### Discover drives - -Discover the block devices present in the cluster - -```sh -$ kubectl directpv discover --help +## `discover` command +``` Discover new drives USAGE: @@ -97,17 +126,10 @@ EXAMPLES: 5. Discover specific drives from specific nodes $ kubectl directpv discover --nodes=node{1...4} --drives=sd{a...f} - ``` -### Initialize the available drives present in the cluster - -Initializing the drives will format the selected drives with XFS filesystem and mount them to `/var/lib/directpv/mnt/`. DirectPV can then use the initialized drives for provisioning Persistent Volumes in respones to PVC with the `directpv-min-io` storage class. - -**Warning**: This command will completely and irreversibly erase the data (mkfs) in the selected disks by formatting them - -```sh -$ kubectl directpv init --help +## `init` command +``` Initialize the drives USAGE: @@ -115,6 +137,7 @@ USAGE: FLAGS: --timeout duration specify timeout for the initialization process (default 2m0s) + --dangerous Perform initialization of drives which will permanently erase existing data -h, --help help for init GLOBAL FLAGS: @@ -124,13 +147,10 @@ GLOBAL FLAGS: EXAMPLES: 1. Initialize the drives $ kubectl directpv init drives.yaml - ``` -### Show overall information about the DirectPV installation in the cluster - -```sh -$ kubectl directpv info --help +## `info` command +``` Show information about DirectPV installation USAGE: @@ -142,13 +162,35 @@ FLAGS: GLOBAL FLAGS: --kubeconfig string Path to the kubeconfig file to use for CLI requests --quiet Suppress printing error messages +``` +## `list` command ``` +List drives and volumes -### List the drives initialized and managed by DirectPV +USAGE: + directpv list [command] -```sh -$ kubectl directpv list drives --help +FLAGS: + -n, --nodes strings Filter output by nodes; supports ellipses pattern e.g. node{1...10} + -d, --drives strings Filter output by drive names; supports ellipses pattern e.g. sd{a...z} + -o, --output string Output format. One of: json|yaml|wide + --no-headers When using the default or custom-column output format, don't print headers (default print headers) + -h, --help help for list + +GLOBAL FLAGS: + --kubeconfig string Path to the kubeconfig file to use for CLI requests + --quiet Suppress printing error messages + +AVAILABLE COMMANDS: + drives List drives + volumes List volumes + +Use "directpv list [command] --help" for more information about this command. +``` + +### `drives` command +``` List drives USAGE: @@ -196,13 +238,10 @@ EXAMPLES: 8. List drives filtered by labels $ kubectl directpv list drives --labels tier=hot - ``` -### List the volumes provisioned and managed by DirectPV - -```sh -$ kubectl directpv list volumes --help +### `volumes` command +``` List volumes USAGE: @@ -260,13 +299,35 @@ EXAMPLES: 10. List volumes filtered by labels $ kubectl directpv list volumes --labels tier=hot +``` +## `label` command ``` +Set labels to drives and volumes -### Set lables on the drives managed by DirectPV +USAGE: + directpv label [command] -```sh -$ kubectl directpv label drives --help +FLAGS: + -n, --nodes strings If present, filter objects from given nodes; supports ellipses pattern e.g. node{1...10} + -d, --drives strings If present, filter objects by given drive names; supports ellipses pattern e.g. sd{a...z} + --all If present, select all objects + --dry-run Run in dry run mode + -h, --help help for label + +GLOBAL FLAGS: + --kubeconfig string Path to the kubeconfig file to use for CLI requests + --quiet Suppress printing error messages + +AVAILABLE COMMANDS: + drives Set labels to drives + volumes Set labels to volumes + +Use "directpv label [command] --help" for more information about this command. +``` + +### `drives` command +``` Set labels to drives USAGE: @@ -298,13 +359,10 @@ EXAMPLES: 3. Remove 'tier: hot' label from all drives in all nodes $ kubectl directpv label drives tier- --all - ``` -### Set labels on the volumes managed by DirectPV - -```sh -$ kubectl directpv label volumes --help +### `volumes` command +``` Set labels to volumes USAGE: @@ -339,13 +397,10 @@ EXAMPLES: 3. Remove 'tier: hot' label from all volumes in all nodes $ kubectl directpv label volumes tier- --all - ``` -### Cordon the drives to make them unschedulable - -```sh -$ kubectl directpv cordon --help +## `cordon` command +``` Mark drives as unschedulable USAGE: @@ -378,13 +433,10 @@ EXAMPLES: 5. Cordon drives which are in 'error' status $ kubectl directpv cordon --status=error - ``` -### Uncordon the cordoned drives - -```sh -$ kubectl directpv uncordon --help +## `uncordon` command +``` Mark drives as schedulable USAGE: @@ -415,18 +467,32 @@ EXAMPLES: 4. Uncordon specific drives from specific nodes $ kubectl directpv uncordon --nodes=node{1...4} --drives=sd{a...f} -5. Uncordon drives which are in 'warm' access-tier - $ kubectl directpv uncordon --access-tier=warm - -6. Uncordon drives which are in 'error' status +5. Uncordon drives which are in 'error' status $ kubectl directpv uncordon --status=error +``` +## `migrate` command ``` +Migrate drives and volumes from legacy DirectCSI + +USAGE: + directpv migrate [flags] -### Move volumes from one drive to another drive within a node (excluding data) +FLAGS: + --dry-run Run in dry run mode + -h, --help help for migrate + +GLOBAL FLAGS: + --kubeconfig string Path to the kubeconfig file to use for CLI requests + --quiet Suppress printing error messages -```sh -$ kubectl directpv move --help +EXAMPLES: +1. Migrate drives and volumes from legacy DirectCSI + $ kubectl directpv migrate +``` + +## `move` command +``` Move volumes excluding data from source drive to destination drive on a same node USAGE: @@ -445,13 +511,10 @@ GLOBAL FLAGS: EXAMPLES: 1. Move volumes from drive af3b8b4c-73b4-4a74-84b7-1ec30492a6f0 to drive 834e8f4c-14f4-49b9-9b77-e8ac854108d5 $ kubectl directpv drives move af3b8b4c-73b4-4a74-84b7-1ec30492a6f0 834e8f4c-14f4-49b9-9b77-e8ac854108d5 - ``` -### Cleanup stale (released|deleted) volumes - -```sh -$ kubectl directpv clean --help +## `clean` command +``` Cleanup stale volumes USAGE: @@ -492,13 +555,10 @@ EXAMPLES: 7. Clean volumes by pod namespace $ kubectl directpv clean --pod-namespaces=tenant-{1...3} - ``` -### Remove unused drives from DirectPV - -```sh -$ kubectl directpv remove --help +## `remove` command +``` Remove unused drives from DirectPV USAGE: @@ -531,13 +591,10 @@ EXAMPLES: 5. Remove drives are in 'error' status $ kubectl directpv remove --status=error - ``` -### Uninstall DirectPV from the kubernetes cluster - -```sh -$ kubectl directpv uninstall --help +## `uninstall` command +``` Uninstall DirectPV in Kubernetes USAGE: @@ -549,5 +606,4 @@ FLAGS: GLOBAL FLAGS: --kubeconfig string Path to the kubeconfig file to use for CLI requests --quiet Suppress printing error messages - ``` diff --git a/docs/developer.md b/docs/developer.md new file mode 100644 index 000000000..8fe9710ae --- /dev/null +++ b/docs/developer.md @@ -0,0 +1,17 @@ +# Development and Testing +We welcome any contribution to DirectPV. All your pull requests are welcomed. + +## Building DirectPV from source +1. Get source code from [DirectPV Github repository](https://github.com/minio/directpv) +2. Run `./build.sh`. You will get `directpv` CSI driver binary and `kubectl-directpv` plugin binary. + +## Building DirectPV container image +1. Run `podman build -t quay.io/minio/directpv:$(./kubectl-directpv --version | awk '{ print $NF }') .` + +## Pushing container image to kubernetes nodes +1. Run `podman save --output directpv.tar quay.io/minio/directpv:$(./kubectl-directpv --version | awk '{ print $NF }')` to get image tarball. +2. Copy `directpv.tar` to all your kubernetes nodes. +3. Import `directpv.tar` by running `/usr/local/bin/ctr images import --digests --base-name quay.io/minio/directpv directpv.tar` on each node. This step works for `K3S` cluster. + +## Installing DirectPV build +1. Run `./kubectl-directpv install` diff --git a/docs/development-and-testing.md b/docs/development-and-testing.md deleted file mode 100644 index f92c33cb0..000000000 --- a/docs/development-and-testing.md +++ /dev/null @@ -1,154 +0,0 @@ -## Development and Testing -1. Have `quay.io` account, running docker and kubernetes. You could use any registry account and replace `quay.io` with such registry name in below steps. -2. Docker login to your `quay.io` account. -```bash -$ docker login --username https://quay.io -``` -3. Create `csi-provisioner`, `livenessprobe` and `csi-node-driver-registrar` repositories by pull/tag/push respective images to your `quay.io` account. -```bash -$ docker pull quay.io/minio/csi-provisioner:v3.5.0 && \ -docker tag quay.io/minio/csi-provisioner:v3.5.0 quay.io//csi-provisioner:v3.5.0 && \ -docker push quay.io//csi-provisioner:v3.5.0 - -$ docker pull quay.io/minio/livenessprobe:v2.10.0 && \ -docker tag quay.io/minio/livenessprobe:v2.10.0 quay.io//livenessprobe:v2.10.0 && \ -docker push quay.io//livenessprobe:v2.10.0 - -$ docker pull quay.io/minio/csi-node-driver-registrar:v2.8.0 && \ -docker tag quay.io/minio/csi-node-driver-registrar:v2.8.0 quay.io//csi-node-driver-registrar:v2.8.0 && \ -docker push quay.io//csi-node-driver-registrar:v2.8.0 - -$ docker pull quay.io/minio/csi-resizer:v1.8.0 && \ -docker tag quay.io/minio/csi-resizer:v1.8.0 quay.io//csi-resizer:v1.8.0 && \ -docker push quay.io//csi-resizer:v1.8.0 -``` -4. Make sure `csi-provisioner`, `livenessprobe`, `csi-node-driver-registrar` and `csi-resizer` repositories are `public` in your `quay.io` account. -5. Go to your DirectPV project root. -```bash -$ cd $GOPATH/src/github.com/minio/directpv -``` -6. Hack, hack, hack... -7. Run ./build.sh -```bash -$ ./build.sh -``` -8. Run docker build to tag image. -```bash -$ docker build -t quay.io//directpv: . -``` -9. Push newly created image to your `quay.io` account. -```bash -$ docker push quay.io//directpv: -``` -10. Make sure `directpv` repository is `public` in your `quay.io` account. -11. Install DirectPV. -```bash -$ ./kubectl-directpv --kubeconfig install \ ---image directpv: --org --registry quay.io -``` -12. Check running DirectPV -```bash -$ ./kubectl-directpv --kubeconfig info - -$ ./kubectl-directpv --kubeconfig list drives -``` - -## Testing with minikube - -1. Setup LVs - -The following script will create 4 LVs backed up by 4 loopback devices - -```bash -sudo truncate --size=1G /tmp/disk-{1..4}.img -for disk in /tmp/disk-{1..4}.img; do sudo losetup --find $disk; done -devices=( $(for disk in /tmp/disk-{1..4}.img; do sudo losetup --noheadings --output NAME --associated $disk; done) ) -sudo pvcreate "${devices[@]}" -vgname="vg0" -sudo vgcreate "$vgname" "${devices[@]}" -for lvname in lv-{0..3}; do sudo lvcreate --name="$lvname" --size=800MiB "$vgname"; done -``` - -2. Start minikube - -```bash -minikube start --driver=none -``` - -3. Install DirectPV - -Install the freshly built version - -```bash -./kubectl-directpv install --image directpv: --org --registry quay.io -``` - -4. Discover and initialize the drives - -```bash -./kubectl-directpv discover --output-file drives.yaml -./kubectl-directpv init drives.yaml -``` - -5. Check if the drives are showing up - -```bash -./kubectl-directpv list drives -``` - -6. Apply the minio.yaml file - -Download and apply a sample MinIO deployment file available [here](https://github.com/minio/directpv/blob/master/functests/minio.yaml) - -```bash -kubectl apply -f minio.yaml -``` - -7. Check if the pods are up and running - -```bash -kubectl get pods -``` - -8. Check the volumes - -```bash -./kubectl-directpv list volumes -``` - -9. Check the drives contain volumes - -```bash -./kubectl-directpv list drives -``` - -10. Uninstall the MinIO deployment - -```bash -kubectl delete -f minio.yaml -``` - -11. Delete the PVCs - -```bash -kubectl delete pvc --all -``` - -After deleting the PVCs, check if the drives are freed up. - -12. Release freed drives - -```bash -./kubectl-directpv remove --all -``` - -13. Cleanup LV setup - -```sh -sudo lvremove vg0 -y -sudo vgremove vg0 -y -sudo pvremove /dev/loop /dev/loop /dev/loop /dev/loop # n can be replaced with the loopbacks created -sudo losetup --detach-all -``` - -Please refer [here](./troubleshooting.md) for any trouble shooting guidelines. diff --git a/docs/drain-node.md b/docs/drain-node.md deleted file mode 100644 index 21ee8a1ec..000000000 --- a/docs/drain-node.md +++ /dev/null @@ -1,27 +0,0 @@ -Drain a node -------------- - -Draining will forcefully remove the DirectPV resources from a node. This [bash script](./tools/drain.sh) can be used for draining and has to be executed cautiously as it is an irreversible operation and may incur data loss. - -You can consider draining a node in the following circumstances - -#### When a node is detached from kubernetes - -If a node which was used by DirectPV is detached from kubernetes, the DirectPV resources from that node will remain intact until the resources are drained. - -#### When DirectPV is unselected to run on a specific node - -If a node which was used by DirectPV is decided to be a "non-storage" node. - -For example, If DirectPV is decided to not run on a specific node by changing the node-selectors like the following example - -```sh -$ kubectl directpv uninstall -$ kubectl directpv install --node-selector node-label-key=node-label-value -``` - -the resources from the detached node can then be cleaned up by - -```sh -./drain.sh -``` diff --git a/docs/drive-management.md b/docs/drive-management.md new file mode 100644 index 000000000..55bff1101 --- /dev/null +++ b/docs/drive-management.md @@ -0,0 +1,112 @@ +# Drive management + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer to the [plugin installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer to the [driver installation guide](./installation.md#directpv-csi-driver-installation). + +## Add drives +Drives are added to DirectPV to provision volumes. This involves a two step process as shown below. + +1. Run `discover` command. +The `discover` command probes eligible drives from DirectPV nodes and stores drive information in a YAML file. You should carefully examine the YAML file and set the `select` field to `yes` or `no` value to indicate drive selection. `select` field is set to `yes` value by default. Below is an example of the `discover` command: + +```sh +# Probe and save drive information to drives.yaml file. +$ kubectl directpv discover + + Discovered node 'master' ✔ + Discovered node 'node1' ✔ + +┌─────────────────────┬────────┬───────┬─────────┬────────────┬──────┬───────────┬─────────────┐ +│ ID │ NODE │ DRIVE │ SIZE │ FILESYSTEM │ MAKE │ AVAILABLE │ DESCRIPTION │ +├─────────────────────┼────────┼───────┼─────────┼────────────┼──────┼───────────┼─────────────┤ +│ 252:16$ud8mwCjPT... │ master │ vdb │ 512 MiB │ - │ - │ YES │ - │ +│ 252:16$gGz4UIuBj... │ node1 │ vdb │ 512 MiB │ - │ - │ YES │ - │ +└─────────────────────┴────────┴───────┴─────────┴────────────┴──────┴───────────┴─────────────┘ + +Generated 'drives.yaml' successfully. + +# Show generated drives.yaml file. +$ cat drives.yaml +version: v1 +nodes: + - name: master + drives: + - id: 252:16$ud8mwCjPTH8147TysmiQ2GGLpffqUht6bz7NtHqReJo= + name: vdb + size: 536870912 + make: "" + select: "yes" + - name: node1 + drives: + - id: 252:16$gGz4UIuBjQlO1KibOv7bZ+kEDk3UCeBneN/UJdqdQl4= + name: vdb + size: 536870912 + make: "" + select: "yes" + +``` + +2. Run `init` command. +The `init` command creates a request to add the selected drives in the YAML file generated using the `discover` command. As this process wipes out all data on the selected drives, wrong drive selection will lead to permanent data loss. Below is an example of `init` command: + +```sh +$ kubectl directpv init drives.yaml + + ███████████████████████████████████████████████████████████████████████████ 100% + + Processed initialization request 'c24e22f5-d582-49ba-a883-2ce56909904e' for node 'master' ✔ + Processed initialization request '7e38a453-88ed-412c-b146-03eef37b23bf' for node 'node1' ✔ + +┌──────────────────────────────────────┬────────┬───────┬─────────┐ +│ REQUEST_ID │ NODE │ DRIVE │ MESSAGE │ +├──────────────────────────────────────┼────────┼───────┼─────────┤ +│ c24e22f5-d582-49ba-a883-2ce56909904e │ master │ vdb │ Success │ +│ 7e38a453-88ed-412c-b146-03eef37b23bf │ node1 │ vdb │ Success │ +└──────────────────────────────────────┴────────┴───────┴─────────┘ +``` + +Refer to the [discover command](./command-reference.md#discover-command) and the [init command](./command-reference.md#init-command) for more information. + +## List drives +To get information of drives from DirectPV, run the `list drives` command. Below is an example: + +```sh +$ kubectl directpv list drives +┌────────┬──────┬──────┬─────────┬─────────┬─────────┬────────┐ +│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ +├────────┼──────┼──────┼─────────┼─────────┼─────────┼────────┤ +│ master │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +│ node1 │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +└────────┴──────┴──────┴─────────┴─────────┴─────────┴────────┘ +``` + +Refer to the [list drives command](./command-reference.md#drives-command) for more information. + +## Label drives +Drives are labeled to set custom tagging which can be used in volume provisioning. Below is an example: +```sh +# Set label 'tier' key to 'hot' value. +$ kubectl directpv label drives tiet=hot + +# Remove label 'tier'. +$ kubectl directpv label drives tier- +``` + +Refer to the [label drives command](./command-reference.md#drives-command-1) for more information. + +## Replace drive +Replace a faulty drive with a new drive on a same node. In this process, all volumes in the faulty drive are moved to the new drive then faulty drive is removed from DirectPV. Currently, DirectPV does not support moving data on the volume to the new drive. Use [replace.sh](./tools/replace.sh) script to perform drive replacement. Below is an example: +```sh +# Replace 'sdd' drive by 'sdf' drive on 'node1' node +$ replace.sh sdd sdf node1 +``` + +## Remove drives +Drives that do not contain any volumes can be removed. Below is an example: +```sh +# Remove drive 'vdb' from 'node1' node +$ kubectl directpv remove --drives=vdb --nodes=node1 +``` + +Refer [remove command](./command-reference.md#remove-command) for more information. diff --git a/docs/drive-replacement.md b/docs/drive-replacement.md deleted file mode 100644 index 593d92090..000000000 --- a/docs/drive-replacement.md +++ /dev/null @@ -1,114 +0,0 @@ -Drive replacement (applies for versions > v3.2.2) -------------- - -These are the steps to move the volumes from one drive to another drive from the **same node**. These steps will be useful during drive replacement. - -(NOTE: This will just move the volume references from source to the destination drive without moving the actual data) - -## STEP-BY-STEP Guide for drive replacement - -### STEP 1: Add new drive - -Once the new drive has been attached to the node as a replacement of an older drive, Discover and initialize the newly added drive by executing the following steps - -```sh -$ kubectl directpv discover --drives /dev/ --nodes -``` - -This command will generate `drives.yaml` which will be used later for initialization. Please refer [Discover CLI](./cli.md#discover-drives) for more helpers on discovery. - -Now, initialize the drive using the `drives.yaml` file that was created by the previous command - -```sh -$ kubectl directpv init drives.yaml -``` - -On success, you should see the new drive listed in `kubectl directpv list drives --drives /dev/` - -### STEP 2: Get source and destination drive IDs - -```sh -$ kubectl directpv list drives --drives , --nodes -o wide -``` - -### STEP 3: Cordon source and destination drives - -By cordoning, we mark the drives as unschedulable so that no new volumes will be scheduled on these drives - -```sh -$ kubectl directpv cordon -``` - -### STEP 4: Get the pods used by the source drive - -```sh -$ kubectl directpv list volumes --drives /dev/ --nodes -``` - -### STEP 5: Cordon the corresponding kubernetes node and delete the pods used by the source drive - -Cordon the corresponding k8s node and delete the corresponding workload pods used by the source drive, this will take the workloads offline and also doesn't schedule any new pods on this node. - -```sh -$ kubectl cordon -``` - -(NOTE: Avoid force deleting here as force deletion might not unpublish the volumes) - -```sh -$ kubectl delete pod/ -n -``` - -This pod will fall in "pending" state after deleting as the node is cordoned - -### STEP 6: Wait for associated DirectPV volumes to be unbounded - -There shouldn't be any volumes in "Bounded" state - -```sh -$ kubectl directpv list volumes --pod-names= --pod-namespaces= -``` - -### STEP 7: Run move command - -```sh -$ kubectl directpv move -``` - -After moving, list the drives to see if the volumes got moved to destination - -``` -$ kubectl directpv list drives --drives , --nodes -o wide -``` - -### STEP 8: Uncordon the destination drive - -```sh -$ kubectl directpv uncordon -``` - -### STEP 9: Uncordon the kubernetes node - -```sh -$ kubectl uncordon -``` - -Now, the corresponding workload pods should be coming up as the node is uncordoned. This pod will be using the new drive. - -### STEP 10 (Optional): You can remove the old drive which was replaced - -This will release the drive as it is not used anymore - -```sh -$ kubectl directpv remove -``` - -## Bash script for drive replacement - -Alternatively, after the "STEP 2" in the previous section, you can use this [bash script](./tools/replace.sh) to move the drives. - -#### Example Usage - -```sh -./replace.sh -``` diff --git a/docs/faq.md b/docs/faq.md index e3b6ab562..752312464 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,72 +1,67 @@ -FAQ ----- - -#### What type of disks are recommended for DirectPV? - -DirectPV is specifically meant for [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage) such as hard drives, solid-state drives and JBODs etc. - -Avoid using DirectPV with SAN and NAS based storage options, as they inherently involve extra network hops in the data path. This leads to poor performance and increased complexity. - -#### How DirectPV is different from LocalPV and HostPath? - -Hostpath volumes are ephemeral and are tied to the lifecyle of pods. Hostpath volumes are lost when a pod is restarted or deleted, resulting in the loss of any stored data. - -DirectPV volumes are persistent through node and pod reboots. The lifecycle of a DirectPV volume is managed by the associated [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). - -LocalPVs are statically provisioned from a local storage resource on the nodes where persistent volumes are required, and must be created prior to the workloads using them. User management is required down to creating the actual object resource for the local PV. - -DirectPV also creates statically provisioned volume resources, but does not require user intervention for the creation of the object resource. Instead, DirectPV dynamically provisions the persistent volume in response to a PVC requesting a DirectPV storage volume. This significantly reduces complexity of management. - -#### How are the disks selected for a Pod? - -DirectPV operates only on those disks which it [explicitly manages](./cli.md#initialize-the-available-drives-present-in-the-cluster) - - 1. DirectPV selects managed disks local to the node where the pod is scheduled. This provides direct access for the pods to the disks. - - 2. DirectPV runs a selection algorithm to choose a disk for a volume creation request - - 3. DirectPV then creates a sub-directory for the volume with quota set on the sub-directory for the requested volume size - - 4. DirectPV publishes the volume to the pod. - - To know more on the selection algorithm, please refer [here](./volume-scheduling.md). - -#### What does drive initialization do? - -DirectPV command `kubectl directpv init` command will prepare the drives by formatting them with XFS filesystem and mounting them in a desired path (/var/lib/directpv/mnt/). Upon success, these drives will be ready for volumes to be scheduled and you can see the initialized drives in `kubectl directpv list drives`. - -#### What are the conditions that a drive must satisfy to be used for DirectPV? - -A drive must meet the following requirements for it to be used for initialization - -- The size of the drive should not be less then 512MiB. -- The drive must not be hidden. (Check /sys/class/block//hidden, It should not be "1") -- The drive must not be a read-only. -- The drive must not be partitioned. -- The drive must not be held by other devices. (Check /sys/class/block//holders, It should not have any entries) -- The drive must not be mounted. -- The drive must not have swap-on enabled. -- The drive must not be a CDROM. - -DirectPV can support any storage resource presenting itself as a block device. While this includes LVMs, LUKs, LUNs, and similar virtualized drives, we do not recommend using DirectPV with those in production environments. - -#### What does error "no drive found for requested topology" mean? - -This error can be seen in PVC and pod description when - -- Kubernetes scheduled the pod on a node where DirectPV doesn't have any drives initialized. This can be fixed by using necessary selectors in the workload to make k8s schedule the pods on the correct set of nodes. Please refer [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) to know more on assigning the pod to specific nodes. -- The requested topology specifications like size, zone etc cannot be satisified by any of initialized drives in the selected node. This can be fixed by changing the topology parameter in the workloads to suitable value. - -#### Is `direct-csi-min-io` storage class still supported? - -No, the support for `direct-csi-min-io` is removed from DirectPV version v4.0.0. No new volumes will be provisioned using `direct-csi-min-io` storage class. However, the existing volumes which are already provisioned by `direct-csi-min-io` storage class will still be managed by the latest DirectPV versions. - -Alternatively, you can use `directpv-min-io` storage class for provisioning new volumes. And there are no functional or behavioral differences between 'direct-csi-min-io' and 'directpv-min-io'. - -#### Does DirectPV support volume snapshotting? - -No, DirectPV doesn't support CSI snapshotting. DirectPV is specifically meant for use cases like MinIO where the data availability and resiliency is taken care by the application itself. Additionally, with the AWS S3 versioning APIs and internal healing, snapshots isn't required. - -#### Does DirectPV support _ReadWriteMany_? - -No, DirectPV doesn't support _ReadWriteMany_. The workloads using DirectPV run local to the node and are provisioned from local disks in the node. This makes the workloads to direcly access the data path without any additional network hops unlike remote volumes (network PVs). The additional network hops may lead to poor performance and increases the complexity. So, DirectPV doesn't support _ReadWriteMany_ and only supports _ReadWriteOnce_. +# Frequently Asked Questions +* [DirectPV installation fails in my Kubernetes. Why?](#directpv-installation-fails-in-my-kubernetes-why) +* [After upgrading DirectPV to v4.x.x, I do not find `direct-csi-min-io` storage class. Why?](#after-upgrading-directpv-to-v4xx-i-do-not-find-direct-csi-min-io-storage-class-why) +* [In the YAML output of `discover` command, I do not find my storage drive(s). Why?](#in-the-yaml-output-of-discover-command-i-do-not-find-my-storage-drives-why) +* [Do you support SAN, NAS, iSCSI, network drives etc.,?](#do-you-support-san-nas-iscsi-network-drives-etc) +* [Do you support LVM, Linux RAID, Hardware RAID, Software RAID etc.,?](#do-you-support-lvm-linux-raid-hardware-raid-software-raid-etc) +* [Is LUKS device supported?](#is-luks-device-supported) +* [I am already using Local Persistent Volumes (Local PV) for storage. Why do I need DirectPV?](#i-am-already-using-local-persistent-volumes-local-pv-for-storage-why-do-i-need-directpv) +* [I see `no drive found ...` error message in my Persistent Volume Claim. Why?](#i-see-no-drive-found--error-message-in-my-persistent-volume-claim-why) +* [I see Persistent Volume Claim is created, but respective DirectPV volume is not created. Why?](#i-see-persistent-volume-claim-is-created-but-respective-directpv-volume-is-not-created-why) +* [I see volume consuming Pod still in `Pending` state. Why?](#i-see-volume-consuming-pod-still-in-pending-state-why) +* [I see `volume XXXXX is not yet staged, but requested with YYYYY` error. Why?](#i-see-volume-xxxxx-is-not-yet-staged-but-requested-with-yyyyy-error-why) +* [I see ```unable to find device by FSUUID xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; either device is removed or run command `sudo udevadm control --reload-rules && sudo udevadm trigger` on the host to reload``` error. Why?](#i-see-unable-to-find-device-by-fsuuid-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-either-device-is-removed-or-run-command-sudo-udevadm-control---reload-rules--sudo-udevadm-trigger-on-the-host-to-reload-error-why) + +### DirectPV installation fails in my Kubernetes. Why? +You need to have necessary privileges and permissions to perform installation. Go though the [specifications documentation](./specifications.md). For Red Hat OpenShift, refer to the [OpenShift specific documentation](./openshift.md). + +### After upgrading DirectPV to v4.x.x, I do not find `direct-csi-min-io` storage class. Why? +Legacy DirectCSI is deprecated including storage class `direct-csi-min-io` and it is no longer supported. Previously created volumes continue to work normally. For new volume requests, use the `directpv-min-io` storage class. + +### In the YAML output of `discover` command, I do not find my storage drive(s). Why? +DirectPV ignores drives that meet any of the below conditions: +* The size of the drive is less than 512MiB. +* The drive is hidden. +* The drive is read-only. +* The drive is parititioned. +* The drive is held by other devices. +* The drive is mounted or in use by DirectPV already. +* The drive is in-use swap partition. +* The drive is a CDROM. + +Check the last column of the `discover --all` command output to see what condition(s) exclude the drive. Resolve the conditions and try again. + +### Do you support SAN, NAS, iSCSI, network drives etc.,? +DirectPV is meant for high performance local volumes with Direct Attached Storage. We do not recommend any remote drives, as remote drives may lead to poor performance. + +### Do you support LVM, Linux RAID, Hardware RAID, Software RAID etc.,? +It works, but we strongly recommend to use raw devices for better performance. + +### Is LUKS device supported? +Yes + +### I am already using Local Persistent Volumes (Local PV) for storage. Why do I need DirectPV? +Local Persistent Volumes are statically created `PersistentVolume` which requires administrative skills. Whereas DirectPV dynamically provisions volumes on-demand; they are persistent through pod/node restarts. The lifecycle of DirectPV volumes are managd by associated Persistent Volume Claims (PVCs) which simplifies volume management. + +### I see `no drive found ...` error message in my Persistent Volume Claim. Why? +Below is the reason and solution +| Reason | Solution | +|:-------------------------------------------------------------|:---------------------------------------------------| +| Volume claim is made without adding any drive into DirectPV. | Please add drives. | +| No drive has free space for requested size. | 1. Please add new drives. 2. Remove stale volumes. | +| Requested topology is not met. | Please modify your Persistent Volume Claim. | +| Requested drive is not found on the requested node. | Please modify Persistent Volume Claim. | +| Requested node is not DirectPV node. | Please modify Persistent Volume Claim. | + +### I see Persistent Volume Claim is created, but respective DirectPV volume is not created. Why? +DirectPV comes with [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) volume binding mode i.e. Pod consuming volume must be scheduled first. + +### I see volume consuming Pod still in `Pending` state. Why? +* If you haven't created the respective Persistent Volume Claim, create it. +* You may be facing Kubernetes scheduling problem. Please refer to the Kubernetes documentation [on scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) + +### I see `volume XXXXX is not yet staged, but requested with YYYYY` error. Why? +According to CSI specification, `Kubelet` should call `StageVolume` RPC first, then `PublishVolume` RPC next. In a rare event, `StageVolume` RPC is not fired/called, but `PublishVolume` RPC is called. Please restart your Kubelet and report this issue to your Kubernetes provider. + +### I see ```unable to find device by FSUUID xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; either device is removed or run command `sudo udevadm control --reload-rules && sudo udevadm trigger` on the host to reload``` error. Why? +In a rare event, `Udev` in your system missed updating `/dev` directory. Please run command `sudo udevadm control --reload-rules && sudo udevadm trigger` and report this issue to your OS vendor. diff --git a/docs/index.md b/docs/index.md deleted file mode 100644 index 73e9a0d3b..000000000 --- a/docs/index.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Documentation ---- - -Documentation -------------- - -### Setup and Usage - - [Installation](./installation.md) - - [CLI reference](./cli.md) - - [Scheduling](./scheduling.md) - - [Version upgrade](./upgrade.md) - -### Advanced - - [Architecture](./architecture.md) - - [Security](./security.md) - - [CSI Specification](./specification.md) - - [Metrics](./metrics.md) - - [Troubleshooting](./troubleshooting.md) - - [Drive States](./drive-states.md) - - - -### References - - [Kubernetes CSI](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) diff --git a/docs/installation.md b/docs/installation.md index 22b7fc010..e2477d598 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,150 +1,104 @@ ---- -title: Installation ---- +# Installation +DirectPV comes with two components: +1. DirectPV plugin - installed on client machine. +2. DirectPV CSI driver - installed on Kubernetes cluster. -Installation -------------- +## DirectPV plugin installation +The plugin needs to be installed to manage DirectPV CSI driver in Kubernetes. ### Prerequisites +* Access to Kubernetes cluster. -| Name | Version | -| -------------|----------| -| kubectl | v1.19+ | -| kubernetes | v1.19+ | - - -### Plugin Installation - -DirectPV plugin can be installed from kubectl krew index using: - +### Installation using `Krew` +The latest DirectPV plugin is available in `Krew` repository. Use below steps to install the plugin in your system. ```sh -kubectl krew install directpv -``` - -After running this installation: - - - The `directpv` plugin will be installed in your krew installation directory (default: `$HOME/.krew`) - - Run `kubectl directpv --version` to verify that the installation worked - - If the error `Error: unknown command "directpv" for "kubectl"` is shown, try adding `$HOME/.krew/bin` to your `$PATH` - -### Driver Installation - -For installation in production grade environments, ensure that all criteria in the [Production Readiness Checklist](#production-readiness-checklist) are satisfied. - -(NOTE: Please refer the [CLI Guide](./cli.md) for more helpers on the following commands.) +# Update the plugin list. +$ kubectl krew update -#### 1. Install the driver - -```sh -kubectl directpv install +# Install DirectPV plugin. +$ kubectl krew install directpv ``` -This will install `directpv-min-io` CSI driver in the kubernetes cluster. - -**Notes:** - - - DirectPV components are installed in the namespace `directpv` - - alternate kubeconfig path can be specified using `kubectl directpv --kubeconfig /path/to/kubeconfig` - - the rbac requirements for the driver is [here](./specification.md#driver-rbac) - - the driver runs in `privileged` mode, which is required for mounting, unmounting and formatting drives - - The deamonset used by directpv will be using the ports `10443` for metrics and `30443` for readiness handlers. Make sure these ports are open on the nodes. - -#### 2. List discovered drives +### Installation of release binary +The plugin binary name starts by `kubectl-directpv` and is available at https://github.com/minio/directpv/releases/latest. Download the binary as per your operating system and architecture. Below is an example for `GNU/Linux` on `amd64` architecture: ```sh -kubectl directpv discover -``` +# Download DirectPV plugin. +$ release=$(curl -sfL "https://api.github.com/repos/minio/directpv/releases/latest" | awk '/tag_name/ { print substr($2, 3, length($2)-4) }') +$ curl -fLo kubectl-directpv https://github.com/minio/directpv/releases/download/v${release}/kubectl-directpv_${release}_linux_amd64 -This will list all available drives in the kubernetes cluster and will generate an init config file (default: drives.yaml) to initialize these drives. +# Make the binary executable. +$ chmod a+x kubectl-directpv +``` -#### 3. Intitialize the drives +## DirectPV CSI driver installation +Before starting the installation, it is required to have DirectPV plugin installed on your system. For plugin installation refer [this documentation](#directpv-plugin-installation). If you are not using `krew`, replace `kubectl directpv` by `kubectl-directpv` in below steps. +### Prerequisites +* Kubernetes >= v1.18 on GNU/Linux on amd64. +* If you use private registry, below images must be pushed into your registry. You could use [this helper script](tools/push-images.sh) to do that. + - quay.io/minio/csi-node-driver-registrar:v2.8.0 + - quay.io/minio/csi-provisioner:v3.5.0 _(for Kubernetes >= v1.20)_ + - quay.io/minio/csi-provisioner:v2.2.0-go1.18 _(for kubernetes < v1.20)_ + - quay.io/minio/livenessprobe:v2.10.0 + - quay.io/minio/csi-resizer:v1.8.0 + - quay.io/minio/directpv:latest +* If `seccomp` is enabled, load [DirectPV seccomp profile](../seccomp.json) on nodes where you want to install DirectPV and use `--seccomp-profile` flag to `kubectl directpv install` command. For more information, refer Kubernetes documentation [here](https://kubernetes.io/docs/tutorials/clusters/seccomp/) +* If `apparmor` is enabled, load [DirectPV apparmor profile](../apparmor.profile) on nodes where you want to install DirectPV and use `--apparmor-profile` flag to `kubectl directpv install` command. For more information, refer to the [Kubernetes documentation](https://kubernetes.io/docs/tutorials/clusters/apparmor/). +* Enabled `ExpandCSIVolumes` [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) for [volume expansion](https://kubernetes-csi.github.io/docs/volume-expansion.html) feature. +* Review the [driver specification documentation](./specification.md) +* For Red Hat Openshift users, refer to the [Openshift specific documentation](./openshift.md) for configuration prior to install DirectPV. + +### Default installation +To install DirectPV in all Kubernetes nodes, run ```sh -kubectl directpv init drives.yaml +$ kubectl directpv install ``` -This will initialize selected drives in drives.yaml - -**Notes:** - - - initialization will erase all data on the drives. Double check to make sure that only intended drives are specified - -#### 4. Verify installation +### Customized installation +To install DirectPV on selected Kubernetes nodes and/or with tolerations and/or with non-standard `kubelet` directory, use below steps accordingly. +#### Installing on selected nodes +To install DirectPV on selected nodes, use `--node-selector` flag to `install` command. Below is an example: ```sh -kubectl directpv info +# Install DirectPV on nodes having label 'group-name' key and 'bigdata' value +$ kubectl directpv install --node-selector group-name=bigdata ``` -This will show information about the drives formatted and added to DirectPV. - -After running this installation: - - - storage class named `directpv-min-io` is created - - `directpv-min-io` can be specified as the `storageClassName` in in `PodSpec.VolumeClaimTemplates` to provision DirectPV volumes - - example statefulset using `directpv-min-io` can be found [here](../minio.yaml#L61) - - optional: view the [driver specification](./specification.md) - - -## Air-gapped Installation (Private Registry) - -Push the following images to your private registry - - - quay.io/minio/csi-node-driver-registrar:v2.8.0 - - quay.io/minio/csi-provisioner:v3.5.0 - - quay.io/minio/livenessprobe:v2.10.0 - - quay.io/minio/csi-resizer:v1.8.0 - - quay.io/minio/directpv:${latest_tag_name} - - **Notes:** - - - If your kubernetes version is less than v1.20, you need push `quay.io/minio/csi-provisioner:v2.2.0-go1.18` - -Here is a shell script to Copy-Paste into your terminal to do the above steps: +#### Installing on tainted nodes +To install DirectPV on tainted nodes, use `--toleration` flag to `install` command. Below is an example: ```sh -/bin/bash -e - -# set this to private registry URL (the URL should NOT include http or https) -if [ -z $PRIVATE_REGISTRY_URL ]; then "PRIVATE_REGISTRY_URL env var should be set"; fi - -images[0]=quay.io/minio/csi-node-driver-registrar:v2.8.0 -images[1]=quay.io/minio/csi-provisioner:v3.5.0 -images[2]=quay.io/minio/livenessprobe:v2.10.0 -images[3]=quay.io/minio/csi-resizer:v1.8.0 -images[4]=quay.io/minio/directpv:$(curl -s "https://api.github.com/repos/minio/directpv/releases/latest" | grep tag_name | sed -E 's/.*"([^"]+)".*/\1/') +# Install DirectPV on tainted nodes by tolerating 'key1' key, 'Equal' operator for 'value1' value with 'NoSchedule' effect +$ kubectl directpv install --tolerations key1=value1:NoSchedule -function privatize(){ echo $1 | sed "s#quay.io#${PRIVATE_REGISTRY_URL}#g"; } -function pull_tag_push(){ docker pull $1 && docker tag $1 $2 && docker push $2; } -for image in ${images[*]}; do pull_tag_push $image $(privatize $image); done +# Install DirectPV on tainted nodes by tolerating 'key2' key, 'Exists' operator with 'NoExecute' effect +$ kubectl directpv install --tolerations key2:NoExecute ``` -## Custom Installation - -If any other customization is desired, - -Step 1: Generate the specification for installing DirectPV +#### Installing on non-standard `kubelet` directory +To install on non-standard `kubelet` directory, set the `KUBELET_DIR_PATH` environment variable and start the installation. Below is an example: ```sh -$ kubectl directpv install -o yaml > directpv-install.yaml +$ export KUBELET_DIR_PATH=/path/to/my/kubelet/dir +$ kubectl directpv install ``` -Step 2: Make appropriate changes to the resources -``` -$ emacs directpv-install.yaml +#### Installing on Openshift +To install DirectPV on Openshift with specific configuration, use the `--openshift` flag. Below is an example: +```sh +$ kubectl directpv install --openshift ``` -Step 3: Install DirectPV -``` -$ kubectl create -f directpv-install.yaml +#### Installing by generating DirectPV manifests +To install using generated manifests file, run below command +```sh +$ curl -sfL https://github.com/minio/directpv/raw/master/docs/tools/install.sh | sh - apply ``` -**Notes:** - -- Client-side upgrade functionality will not be available for custom installations. You need to execute `kubectl directpv migrate` to migrate the old resources. - -## Production Readiness Checklist - -Make sure the following check-boxes are ticked before production deployment +## What's next +* [Add drives](./drive-management.md#add-drives) +* [Provision volumes](./volume-provisioning.md) - - [ ] If using a private registry, all the images listed in [air-gapped installation](#air-gapped-installation-private-registry) should be available in the private registry - - [ ] If seccomp is enabled in the system, DirectPV [seccomp policy](../seccomp.json) should be loaded on all nodes. Instructions available [here](https://kubernetes.io/docs/tutorials/clusters/seccomp/) - - [ ] If apparmor is enabled in the system, DirectPV [apparmor profile](../apparmor.profile) should be loaded on all nodes. Instructions available [here](https://kubernetes.io/docs/tutorials/clusters/apparmor/) - - [ ] Review and Sign-off the [Security Checklist](../security-checklist.md) for providing elevated privileges to DirectPV. +## Further reads +* [Drive management guide](./drive-management.md) +* [Volume management guide](./volume-management.md) +* [Troubleshooting guide](./faq.md) diff --git a/docs/security.md b/docs/issue-reporting.md similarity index 79% rename from docs/security.md rename to docs/issue-reporting.md index 5d24e9744..83dc891b8 100644 --- a/docs/security.md +++ b/docs/issue-reporting.md @@ -1,23 +1,13 @@ ---- -title: Security ---- +# Issue reporting -Security ---------- +## Reporting bugs +Please open Github issue at https://github.com/minio/directpv/issues -**Important**: Report security issues to security@min.io. Please do not report security issues here. +## Reporting security issues -## Supported Versions +The [latest release](https://github.com/minio/minio/releases/latest) of DirectPV comes with security and vulnerability fixes in previous releases. It is strongly recommended to use the latest version. Before reporting security/vulnerability issues, make sure those issues are not fixed in the latest release. -We always provide security updates for the [latest release](https://github.com/minio/minio/releases/latest). -Whenever there is a security update you just need to upgrade to the latest version. - -## Reporting a Vulnerability - -All security bugs in [minio/directpv](https://github,com/minio/directpv) (or other minio/* repositories) -should be reported by email to security@min.io. Your email will be acknowledged within 48 hours, -and you'll receive a more detailed response to your email within 72 hours indicating the next steps -in handling your report. +All security issues should be reported by email to security@min.io. Your email will be acknowledged within 48 hours, and you'll receive a more detailed response to your email within 72 hours indicating the next steps in handling your report. DO NOT OPEN GITHUB ISSUES for security bugs. Please, provide a detailed explanation of the issue. In particular, outline the type of the security issue (DoS, authentication bypass, information disclose, ...) and the assumptions you're making (e.g. do @@ -47,15 +37,14 @@ This process can take some time, especially when coordination is required with m Every effort will be made to handle the bug in as timely a manner as possible, however it's important that we follow the process described above to ensure that disclosures are handled consistently. -## Vulnerability Management Policy -------------------------------------- +### Vulnerability Management Policy This document formally describes the process of addressing and managing a reported vulnerability that has been found in the MinIO server code base, any directly connected ecosystem component or a direct / indirect dependency of the code base. -### Scope +#### Scope The vulnerability management policy described in this document covers the process of investigating, assessing and resolving a vulnerability report @@ -64,7 +53,7 @@ opened by a MinIO employee or an external third party. Therefore, it lists pre-conditions and actions that should be performed to resolve and fix a reported vulnerability. -### Vulnerability Management Process +#### Vulnerability Management Process The vulnerability management process requires that the vulnerability report contains the following information: diff --git a/docs/metrics.md b/docs/monitoring.md similarity index 57% rename from docs/metrics.md rename to docs/monitoring.md index b9bbfb35e..113927b95 100644 --- a/docs/metrics.md +++ b/docs/monitoring.md @@ -1,22 +1,34 @@ ---- -title: Metrics ---- +# Monitoring using Prometheus -Monitoring guidelines ----------------------- +DirectPV nodes export Prometheus compatible metrics data via port `10443`. The metrics data includes +* directpv_stats_bytes_used +* directpv_stats_bytes_total +and categorized by labels `tenant`, `volumeID` and `node`. -DirectPV nodes export Prometheus compatible metrics data by exposing a metrics endpoint at /directpv/metrics. Users looking to monitor their tenants can point Prometheus configuration to scrape data from this endpoint. +To scrape data in Prometheus, each node must be accessible by port `10443`. A simple example is below -DirectPV node server exports the following metrics +1. Make node server metrics port accessible by localhost:8080 +``` +$ kubectl -n directpv port-forward node-server-4nd6q 8080:10443 +``` -- directpv_stats_bytes_used -- directpv_stats_bytes_total +2. Add below YAML configuration into Prometheus configuration. +```yaml +scrape_configs: + - job_name: 'directpv-monitor' + # Override the global default and scrape targets from this job every 5 seconds. + scrape_interval: 5s + static_configs: + - targets: ['localhost:8080'] + labels: + group: 'production' +``` -These metrics are categorized by labels ['tenant', 'volumeID', 'node']. These metrics will be representing the volume stats of the published volumes. +3. Run `directpv_stats_bytes_total{node="node-3"}` promQL in Prometheus web interface. -Please apply the following Prometheus config to scrape the metrics exposed. +Below is an example comprehensive YAML configuration: -``` +```yaml global: scrape_interval: 15s external_labels: @@ -67,17 +79,3 @@ scrape_configs: action: replace target_label: kubernetes_name ``` - -For example, use the following promQL to query the volume stats. - -- To filter out the volumes scheduled in `node-3` node :- - -``` -directpv_stats_bytes_total{node="node-3"} -``` - -- To filter out the volumes of tenant `tenant-1` scheduled in `node-5` node :- - -``` -directpv_stats_bytes_used{tenant="tenant-1", node="node-5"} -``` diff --git a/docs/node-management.md b/docs/node-management.md new file mode 100644 index 000000000..e333263b7 --- /dev/null +++ b/docs/node-management.md @@ -0,0 +1,33 @@ +# Node management + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer to the [plugin installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer to the [driver installation guide](./installation.md#directpv-csi-driver-installation). + +## Add node +After Adding a node into DirectPV DaemonSet, run DirectPV plugin `discover` command. Below is an example: +```sh +$ kubectl directpv discover +``` + +## List node +Run DirectPV plugin `info` command to get list of nodes. Below is an example: +```sh +$ kubectl directpv info +┌──────────┬──────────┬───────────┬─────────┬────────┐ +│ NODE │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │ +├──────────┼──────────┼───────────┼─────────┼────────┤ +│ • master │ 512 MiB │ 32 MiB │ 2 │ 1 │ +│ • node1 │ 512 MiB │ 32 MiB │ 2 │ 1 │ +└──────────┴──────────┴───────────┴─────────┴────────┘ + +64 MiB/1.0 GiB used, 4 volumes, 2 drives + +``` + +Refer to the [info command](./command-reference.md#info-command) for more information. + +## Delete node +***CAUTION: THIS IS DANGEROUS OPERATION WHICH LEADS TO DATA LOSS*** + +Before removing a node make sure no volumes or drives on the node are in use, then remove the node from DirectPV DaemonSet and run [remove-node.sh](./tools/remove-node.sh) script. diff --git a/docs/openshift.md b/docs/openshift.md new file mode 100644 index 000000000..80be0a65f --- /dev/null +++ b/docs/openshift.md @@ -0,0 +1,9 @@ +# DirectPV on Red Hat OpenShift +DirectPV runs under project `directpv` in Red Hat OpenShift. Project `directpv` is automatically created after successful DirectPV installation. + +## Settings required +* Add privileges to `directpv` namespace and DirectPV service account by adding `system:serviceaccount:directpv:directpv-min-io` to `users` section of YAML generated by command `$ oc edit scc privileged` + +## Limitations +* DirectPV does not support volume snapshot feature as per CSI specification. DirectPV is specifically meant for use cases like MinIO where the data availability and resiliency is taken care by the application itself. Additionally, with the AWS S3 versioning APIs and internal healing, snapshots is not a requirement. +* DirectPV does not support `ReadWriteMany` volume access mode. The workloads using DirectPV run local to the node and are provisioned from local storage drives in the node. This allows the workloads to directly access data without any additional network hops, unlike remote volumes, network PVs, etc. The additional network hops may lead to poor performance and increases the complexity. With `ReadWriteOnce` access mode, DirectPV provides high performance storage for Pods. diff --git a/docs/scheduling.md b/docs/scheduling.md deleted file mode 100644 index 3ff0cd6ca..000000000 --- a/docs/scheduling.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Scheduling ---- - -Scheduling guidelines -------------- - -### Volume scheduling based on drive labels - -In addition to scheduling based on resource constraints (available space) and node topology (affinity/anti-affinity etc.), it is possible to further influence the scheduling of workloads to specific volumes based on drive labels. The DirectPV drives can be labeled based on its classification and such labels can be used in the storage class parameters to control scheduling to pick up the chosen drives for volumes. - -By default, the DirectPV drives will not have any user defined labels set on them. Use `kubectl directpv label drives` command to set user defined labels to DirectPV drives. - -**Notes:** - -- This applies only for creating new volumes as this is a schedule time process. -- In the following example, please replace the place holders ``,`` and `` with appropriate values based on the classification you chose - -#### Step 1: Set the label on the DirectPV drive(s) - -```sh -kubectl directpv label drives = --drives /dev/ -``` - -To Verify if the labels are properly set, list the drives with `--show-labels` flag - -```sh -kubectl directpv list drives --drives /dev/ --show-labels -``` - -#### Step 2: Set the 'directpv-min-io/: ' parameter in storage class definition - -Create a storage class with the following parameter set - -```yaml -parameters: - directpv.min.io/: -``` - -For example, create a new storage class with the `parameters` section - -(NOTE: Please refer the exiting storage class `kubectl get storageclass directpv-min-io -n directpv -o yaml` to compare and check if all the fields are present on the new storage class) - -```yaml -allowVolumeExpansion: false -allowedTopologies: -- matchLabelExpressions: - - key: directpv.min.io/identity - values: - - directpv-min-io -apiVersion: storage.k8s.io/v1 -kind: StorageClass -metadata: - finalizers: - - foregroundDeletion - labels: - application-name: directpv.min.io - application-type: CSIDriver - directpv.min.io/created-by: kubectl-directpv - directpv.min.io/version: v1beta1 - name: directpv-min-io-new # Please choose any storage class name of your choice - resourceVersion: "511457" - uid: e93d8dab-b182-482f-b8eb-c69d4a1ec62d -parameters: - fstype: xfs - directpv.min.io/: -provisioner: directpv-min-io -reclaimPolicy: Delete -volumeBindingMode: WaitForFirstConsumer -``` - -#### Step 3: Deploy the workload with the new storage class name set - -You will see volumes placed on the labeled drives only. You can verify this by the following command - -```sh -kubectl directpv list drives --labels : -kubectl directpv list volumes -``` diff --git a/docs/specification.md b/docs/specification.md index 6205a8eb3..b060fe911 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -1,10 +1,6 @@ ---- -title: Driver Specification ---- +# DirectPV CSI Driver Specification -## Driver Specification - -### CSIDriver +## CSIDriver | Key | Value | |---------------------|---------------------------| @@ -16,55 +12,55 @@ title: Driver Specification | `storageCapacity` | `false` | | `modes` | `Persistent`, `Ephemeral` | -### StorageClass +## StorageClass -| Key | Value | -|------------------------|--------------------------| -| `name` | `directpv-min-io` | -| `provisioner` | `directpv-min-io` | -| `reclaimPolicy` | `Delete` | -| `allowVolumeExpansion` | `false` | -| `volumeBindingMode` | `WaitForFirstConsumer` | +| Key | Value | +|------------------------|------------------------| +| `name` | `directpv-min-io` | +| `provisioner` | `directpv-min-io` | +| `reclaimPolicy` | `Delete` | +| `allowVolumeExpansion` | `true` | +| `volumeBindingMode` | `WaitForFirstConsumer` | -### DirectPVDrives CRD +## DirectPVDrives CRD -| Key | Value | -| -------------|-----------------------| -| `name` | `directpvdrives` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|-------------------| +| `name` | `directpvdrives` | +| `apigroup` | `directpv.min.io` | -### DirectPVVolumes CRD +## DirectPVVolumes CRD -| Key | Value | -| -------------|-----------------------| -| `name` | `directpvvolumes` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|-------------------| +| `name` | `directpvvolumes` | +| `apigroup` | `directpv.min.io` | -### DirectPVNodes CRD +## DirectPVNodes CRD -| Key | Value | -| -------------|-----------------------| -| `name` | `directpvnodes` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|-------------------| +| `name` | `directpvnodes` | +| `apigroup` | `directpv.min.io` | -### DirectPVInitRequests CRD +## DirectPVInitRequests CRD -| Key | Value | -| -------------|------------------------| -| `name` | `directpvinitrequests` | -| `apigroup` | `directpv.min.io` | +| Key | Value | +|------------|------------------------| +| `name` | `directpvinitrequests` | +| `apigroup` | `directpv.min.io` | -### Driver RBAC +## Driver RBAC -| apiGroup | Resources | Verbs | -| --------------------------|-----------------------------|------------------------------------------------------| -| (core) | `endpoints` | `get`, `list`, `watch`, `create`, `update`, `delete` | -| (core) | `events` | `list`, `watch`, `create`, `update`, `patch` | -| (core) | `nodes` | `get`, `list`, `watch` | -| (core) | `persistentvolumes` | `get`, `list`, `watch`, `create`, `delete` | -| (core) | `persistentvolumeclaims` | `get`, `list`, `watch`, `update` | -| (core) | `pods,pod` | `get`, `list`, `watch` | -| `policy` | `podsecuritypolicies` | `use` | +| apiGroup | Resources | Verbs | +|---------------------------|-----------------------------|------------------------------------------------------| +| (core) | `endpoints` | `get`, `list`, `watch`, `create`, `update`, `delete` | +| (core) | `events` | `list`, `watch`, `create`, `update`, `patch` | +| (core) | `nodes` | `get`, `list`, `watch` | +| (core) | `persistentvolumes` | `get`, `list`, `watch`, `create`, `delete` | +| (core) | `persistentvolumeclaims` | `get`, `list`, `watch`, `update` | +| (core) | `pods,pod` | `get`, `list`, `watch` | +| `policy` | `podsecuritypolicies` | `use` | | `apiextensions.k8s.io` | `customresourcedefinitions` | `get`, `list`, `watch`, `create`, `update`, `delete` | | `coordination.k8s.io` | `leases` | `get`, `list`, `watch`, `update`, `delete`, `create` | | `directpv.min.io` | `directpvdrives` | `get`, `list`, `watch`, `create`, `update`, `delete` | @@ -77,5 +73,4 @@ title: Driver Specification | `storage.k8s.io` | `storageclasses` | `get`, `list`, `watch` | | `storage.k8s.io` | `volumeattachments` | `get`, `list`, `watch` | - -The service account binded to the above clusterrole is `directpv-min-io` in `directpv` namespace and the corresponding clusterrolebinding is `directpv-min-io` +The service account binded to the above clusterrole is `directpv-min-io` in `directpv` namespace and the corresponding clusterrolebinding is `directpv-min-io`. diff --git a/docs/tools/create-storage-class.sh b/docs/tools/create-storage-class.sh new file mode 100755 index 000000000..fc4739c96 --- /dev/null +++ b/docs/tools/create-storage-class.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# +# This file is part of MinIO DirectPV +# Copyright (c) 2023 MinIO, Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +set -e -C -o pipefail + +declare NAME DRIVE_LABEL + +function init() { + if [[ $# -ne 2 ]]; then + cat < + +ARGUMENTS: + NAME new storage class name. + DRIVE-LABEL drive labels to be attached. + +EXAMPLE: + # Create new storage class 'fast-tier-storage' with drive labels 'directpv.min.io/tier: fast' + $ create-storage-class.sh fast-tier-storage 'directpv.min.io/tier: fast' +EOF + exit 255 + fi + + NAME="$1" + DRIVE_LABEL="$2" + + if ! which kubectl >/dev/null 2>&1; then + echo "kubectl not found; please install" + exit 255 + fi +} + +function main() { + kubectl apply -f - <. - -# -# This script drains the DirectPV resources from a selected node -# -# **CAUTION** -# -# This operation is irreversible and may incur data loss if not used cautiously. -# - -set -e -C -o pipefail - - -function drain() { - selector="directpv.min.io/node=${2}" - - # unset the finalizers - kubectl get "${1}" --selector="${selector}" -o custom-columns=NAME:.metadata.name --no-headers | while read -r resource_name; do - kubectl patch "${1}" "${resource_name}" -p '{"metadata":{"finalizers":null}}' --type=merge - done - - # delete the objects - kubectl delete "${1}" --selector="${selector}" --ignore-not-found=true -} - -function init() { - - if [[ $# -ne 1 ]]; then - echo "usage: drain.sh " - echo - echo "This script forcefully removes all the DirectPV resources from the node" - echo "This operation is irreversible and may incur data loss if not used cautiously." - exit 255 - fi - - if ! which kubectl >/dev/null 2>&1; then - echo "kubectl not found; please install" - exit 255 - fi - - if kubectl get csinode "${1}" -o go-template="{{range .spec.drivers}}{{if eq .name \"directpv-min-io\"}}{{.name}}{{end}}{{end}}" --ignore-not-found | grep -q .; then - echo "the node is still under use by DirectPV CSI Driver; please remove DirectPV installation from the node to drain" - exit 255 - fi -} - -function main() { - node="$1" - - drain "directpvvolumes" "${node}" - drain "directpvdrives" "${node}" - drain "directpvinitrequests" "${node}" - kubectl delete directpvnode "${node}" --ignore-not-found=true -} - -init "$@" -main "$@" diff --git a/docs/tools/push-images.sh b/docs/tools/push-images.sh new file mode 100755 index 000000000..706affac9 --- /dev/null +++ b/docs/tools/push-images.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# +# This file is part of MinIO DirectPV +# Copyright (c) 2023 MinIO, Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +# +# This script pushes DirectPV and its sidecar images to private registry. +# + +set -o errexit +set -o nounset +set -o pipefail + +declare registry podman + +function init() { + if [ "$#" -ne 1 ]; then + cat < + +ARGUMENT: + Image registry without 'http' prefix. + +EXAMPLE: +$ push-images.sh example.org/myuser +EOF + exit 255 + fi + registry="$1" + + if which podman >/dev/null 2>&1; then + podman=podman + elif which docker >/dev/null 2>&1; then + podman=docker + else + echo "no podman or docker found; please install" + exit 255 + fi +} + +# usage: push_image +function push_image() { + image="$1" + private_image="${image/quay.io\/minio/$registry}" + echo "Pushing image ${image}" + "${podman}" pull --quiet "${image}" + "${podman}" tag "${image}" "${private_image}" + "${podman}" push --quiet "${private_image}" +} + +function main() { + push_image "quay.io/minio/csi-node-driver-registrar:v2.8.0" + push_image "quay.io/minio/csi-provisioner:v3.5.0" + push_image "quay.io/minio/csi-provisioner:v2.2.0-go1.18" + push_image "quay.io/minio/livenessprobe:v2.10.0" + push_image "quay.io/minio/csi-resizer:v1.8.0" + release=$(curl -sfL "https://api.github.com/repos/minio/directpv/releases/latest" | awk '/tag_name/ { print substr($2, 3, length($2)-4) }') + push_image "quay.io/minio/directpv:v${release}" +} + +init "$@" +main "$@" diff --git a/docs/tools/remove-node.sh b/docs/tools/remove-node.sh new file mode 100755 index 000000000..4880180bd --- /dev/null +++ b/docs/tools/remove-node.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# +# This file is part of MinIO DirectPV +# Copyright (c) 2023 MinIO, Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +set -e -C -o pipefail + +declare NODE + +function delete_resource() { + resource="$1" + selector="directpv.min.io/node=${NODE}" + + # unset the finalizers + kubectl get "${resource}" --selector="${selector}" -o custom-columns=NAME:.metadata.name --no-headers | while read -r name; do + kubectl patch "${resource}" "${name}" -p '{"metadata":{"finalizers":null}}' --type=merge + done + + # delete the objects + kubectl delete "${resource}" --selector="${selector}" --ignore-not-found=true +} + +function init() { + if [[ $# -ne 1 ]]; then + cat < + +This script forcefully removes all the DirectPV resources from the node. +CAUTION: Remove operation is irreversible and may incur data loss if not used cautiously. +EOF + exit 255 + fi + + if ! which kubectl >/dev/null 2>&1; then + echo "kubectl not found; please install" + exit 255 + fi + + NODE="$1" + + if kubectl get --ignore-not-found=true csinode "${NODE}" -o go-template='{{range .spec.drivers}}{{if eq .name "directpv-min-io"}}{{.name}}{{break}}{{end}}{{end}}' | grep -q .; then + echo "node ${NODE} is still in use; remove node ${NODE} from DirectPV DaemonSet and try again" + exit 255 + fi +} + +function main() { + delete_resource directpvvolumes + delete_resource directpvdrives + delete_resource directpvinitrequests + kubectl delete directpvnode "${NODE}" --ignore-not-found=true +} + +init "$@" +main "$@" diff --git a/docs/upgrade.md b/docs/upgrade.md index 71cd07992..84115a628 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -1,41 +1,42 @@ ---- -title: Upgrade ---- +# Upgrade DirectPV installation -Version Upgrade ---------------- +## Upgrade DirectPV CSI driver -### Guidelines to upgrade to the latest DirectPV version - -DirectPV version upgrades are seameless and transparent. The resources will be upgraded automatically when you run the latest version over the existing resources. The latest version of DirectPV should be available in [krew](https://github.com/kubernetes-sigs/krew-index). For more details on the installation, Please refer the [Installation guidelines](./installation.md). - -NOTE: For the users who don't prefer krew, Please find the latest images in [releases](https://github.com/minio/directpv/releases). - - -#### Upgrade from versions < v3.2.x - -If you are on DirectPV version < 3.2.x, it is recommended to upgrade to v3.2.2 and then to the latest - -Please follow https://github.com/minio/directpv/blob/master_old/docs/upgrade.md#upgrade-to-v300 for the upgrade steps from legacy versions - - -#### Upgrade from v3.2.x and above - -In the latest version of DirectPV, the CSI sidecar images have been updated. If private registry is used for images, please make sure the following images are available in your registry before upgrade. +### Upgrade DirectPV CSI driver v4.x.x +#### Offline upgrade +Follow the below steps for an offline upgrade +1. Uninstall DirectPV CSI driver. +```sh +$ kubectl directpv uninstall ``` -quay.io/minio/csi-node-driver-registrar:v2.8.0 -quay.io/minio/csi-provisioner:v3.5.0 -quay.io/minio/livenessprobe:v2.10.0 -quay.io/minio/csi-resizer:v1.8.0 +2. Upgrade DirectPV plugin by [this documentation](#upgrade-directpv-plugin). +3. Install the latest DirectPV CSI driver by [this documentation](./installation.md#directpv-csi-driver-installation). + +#### In-place upgrade +Follow the below steps for an in-place upgrade +1. Upgrade DirectPV plugin by [this documentation](#upgrade-directpv-plugin). +2. Run install script with appropriate node-selector, tolerations, and `KUBELET_DIR_PATH` environment variable. Below is an example: +```sh +$ curl -sfL https://github.com/minio/directpv/raw/master/docs/tools/install.sh | sh - apply ``` -**Notes:** - -- If your kubernetes version is less than v1.20, you need push `quay.io/minio/csi-provisioner:v2.2.0-go1.18` +### Upgrade legacy DirectCSI CSI driver v3.x.x and v2.0.9 +Follow the below steps to upgrade to the latest DirectPV CSI driver from a legacy DirectCSI installation. +1. Uninstall legacy DirectCSI CSI driver. +```sh +$ kubectl directcsi uninstall +``` +2. Install the latest DirectPV plugin by [this documentation](./installation.md#directpv-plugin-installation) or upgrade existing DirectPV plugin by [this documentation](#upgrade-directpv-plugin). +3. Install the latest DirectPV CSI driver by [this documentation](./installation.md#directpv-csi-driver-installation). -- If you are on DirectPV versions < v4.0.0 and if you are using any custom storage classes for controlling volume scheduling based on access-tiers as explained [here](https://github.com/minio/directpv/blob/master_old/docs/scheduling.md), you need to make the following change to these custom storage classes. +## Upgrade DirectPV plugin -You need to change `direct.csi.min.io/access-tier: ` to `directpv.min.io/access-tier: ` in the respective storage class parameters section. +### Upgrade using `Krew` +To upgrade the plugin, run below command +```sh +$ kubectl krew upgrade directpv +``` -- The older CRDs (directcsidrives and directcsivolumes) are deprecated and not used in versions > v4.0.0, it can be removed after upgrading. Please use the [bash script](./tools/remove-directcsi.sh) to remove the older objects after upgrading to latest. +### Upgrade of release binary +Refer to the [binary installation documentation](./installation.md#installation-of-release-binary). diff --git a/docs/volume-expansion.md b/docs/volume-expansion.md deleted file mode 100644 index 038fc8c8e..000000000 --- a/docs/volume-expansion.md +++ /dev/null @@ -1,24 +0,0 @@ -Volume Expansion (applies for versions > v3.2.2) ----------------- - -DirectPV supports volume expansion [CSI feature](https://kubernetes-csi.github.io/docs/volume-expansion.html) from versions above v3.2.2. With this support, the DirectPV provisioned volumes can be expanded to the requested size claimed by the PVC. DirectPV supports "online" volume expansion where the workloads will not have any downtimes during the expansion process. - -Volume expansion requires `ExpandCSIVolumes` feature gate to be enabled (enabled by default in k8s v1.16 and above). For more details on the feature gates, please check [here](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/). - -To expand the volume, edit the storage size in the corresponding PVC spec - -```yaml -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: # Set the new size here - storageClassName: directpv-min-io - volumeMode: Filesystem - volumeName: pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -``` - -DirectPV would expand the volume and reflect the new size in `kubectl directpv list volumes pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -o wide`. The corresponding PVC would remain in "Bounded" state with the expanded size. - -(NOTE: As DirectPV supports "online" volume expansion, workload pods doesn't require restarts) diff --git a/docs/volume-management.md b/docs/volume-management.md new file mode 100644 index 000000000..5f9b70e3f --- /dev/null +++ b/docs/volume-management.md @@ -0,0 +1,158 @@ +# Volume management + +## Prerequisites +* Working DirectPV plugin. To install the plugin, refer to the [plugin installation guide](./installation.md#directpv-plugin-installation). +* Working DirectPV CSI driver in Kubernetes. To install the driver, refer to the [driver installation guide](./installation.md#directpv-csi-driver-installation). +* Added drives in DirectPV. Refer to the [drive management guide](./drive-management.md). + +## Add volume +Refer to the [volume provisioning guide](./volume-provisioning.md). + +## List volume +To get information of volumes from DirectPV, run the `list volumes` command. Below is an example: + +```sh +$ kubectl directpv list drives +┌────────┬──────┬──────┬─────────┬─────────┬─────────┬────────┐ +│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ +├────────┼──────┼──────┼─────────┼─────────┼─────────┼────────┤ +│ master │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +│ node1 │ vdb │ - │ 512 MiB │ 506 MiB │ - │ Ready │ +└────────┴──────┴──────┴─────────┴─────────┴─────────┴────────┘ +``` + +Refer to the [list volumes command](./command-reference.md#volumes-command) for more information. + +## Expand volume +DirectPV supports online volume expansion which does not require restart of pods using those volumes. This is automatically done after setting expanded size to `Persistent Volume Claim`. Below is an example: +```sh +# Get 'minio-data-1-minio-0' Persistent volume claim. +$ kubectl get pvc minio-data-1-minio-0 -o yaml > minio-data-1-minio-0.yaml + +$ cat minio-data-1-minio-0.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + annotations: + pv.kubernetes.io/bind-completed: "yes" + pv.kubernetes.io/bound-by-controller: "yes" + volume.beta.kubernetes.io/storage-provisioner: directpv-min-io + volume.kubernetes.io/selected-node: master + volume.kubernetes.io/storage-provisioner: directpv-min-io + creationTimestamp: "2023-06-08T04:46:02Z" + finalizers: + - kubernetes.io/pvc-protection + labels: + app: minio + name: minio-data-1-minio-0 + namespace: default + resourceVersion: "76360" + uid: d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 16Mi + storageClassName: directpv-min-io + volumeMode: Filesystem + volumeName: pvc-d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +status: + accessModes: + - ReadWriteOnce + capacity: + storage: 16Mi + phase: Bound + +# Edit 'minio-data-1-minio-0' PVC to increase the size from 16MiB to 64MiB. +$ cat minio-data-1-minio-0.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + annotations: + pv.kubernetes.io/bind-completed: "yes" + pv.kubernetes.io/bound-by-controller: "yes" + volume.beta.kubernetes.io/storage-provisioner: directpv-min-io + volume.kubernetes.io/selected-node: master + volume.kubernetes.io/storage-provisioner: directpv-min-io + creationTimestamp: "2023-06-08T04:46:02Z" + finalizers: + - kubernetes.io/pvc-protection + labels: + app: minio + name: minio-data-1-minio-0 + namespace: default + resourceVersion: "76360" + uid: d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 64Mi # <--- increase size here + storageClassName: directpv-min-io + volumeMode: Filesystem + volumeName: pvc-d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +status: + accessModes: + - ReadWriteOnce + capacity: + storage: 16Mi + phase: Bound + +# Apply changes +$ kubectl apply -f minio-data-1-minio-0.yaml + +# After successful expansion, you will see updated YAML +$ kubectl get pvc minio-data-1-minio-0 -o yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + annotations: + pv.kubernetes.io/bind-completed: "yes" + pv.kubernetes.io/bound-by-controller: "yes" + volume.beta.kubernetes.io/storage-provisioner: directpv-min-io + volume.kubernetes.io/selected-node: master + volume.kubernetes.io/storage-provisioner: directpv-min-io + creationTimestamp: "2023-06-08T04:46:02Z" + finalizers: + - kubernetes.io/pvc-protection + labels: + app: minio + name: minio-data-1-minio-0 + namespace: default + resourceVersion: "76651" + uid: d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 64Mi + storageClassName: directpv-min-io + volumeMode: Filesystem + volumeName: pvc-d7fad69a-d267-43c0-9baf-19fd5f65bdb5 +status: + accessModes: + - ReadWriteOnce + capacity: + storage: 64Mi # <--- increased size here + phase: Bound +``` + +## Delete volume +***CAUTION: THIS IS DANGEROUS OPERATION WHICH LEADS TO DATA LOSS*** + +Volume can be deleted only if it is in `Ready` state (that is, no pod is using it). Run the `kubectl delete pvc` command which triggers DirectPV volume deletion. As removing a volume leads to data loss, double check what volume you are deleting. Below is an example: +```sh +# Delete `sleep-pvc` volume +kubectl delete pvc sleep-pvc +``` + +## Clean stale volumes +When Pods and/or Persistent Volume Claims are deleted forcefully, associated DirectPV volumes might be left undeleted and they becomes stale. These stale volumes are removed by running `clean` command. Below is an example: +```sh +$ kubectl directpv clean --all +``` + +Refer [clean command](./command-reference.md#clean-command) for more information. diff --git a/docs/volume-provisioning.md b/docs/volume-provisioning.md new file mode 100644 index 000000000..80af64513 --- /dev/null +++ b/docs/volume-provisioning.md @@ -0,0 +1,131 @@ +# Volume provisioning +Volume provisioning involves making a `PersistentVolumeClaim` using default `directpv-min-io` storage class or custom storage class having `directpv-min-io` provisioner. As the storage classes come with volume binding mode [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode); this mode delays volume binding and provisioning of a `PersistentVolume` until a `Pod` using the `PersistentVolumeClaim` is created. PersistentVolumes will be selected or provisioned conforming to the topology that is specified by the Pod's scheduling constraints. These include, but are not limited to, resource requirements, node selectors, pod affinity and anti-affinity, and taints and tolerations. Pods consuming volumes are scheduled to nodes where volumes were scheduled. This ensures high performance data access to the pod. + +DirectPV volume in `Ready` state indicates that the volume is ready for binding to the pod. After binding, `Bound` state is set to the volume. + +## Making Persistent volume claim +PV claim must be defined with specific parameters in `PersistentVolumeClaim` specification. These parameters are + +| Parameter | Value | +|:-------------------|:---------------------------------------------------------------------------------| +| `volumeMode` | `Filesystem` | +| `storageClassName` | `directpv-min-io` or any storage class name having `directpv-min-io` provisioner | +| `accessModes` | `[ "ReadWriteOnce" ]` | + +Below is an example claiming `8MiB` storage from `directpv-min-io` storage class for `sleep-pvc` PVC: +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: sleep-pvc +spec: + volumeMode: Filesystem + storageClassName: directpv-min-io + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 8Mi +``` + +For `WaitForFirstConsumer` volume binding mode, a pod consuming `sleep-pvc` must be defined. Below is an example which uses `sleep-volume` mounted on `/mnt`: +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: sleep-pod +spec: + volumes: + - name: sleep-volume + persistentVolumeClaim: + claimName: sleep-pvc + containers: + - name: sleep-container + image: example.org/test/sleep:v0.0.1 + volumeMounts: + - mountPath: "/mnt" + name: sleep-volume +``` + +## Making Persistent volume claim in StatefulSet +PV claim must be defined with specific parameters in `volumeClaimTemplates` specification. These parameters are + +| Parameter | Value | +|:-------------------|:---------------------------------------------------------------------------------| +| `storageClassName` | `directpv-min-io` or any storage class name having `directpv-min-io` provisioner | +| `accessModes` | `[ "ReadWriteOnce" ]` | + +Below is an example claiming two `16MiB` storage from `directpv-min-io` storage class for `minio-data-1` and `minio-data-2` PVC to two `minio` pods: + +```yaml +kind: Service +apiVersion: v1 +metadata: + name: minio + labels: + app: minio +spec: + selector: + app: minio + ports: + - name: minio + port: 9000 + +--- + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: minio + labels: + app: minio +spec: + serviceName: "minio" + replicas: 2 + selector: + matchLabels: + app: minio + template: + metadata: + labels: + app: minio + directpv.min.io/organization: minio + directpv.min.io/app: minio-example + directpv.min.io/tenant: tenant-1 + spec: + containers: + - name: minio + image: minio/minio + env: + - name: MINIO_ACCESS_KEY + value: minio + - name: MINIO_SECRET_KEY + value: minio123 + volumeMounts: + - name: minio-data-1 + mountPath: /data1 + - name: minio-data-2 + mountPath: /data2 + args: + - "server" + - "http://minio-{0...1}.minio.default.svc.cluster.local:9000/data{1...2}" + volumeClaimTemplates: + - metadata: + name: minio-data-1 + spec: + storageClassName: directpv-min-io + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 16Mi + - metadata: + name: minio-data-2 + spec: + storageClassName: directpv-min-io + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 16Mi +``` + +## Further reads +* [Volume scheduling guide](./volume-scheduling.md) diff --git a/docs/volume-scheduling.md b/docs/volume-scheduling.md index 858d7b71f..3b2936b7d 100644 --- a/docs/volume-scheduling.md +++ b/docs/volume-scheduling.md @@ -1,8 +1,8 @@ # Volume Scheduling -`DirectPV` comes with storage class named `directpv-min-io` with volume binding mode [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode). This mode delays volume binding and provisioning of a `PersistentVolume` until a `Pod` using the `PersistentVolumeClaim` is created. PersistentVolumes will be selected or provisioned conforming to the topology that is specified by the Pod's scheduling constraints. These include, but are not limited to, resource requirements, node selectors, pod affinity and anti-affinity, and taints and tolerations. +`DirectPV` comes with default storage class `directpv-min-io` or custom storage class having `directpv-min-io` provisioner with volume binding mode [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode). This volume binding mode delays volume binding and provisioning of a `PersistentVolume` until a `Pod` using the `PersistentVolumeClaim` is created. PersistentVolumes will be selected or provisioned conforming to the topology that is specified by the Pod's scheduling constraints. These include, but are not limited to, resource requirements, node selectors, pod affinity and anti-affinity, and taints and tolerations. -## Drive selection +## Drive selection algorithm DirectPV CSI controller selects suitable drive for `CreateVolume` request like below 1. Filesystem type and/or access-tier in the request is validated. DirectPV supports `xfs` filesystem only. @@ -19,44 +19,76 @@ DirectPV CSI controller selects suitable drive for `CreateVolume` request like b 9. In case of parallel requests and the same drive is selected, step (6) succeeds for any one of the request and fails for rest of the requests by Kubernetes. ```text - +--------------+ - No | Is Valid | - +------------| CreateVolume | - | | request? | - | +==============+ - | | Yes - | +=======V=======+ Loop In - | | Loop each |-----------------+ - | | DirectPVDrive | | - | | CRD |<--+ +-----V-----+ - | +===============+ | | Is | +-------------+ - | | Loop Out | | requested | Yes | Return this | -+----V---+ +------V------+ | | volume |----->| drive | -| Return | Yes | Is no drive | | | present? | +-------------+ -| error |<-------| matched? | | +===========+ -+--------+ +=============+ | | No - | No | +-----V-----+ -+-----------+ +------V-------+ | No | Match by | -| Return | No | Is more than | |<------| capacity? | -| the first |<----| one drive | | +===========+ -| drive | | matched? | | | Yes -+-----^-----+ +==============+ | +-------V-------+ - | | Yes | No | Match by | - | +------V--------+ |<----| access-tier | - | | Filter drives | | | if requested? | - | | by maximum | | +===============+ - | | free capacity | | | Yes - | +---------------+ | +-------V-------+ - | | | | Match by | - | +------V-------+ | No | topology | - | No | Is more than | |<----| constraints | - +-----------| one drive | | | if requested? | - | matched? | | +===============+ - +==============+ | | Yes - | Yes | +-----V-----+ - +------V-----+ | | Append to | - | Randomly | +<-----| matched | - | select | | drives | - | a drive | +-----------+ - +------------+ + ╭╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╮ + No │ Is Valid │ + +-----------│ CreateVolume │ + | │ request? │ + | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ + | | Yes + | ╒═══════V═══════╕ Loop In + | │ Loop each │-----------------+ + | │ DirectPVDrive │ | + | │ CRD │<--+ ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ ┌─────────────┐ + | ╘═══════════════╛ | │ Is requested │ Yes │ Return this │ + | | Loop Out | │volume present?│----->│ drive │ +┌─────V─────┐ ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ └─────────────┘ +│ Return │ Yes │ Is no drive │ | | No +│ error │<----│ matched? │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ +└───────────┘ ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ | No │ Match by │ + | No |<----│ capacity? │ +┌───────────┐ ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ +│ Return │ No │ Is more than │ | | Yes +│ the first │<----│ one drive │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ +│ drive │ │ matched? │ | No │ Match by │ +└─────^─────┘ ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ |<----│ access-tier │ + | | Yes | │ if requested? │ + | ┌───────V───────┐ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ + | │ Filter drives │ | | Yes + | │ by maximum │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ + | │ free capacity │ | │ Match by │ + | └───────────────┘ | No │ topology │ + | | |<----│ constraints │ + | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | │ if requested? │ + | No │ Is more than │ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ + +-----------│ one drive │ | | Yes + │ matched? │ | ┌───────V───────┐ + ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ | │ Append to │ + | Yes +<----│ matched drives│ + ┌───────V───────┐ └───────────────┘ + │ Return │ + │ Randomly │ + │ selected drive│ + └───────────────┘ +``` + +## Customizing drive selection +Apart from controlling drive selection based on node selectors, pod affinity and anti-affinity, and taints and tolerations, drive labels are used to instruct DirectPV to pick up specific drives with custom storage class for volume scheduling. Below steps are involved for this process. + +* Label selected drives by [label drives](./command-reference.md#drives-command-1) command. Below is an example: +```sh +# Label 'nvme1n1' drive in all nodes as 'fast' value to 'tier' key. +$ kubectl directpv label drives --drives=nvme1n1 tier=fast +``` + +* Create new storage class with drive labels using [create-storage-class.sh script](../tools/create-storage-class.sh). Below is an example: +```sh +# Create new storage class 'fast-tier-storage' with drive labels 'directpv.min.io/tier: fast' +$ create-storage-class.sh fast-tier-storage 'directpv.min.io/tier: fast' +``` + +* Use newly created storage class in [volume provisioning](./volume-provisioning.md). Below is an example: +```sh +$ kubectl apply -f - <