node-disk-manager
aims to make it easy to manage the disks attached to the node. It treats disks as resources that need to be monitored and managed just like other resources like CPU, Memory and Network. It is a daemon which runs on each node, detects attached disks and loads them as Disk objects (custom resource) into Kubernetes.
While PVs are well suited for stateful workloads, the Disk objects are aimed towards helping hyper-converged Storage Operators by providing abilities like:
- Easy to access the inventory of Disks available across the Kubernetes Cluster.
- Predict failures on the Disks, to help with taking preventive actions.
- Allow for dynamically attaching/detaching Disks to a Storage Pod, without requiring a restart.
The design and implementation are currently in progress. The design is covered under this design proposal
A detailed usage documentation is maintained in the wiki.
- Edit ndm-operator.yaml to fit your environment: Set the
namespace
,serviceAccount
, configure filters in thenode-disk-manager-config-map
. - Switch to Cluster Admin context and create the DaemonSet with
kubectl create -f ndm-operator.yaml
.
kubectl get disks --show-labels
displays the disks across the cluster, withkubernetes.io/hostname
showing the node to which disk is attached.kubectl get disks -l "kubernetes.io/hostname=<hostname>"
displays the disks attached to node with the provided hostname.kubectl get disk <disk-cr-name> -o yaml
displays all the details of the disk captured byndm
for given disk resource.
-
go get
orgit clone
node-disk-manager repo into$GOPATH/src/github.com/openebs/
with one of the below directions:cd $GOPATH/src/github.com/openebs && git clone [email protected]:openebs/node-disk-manager.git
cd $GOPATH/src/github.com/openebs && go get github.com/openebs/node-disk-manager
-
Setup build tools:
- By default node-disk-manager enables fetching disk attributes using udev. This requires udev develop files. For Ubuntu,
libudev-dev
package should be installed. make bootstrap
installs the required Go tools.- NDM uses SeaChest to probe for drive details. Use the following to setup the SeaChest libraries:
pushd . cd .. git clone --recursive --branch Release-19.06.02 https://github.com/openebs/openSeaChest.git cd openSeaChest/Make/gcc make release cd ../../ sudo cp opensea-common/Make/gcc/lib/libopensea-common.a /usr/lib sudo cp opensea-operations/Make/gcc/lib/libopensea-operations.a /usr/lib sudo cp opensea-transport/Make/gcc/lib/libopensea-transport.a /usr/lib popd
- By default node-disk-manager enables fetching disk attributes using udev. This requires udev develop files. For Ubuntu,
-
run
make
in the top directory. It will:- Build the binary.
- Build the docker image with the binary.
-
Test your changes
sudo -E env "PATH=$PATH" make test
execute the unit tests.make integration-test
will launch minikube to run the tests. Make sure that minikube can be executed viasudo -E minikube start --vm-driver=none
.
By default travis pushes the docker image to openebs/node-disk-manager-amd64
, with ci as well as commit tags.
You can push to your custom registry and modify the ndm-operator.yaml file for your testing.
- Thanks to Daniel for setting up the go-based SMART library.
- Thanks to Humble, Jan and other from the Kubernetes Storage Community for reviewing the approach and evaluating the use-case.