Skip to content

Latest commit

 

History

History
 
 

design

README.md

Overview

OpenEBS is the most widely deployed open source example of a category of storage solutions sometimes called Container Attached Storage. OpenEBS is itself deployed as a set of containers on Kubernetes worker nodes. This document describes the high level architecture of OpenEBS and the links to the Source Code and its Dependencies.

Some key aspects that make OpenEBS different compared to other traditional storage solutions:

  • Built using the micro-services architecture like the applications it serves. Use Kubernetes itself to orchestrate and manage the OpenEBS components
  • Built completely in userspace making it highly portable to run across any OS / Platform
  • Completely intent driven, inheriting the same principles that drive the ease of use with Kubernetes

Architecture

The architecture of OpenEBS is container native and horizontally scalable. OpenEBS is a collection of different microservices that can be grouped into 3 major areas (or planes):

Data Engines or the Data Plane

The data engines are the containers responsible for interfacing with the underlying storage devices such as host filesystem, rotational drives, SSDs and NVMe devices. The data engines provide volumes with required capabilities like high availability, snapshots, clones, etc. Volume capabilities can be optimized based on the workload they serve. Depending on the capabilities requested, OpenEBS selects different data engines like cStor ( a CoW based) or Jiva or even Local PVs for a given volume.

The high availability is achieved by abstracting the access to the volume into the target container - which in turn does the synchronous replication to multiple different replica containers. The replica containers save the data to the underlying storage devices. If a node serving the application container and the target container fails, the application and target are rescheduled to a new node. The target connects with the other available replicas and will start serving the IO.

Storage Management or Control Plane

The Storage Management or Control Plane is responsible for interfacing between Kubernetes (Volume/CSI interface) and managing the volumes created using the OpenEBS Data Engines. The Storage Management Plane is implemented using a set of containers that are either running at the cluster level or the node level. Some of the storage management options are also provided by containers running as side-cars to the data engine containers.

The storage management containers are responsible for providing APIs for gathering details about the volumes. The APIs can be used by Kubernetes Provisioners for managing volumes, snapshots, backups, so forth; used by Prometheus to collect metrics of volumes; used by custom programs like CLI or UI to provide insights into the OpenEBS Storage status or management.

Storage Device Management Plane

While this plane is an integral part of the OpenEBS Storage Management Plane, the containers and custom resources under this plane can be used by other projects that require a Kubernetes native way of managing the Storage Devices (rotational drives, SSDs and NVMe, etc.) attached to Kubernetes nodes.

Storage Device Management Plane can be viewed as an Inventory Management tool, that discovers devices and keeps track of their usage via device claims (akin to PV/PVC concept). All the operations like device listing, identifying the topology or details of a specific device can be accessed via kubectl and Kubernetes Custom Resources.

Source Code and Dependencies

OpenEBS source code is spread across multiple repositories, organized either by the storage engine or management layer. This section describes the various actively maintained repositories.

Common or Generic

  • openebs/openebs is OpenEBS meta repository that contains design documents, project management, community and contributor documents, deployment and workload examples.
  • openebs/openebs-docs contains the source code for OpenEBS Documentation portal (https://docs.openebs.io) implemented using Docusaurus framework and other libraries listed in package.json.
  • openebs/website contains the source code for OpenEBS portal (https://openebs.io) implemented using Gatsby framework and other libraries listed in package.json.
  • openebs/charts contains the Helm chart source code for OpenEBS and also hosts a gh-pages website for install artifacts and Helm packages.
  • openebs/maya contains OpenEBS Storage Management components that help with managing cStor, Jiva and Local Volumes. This repository contains the non-CSI drivers. The code is being moved from this repository to engine specific CSI drivers. Detailed dependency list can be found in: go.mod. OpenEBS also maintains a forked copy of the Kubernetes external-storage repository to support the external-provisioners for cStor and Jiva volumes.
    • openebs/external-storage contains OpenEBS extensions for Kubernetes External Dynamic Provisioners. These provisioners will be deprecated in the near term in favor of the CSI drivers that are under beta and alpha stage at the moment. This is a forked repository from kubernetes-incubator/external-storage.
  • openebs/velero-plugin has the plugin code to perform cStor and ZFS Local PV based Backup and Restore using Velero.
  • openebs/linux-utils is a general purpose alpine container used to launch some management jobs by OpenEBS Operators.
  • openebs/api contains the OpenEBS related Kubernetes custom resource specifications and the related Go-client API to manage those resources. This functionality is being split from the mono-repo openebs/maya into its own repository.
  • openebs/upgrade contains management tools for upgrading and migrating OpenEBS volumes and pools. This funtionality is being split from the mono-repo openebs/maya into its own repository. Go dependencies are listed here.
  • openebs/e2e-tests contains the Litmus based e2e tests that are executed on GitLab pipelines. Contains tests for Jiva, cStor and Local PV.

Experimental repos

  • openebs/openebsctl contains the OpenEBS CLI that can be run as kubectl plugin. This functionality is being split from the mono-repo openebs/maya into its own repository. (Currently in Alpha).
  • openebs/performance-benchmark contains tools/scripts for running performance benchmarks on Kubernetes volumes. This is an experimental repo. The work in this repo can move to other repos like e2e-tests.
  • openebs/monitor-pv is a prometheus exporter for sending capacity usage statistics using du from hostpath volumes.
  • openebs/helm-operator is wrapper around OpenEBS Helm to allow installation the Operator Hub.

Node Disk Manager (aka Storage Device Management plane)

  • openebs/node-disk-manager contains Kubernetes native Device Inventory Management functionality. A detailed dependency list can be found in go.mod. Along with being dependent on Kubernetes and Operator SDK for managing the Kubernetes custom resources, NDM also optionally depends on the following.
    • openSeaChest (License: MPL 2.0) for discovering device attributes. OpenEBS maintains forked repositories of openSeaChest to fix/upstream the issues found in this libarary.

Jiva

  • openebs/jiva is one of the data engines supported by OpenEBS which was forked from Rancher Longhorn engine and has diverged from the way Jiva volumes are managed within Kubernetes. At the time of the fork, Longhorn was focused towards Docker and OpenEBS was focused on supporting Kubernetes. Jiva engine depends on the following:
    • A fork of Longhorn engine is maintained in OpenEBS to upstream the common changes from Jiva to Longhorn.
    • gostor/gotgt for providing user space iSCSI Target support implemented in Go. A fork of the project is maintained in OpenEBS to keep the dependencies in sync and upstream the changes.
    • longhorn/sparse-tools fork is also maintained by OpenEBS to manage the differences between Jiva way of writing into the sparse files.
    • go.mod provides a complete list of dependencies used by the Jiva project.
  • openebs/jiva-csi contains the CSI Driver for Jiva Volumes. Currently in alpha. Dependencies are in: go.mod.
  • openebs/jiva-operator contains Kubernetes custom resources and operators to manage Jiva volumes. Currently in alpha used by Jiva CSI Driver. This will replace the volume management functionality offered by OpenEBS API Server. Dependencies are in: go.mod.

cStor

Mayastor

Dynamic Local PVs

  • openebs/zfs-localpv contains the CSI driver for provisioning Kubernetes Local Volumes on ZFS installed on the nodes.
  • openebs/rawfile-localpv contains the CSI driver for provisioning Kubernetes Local Volumes on Hostpath by creating sparse files. (Currently in Alpha).

Additional details

  • Architectural overview on how each of the Data engines operate are provided in this Presentation
  • Design Documents for various components and features are listed here

Getting involved with Contributions

There is always something more that is required, to make it easier to suit your use-cases. Feel free to join the discussion on new features or raise a PR with your proposed change.