Releases: openebs/mayastor
v1.0.9
570c103
tiagolobocastro
Tiago Castro
Release v1.0.9
Release Date: 3rd Nov 2023
Summary
This patch release contains an important fixes for issues identified post the release of v1.0.8
Users of prior 1.0.x versions are advised to upgrade to this version.
Fixes
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.4_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
Known Issues
-
The Pool Operator is unable to provision pools directly using a file as the backing device. The operator attempts to validate any device path supplied in the pool specification as an accessible block device attached to the corresponding Mayastor node. In the case of a file store, there is no block device and the validation fails, causing provisioning of the pool to be aborted. This will be addressed in a future release
- workaround: Mount the file/image as a loopback device (
losetup) and use the device path of the loopback device in the pool spec
- workaround: Mount the file/image as a loopback device (
-
Deploying an application pod on a worker node which hosts both Mayastor and Prometheus exporter causes that node to restart.
- workaround: Use kernel version 5.13 or later
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrades from versions of Mayastor prior to v1.0.0 are not supported. Any release earlier than v1.0.0 should be removed prior to installing this version.
Upgrades are disruptive - data plane and control plane pods must be unscheduled, and hence all applications using Mayastor volumes must be stopped prior to performing the update. The upgrade process is manual. The K8s definition files corresponding to the deployment of Mayastor components should be patched to reflect the versions of images from this release (see the /deploy) folder in the appropriate repositories and/or the GitBook Quickstart instruction for details of which definition files are affected, and check the v1.0.8 versions to find the appropriate image tag values.
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments
v2.4.0
Release v2.4.0
Release Date: 5th Sept., 2023
Summary
The version 2.4 introduces the standard storage cloning capability to the Mayastor storage engine by implementing Kubernetes volume restore functionality. Mayastor 2.4 also introduces a framework for generating and publishing events. The release enables provisioning persistent storage for Etcd and Loki containers with OpenEBS dynamic localpv hostpath provisioner. In addition, this release has DiskPool CRD updates and fixes related to snapshots, upgrade, availability and supportability.
Features
Volume Restore from Snapshots
A snapshot of a Mayastor volume is a point-in-time read-only copy of the volume. Snapshot support was added in the previous release (v2.3). The current release adds support to create a new volume from an existing snapshot. In Kubernetes semantics, this is the equivalent of restoring a volume from a snapshot by specifying the source of the volume as a snapshot in the PVC manifest.
It is important to note that this feature is being released with the below limitation:
The current release supports creating a restore volume with only one replica. Support for restoring to multi-replica volumes will be added in the future.
It is also important to note that the StorageClass specified in the PVC for a restore volume, must have ‘thin' parameter set to 'true’.
Please refer the documentation for more details.
Eventing
Eventing is an important part of monitoring of any system. Mayastor 2.4 introduces an eventing framework which enables important events to be generated from the core components. The generated events are published onto a NATS message bus. As a part of this framework, we have introduced NATS into the system along-with a statistics module which subscribes for the events on the NATS, aggregates them and exports useful metrics in the Prometheus format.
In the current release, events are generated for creation and deletion of Mayastor pools and volumes.
Please refer the documentation for more details. Eventing can be disabled as per the steps here.
OpenEBS LocalPV Storage for Etcd and Loki
Etcd and Loki are installed as a part of the regular deployment of Mayastor. This is because, Mayastor uses etcd to persist the configuration and state settings, and Loki for storing the system logs. Both etcd and Loki are installed as Kubernetes StatefulSets and the storage for them is provisioned using either the cluster default storage class or static host path persistent volumes. There are limitations with this configuration:
A cluster default StorageClass exists only in managed Kubernetes environments (eg. Google Kubernetes Engine). The on-premise and self-managed clusters may not have a default StorageClass.
In the static host path approach, manual intervention is needed to pre-create persistent volumes on all nodes where the stateful pods of etcd and Loki could be possibly scheduled.
Mayastor 2.4 addresses this limitation using the OpenEBS dynamic localpv provisioner for creation of hostpath PVs. This provisioner is installed as a part of Mayastor 2.4 helm installation and custom storage classes are created, one each for Etcd and Loki. Using these classes, PVs are created dynamically for each statefulset pod of Etcd and Loki. The dynamic provisioning also enables easy scaling up of etcd and Loki replicas.
DiskPool CRD Update
In release 2.4, Mayastor DiskPool CustomResource version has been updated to v1beta1 (previously v1alpha1), where the “state“ field is deprecated and replaced by “cr_state“ and “pool_status“.
Related issues encountered when upgrading to version 2.4 can be resolved by following the steps documented here.
Snapshot Fixes
Fix/SPDK API to set attributes for snapshot by mtzaurus · Pull Request #1471 · openebs/mayastor
Handle discarded snapshots by tiagolobocastro · Pull Request #660 · openebs/mayastor-control-plane
Availability Fixes
Adding Command Retry Delay option feat(nvmf): adding support for configuring target CRD by dsavitskiy · Pull Request #1491 · openebs/mayastor
Supportability Fixes
Upgrade Enhancements and Fixes
refactor(upgrade-job): change dry-run helm upgrade implementation by niladrih · Pull Request #328 · openebs/mayastor-extensions
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.5_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
- 1.24.14
- 1.25.10
Known behavioural limitations
-
As with the previous versions, the Mayastor IO engine makes full utilisation of the allocated CPU cores regardless of I/O load. This is the poller operating at full speed, waiting for I/O.
-
As with the previous versions, a Mayastor DiskPool is limited to a single block device and cannot span across more than one block device.
Known issues
-
Mayastor does not support capacity expansion for volumes as of v2.4.0.
-
Mayastor does not support capacity expansion of DiskPools as of v2.4.0.
-
Under heavy IO and constant scaling up-down of volume replicas, the io-engine pod has been observed to restart occassionally.
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrad...
v2.3.0
Release v2.3.0
Release Date: 22nd July, 2023
Summary
Mayastor 2.3 introduces the industry standard storage snapshotting capabilities in the Mayastor storage engine. Kubernetes application users and administrators will now be able to create, list and delete instantaneous snapshots of Mayastor Persistent Volumes based on standard CSI specifications. Mayastor 2.3 also provides APIs to view the history of volume and replica rebuilds. In addition, this release has fixes related to HA, thin provisioning, rebuilding, stability and node drain operations.
Features
Volume Snapshots
A snapshot of a Mayastor volume is a point-in-time read-only copy of the volume. In this release, Mayastor provides support for industry standard copy-on-write (COW) snapshots, which is a popular methodology for taking snapshots by keeping track of only those blocks that have changed. Mayastor supports snapshots in a Kubernetes-native way. A snapshot controller and side-car are deployed alongside the CSI driver and the Kubernetes CRDs namely VolumeSnapshotClass, VolumeSnapshotContent and VolumeSnapshot are also deployed during the installation. By creating a custom resource of kind VolumeSnapshot, a user can easily take a snapshot of the Mayastor volume.
Please refer the documentation for more details.
It is important to note that this feature is being released with the below limitations:
The current release supports snapshots only for single-replica volumes. Support for snapshotting multi-replica volumes will be provided in the future.
The current release does not support creation of a clone or a new volume from an existing snapshot.
Rebuild History
The Mayastor volume replicas are usually distributed across storage nodes to ensure availability and guard against failures. Whenever a replica becomes inaccessible, the volume target (Mayastor nexus) marks the replica as faulted and a rebuilding process is spawned to recover the faulted replica. If the replica is rebuilt from scratch, it is known as a full rebuild process. If a replica is rebuilt from an intermediate point using a recovery log, it Is known as a partial rebuild process. Mayastor 2.3 provides API to view the history of rebuilds of all replicas attached to the current volume target (nexus). Users can invoke the Mayastor kubectl plugin with a get rebuild-history option to retrieve the rebuild records for a specific volume target.
Please refer the documentation for more details.
HA and Drain Fixes
Thin-Provisioning Fixes
Stability Fixes
Add concurrency limiter for create volume operations upto 10.
Volume Snapshot Commitment by tiagolobocastro · Pull Request #619 · openebs/mayastor-control-plane
chore(core): errno mapping for core errors by mtzaurus · Pull Request #1418 · openebs/mayastor
fix(nvmx): fixing async qpair connection bugs by dsavitskiy · Pull Request #1405 · openebs/mayastor
Supportability Fixes
Upgrade Fixes
[feat(upgrade-job)...
v1.0.8
Release v1.0.8
Release Date: 19th July 2023
Summary
This patch release contains an important fixes for issues identified post the release of v1.0.7
Users of prior 1.0.x versions are advised to upgrade to this version.
Fixes
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.4_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
Known Issues
-
The Pool Operator is unable to provision pools directly using a file as the backing device. The operator attempts to validate any device path supplied in the pool specification as an accessible block device attached to the corresponding Mayastor node. In the case of a file store, there is no block device and the validation fails, causing provisioning of the pool to be aborted. This will be addressed in a future release
- workaround: Mount the file/image as a loopback device (
losetup) and use the device path of the loopback device in the pool spec
- workaround: Mount the file/image as a loopback device (
-
Deploying an application pod on a worker node which hosts both Mayastor and Prometheus exporter causes that node to restart.
- workaround: Use kernel version 5.13 or later
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrades from versions of Mayastor prior to v1.0.0 are not supported. Any release earlier than v1.0.0 should be removed prior to installing this version.
Upgrades are disruptive - data plane and control plane pods must be unscheduled, and hence all applications using Mayastor volumes must be stopped prior to performing the update. The upgrade process is manual. The K8s definition files corresponding to the deployment of Mayastor components should be patched to reflect the versions of images from this release (see the /deploy) folder in the appropriate repositories and/or the GitBook Quickstart instruction for details of which definition files are affected, and check the v1.0.8 versions to find the appropriate image tag values.
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments
v1.0.7
Release v1.0.7
Release Date: 15th June 2023
Summary
This patch release contains an important fix for an issue identified after the release of v1.0.6
Users of prior 1.0.x versions are advised to upgrade to this version.
Fixes
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.4_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
Known Issues
-
The Pool Operator is unable to provision pools directly using a file as the backing device. The operator attempts to validate any device path supplied in the pool specification as an accessible block device attached to the corresponding Mayastor node. In the case of a file store, there is no block device and the validation fails, causing provisioning of the pool to be aborted. This will be addressed in a future release
- workaround: Mount the file/image as a loopback device (
losetup) and use the device path of the loopback device in the pool spec
- workaround: Mount the file/image as a loopback device (
-
Deploying an application pod on a worker node which hosts both Mayastor and Prometheus exporter causes that node to restart.
- workaround: Use kernel version 5.13 or later
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrades from versions of Mayastor prior to v1.0.0 are not supported. Any release earlier than v1.0.0 should be removed prior to installing this version.
Upgrades are disruptive - data plane and control plane pods must be unscheduled, and hence all applications using Mayastor volumes must be stopped prior to performing the update. The upgrade process is manual. The K8s definition files corresponding to the deployment of Mayastor components should be patched to reflect the versions of images from this release (see the /deploy) folder in the appropriate repositories and/or the GitBook Quickstart instruction for details of which definition files are affected, and check the v1.0.7 versions to find the appropriate image tag values.
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments
v2.2.0
Release v2.2.0
Release Date: 26th May 2023
Summary
Mayastor 2.2 introduces significant improvements to the replica rebuild logic in the form of in-memory rebuild log and rebuild history maintenance. Mayastor 2.2 also provides for intelligent placement of volume replicas for Kubernetes statefulset (STS) workloads for increased availability. In addition, this release has fixes related to HA, thin provisioning and node drain operations.
Features
Log-Based Replica Rebuild
A Mayastor volume comprises of an NVMe target (nexus) and desired number of replicas as specified in the storage class. The write I/Os happen synchronously on all replicas and read I/Os happen in a round-robin way across the replicas. Whenever an I/O fails on a replica, it is marked as faulted and a process called “rebuild“ is triggered to bring back the replica to a healthy state.
Prior to Mayastor 2.2, this rebuild for faulty replica has always been a complete or full rebuild i.e. each and every block is copied over from a healthy replica to the rebuilding replica. The full rebuilds can take significant amount of time depending on factors like size of replica and network bandwidth. Even a small glitch in the I/O path caused by temporary network fault or pod restart or node restart will result in replicas being faulted and full rebuilds being triggered.
Mayastor 2.2 attempts to address this situation by maintaining a rebuild log per faulted replica which is an in-memory bitmap data-structure, that significantly speeds up the rebuild process by fetching only the differential data blocks or clusters instead of the full replica data. Internal testing has shown significant improvements in rebuild durations with this approach.
It is important to note that the rebuild log is maintained in-memory on the volume target (nexus) io-engine pod and not persisted. On a restart of the nexus pod, the in-memory contents will be lost and pending rebuilds will have to be full rebuilds.
This feature is being released as an experimental one in this release. We hope to gain valuable feedback from our beloved community users. Please refer the documentation for more details.
StatefulSet Volume Affinity Group
The replicas of a Mayastor volume are always distributed across Mayastor storage nodes to ensure the maximum availability of the volume. Thus, there is an anti-affinity policy during scheduling of replicas for a single volume. When a Kubernetes statefulset (STS) application is deployed, each instance of the STS needs its own persistent volume. Prior to 2.2, volume replicas for different instances of the same STS could end up being provisioned on the same pool or node resulting in single point of failure.
Mayastor 2.2 provides a configuration to specify anti-affinity placement policy between volume replicas of different instances of the same STS. This can be enabled in the storage class by setting a parameter called ‘stsAffinityGroup' as 'true’. This ensures that the Mayastor replica placement scheduler identifies the volume requests as belonging to instances of the same STS and attempts a best-fit placement algorithm to ensure the maximum availability.
Please refer the documentation for more information.
Fixes
HA and Drain Fixes
-
Allows for a range of two controller ids while doing nexus publish feat: allow for a range of 2 controller ids when publishing by tiagolobocastro · Pull Request #524 · openebs/mayastor-control-plane
-
Switchover fixes for few corner cases Several Switchover fixes by tiagolobocastro · Pull Request #503 · openebs/mayastor-control-plane
-
Cleanup nexus fix fix(nexus_cleanup): cleanup old nexuses post volume destroy by abhilashshetty04 · Pull Request #549 · openebs/mayastor-control-plane
Thin-Provisioning Fixes
- Fix for a scenario of enospc fix(capacity/thin): don't remove enospc nexuses with no space in replicas by tiagolobocastro · Pull Request #531 · openebs/mayastor-control-plane
Stability Fixes
-
Fixing an issue with child remove logic fix(nexus): fixing child remove logic by dsavitskiy · Pull Request #1380 · openebs/mayastor
-
Fix for disabling nexus reset on initiator timeout feat(nexus): adding support for disabling nexus reset logic by dsavitskiy · Pull Request #1385 · openebs/mayastor
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.5_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
Known behavioural limitations
-
As with the previous versions, the Mayastor IO engine makes full utilisation of the allocated CPU cores even when there is less or no I/O load. This is the poller operating at full speed, waiting for I/O.
-
As with the previous versions, a Mayastor disk pool is limited to a single block device and cannot span across more than one block device.
Known issues
-
Mayastor does not support creation of volume snapshots and clones as of v2.2.0. This is a work-in-progress feature.
-
Mayastor does not support capacity expansion for volumes v2.2.0.
-
Mayastor does not support capacity expansion of disk pools as of v2.2.0.
-
Under heavy IO and constant scaling up-down of volume replicas, the io-engine pod has been observed to restart occassionally.
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrades from versions of Mayastor prior to v1.0.0 are not supported. Any release earlier than v1.0.0 should be removed prior to installing this version.
Users get the support for upgrading the software from version 2.0.0 | 2.0.1 | 2.1.0 to v2.2.0
Support For Migration From Legacy Versions
Mayastor versions 1.0.5 and prior, are being considered as legacy versions. Due to several breaking changes in the 2.0 codebase of the software, it is not possible to support seamless upgrades from the legacy versions to the current version. Mayastor 2.2 provides a documented migration path for users to move their legacy installations to the latest version.
Please refer the documentation for more information.
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments
v1.0.6
tiagolobocastro
Tiago Castro
Release v1.0.6
Release Date: 26th May 2023
Summary
This patch release contains an important fix for an issue identified after the release of v1.0.5
Users of prior 1.0.x versions are advised to upgrade to this version.
Fixes
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.4_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
Known Issues
-
The Pool Operator is unable to provision pools directly using a file as the backing device. The operator attempts to validate any device path supplied in the pool specification as an accessible block device attached to the corresponding Mayastor node. In the case of a file store, there is no block device and the validation fails, causing provisioning of the pool to be aborted. This will be addressed in a future release
- workaround: Mount the file/image as a loopback device (
losetup) and use the device path of the loopback device in the pool spec
- workaround: Mount the file/image as a loopback device (
-
Deploying an application pod on a worker node which hosts both Mayastor and Prometheus exporter causes that node to restart.
- workaround: Use kernel version 5.13 or later
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrades from versions of Mayastor prior to v1.0.0 are not supported. Any release earlier than v1.0.0 should be removed prior to installing this version.
Upgrades are disruptive - data plane and control plane pods must be unscheduled, and hence all applications using Mayastor volumes must be stopped prior to performing the update. The upgrade process is manual. The K8s definition files corresponding to the deployment of Mayastor components should be patched to reflect the versions of images from this release (see the /deploy) folder in the appropriate repositories and/or the GitBook Quickstart instruction for details of which definition files are affected, and check the v1.0.6 versions to find the appropriate image tag values.
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments
v2.1.0
tiagolobocastro
Tiago Castro
Release v2.1.0
Release Date: 27th April, 2023
Summary
Mayastor 2.1 is the first minor release on top of the 2.0 code-base. This release introduces the ability to create thinly-provisioned volumes and has seamless upgrade support from previous 2.0 versions. This release also provides a guided migration path for users running legacy Mayastor installations (v1.0.5 and below). In addition, there are stability and observability fixes as well.
Features
Thinly Provisioned Volumes
Mayastor 2.1 introduces the ability to create thin-provisioned volumes in addition to the current thickly provisioned volumes. When a thinly provisioned volume is created, space is allocated only for the metadata of the underlying replicas. As the usage increases, allocation of space happens on-demand for the new writes. The thin provisioning feature allows users to over-commit on their existing storage capacity and enables them to plan more efficiently. To track the commitment statistics, the disk pool metrics exporter now exports an additional metric named “disk_pool_committed_size“ in Bytes.
Thin provisioning is being released as an experimental feature in this release. We hope to gain valuable feedback on this from our cloud-native users. Please refer the documentation for more information.
Seamless Upgrade
With Mayastor 2.1 release, users get the support for upgrading of the software from version 2.0.0 or 2.0.1 to version 2.1.0. This upgrade process is made as seamless and non-disruptive as possible. The upgrade software checks and strives to preserve the availability of every volume during the upgrade process. Please note that the process is non disruptive only for volumes which have more than 1 replica. For volumes with a single replica, there will be a short duration when the volume will become unavailable while the upgrade is in-progress.
Please refer the documentation for more information.
Support For Migration From Legacy Versions
Mayastor versions 1.0.5 and prior, are being considered as legacy versions. Due to several breaking changes in the 2.0 codebase of the software, it is not possible to support seamless upgrades from the legacy versions to the current version. Mayastor 2.1 provides a documented migration path for users to move their legacy installations to the latest version.
Please refer the documentation for more information.
Changes
Stability fixes
- Fixes an issue where the ordering of child IO failure impacts the error handling by the nexus.
Usability Fixes
- Fixed an issue with pool destroy when it’s not yet loaded.
- Fixed an idempotency issue with volume replica destroy.
Known behavioural limitations
- As with the previous versions, the Mayastor IO engine makes full utilisation of the allocated CPU cores even when there is less or no I/O load. This is the poller operating at full speed, waiting for I/O.
- As with the previous versions, a Mayastor disk pool is limited to a single block device and cannot span across more than one block device.
Known issues
- Mayastor does not support creation of volume snapshots and clones as of v2.1.0. This is a work-in-progress feature.
- Mayastor does not support capacity expansion for volumes v2.1.0.
- Mayastor does not support capacity expansion of disk pools as of v2.1.0.
- Under heavy IO and constant scaling up-down of volume replicas, the io-engine pod has been observed to restart occasionally.
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.5_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
- 1.22.10
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments
v2.0.1
Release v2.0.1
Release Date: 14th March, 2023
Summary
The 2.0.1 release is a patch release, focusing on reconciliation-based fixes for the Mayastor diskpool operator and stability fixes for the core storage. This release can be independently installed.
Changes
DiskPool operator fixes
- Making the pool deletion idempotent.
- Handling pool import during delete operation.
- Decrypting the 'not found' errors during pool creation.
- Handling the pool deletion when the node or device is offline.
- Handling the pool deletion when there are volume replicas in the pool.
- Introduces a new field “pool_status“ in the DiskPool custom resource for better state management. Details
- The operator now indefinitely reconciles with the custom resource. Prior to 2.0.1, there was an upper limit on the retry attempts.
Stability fixes
- Fixed a crash and restart of io-engine pod under heavy load.
- Fixed a scenario where HA node agent waited indefinitely till new path is connected while replacing a broken NVMe path leading to timeouts and possible resource leaks. Now the agent waits for a timeout of 4s, which is tuneable.
Usability fixes
- The supportability plug-in now collects the running container logs (similar to 'kubectl logs <>') in addition to the logs stored on Loki, as a part of the default “kubectl mayastor dump system“ command. This is important when Loki is not accessible for some reason.
- Fixed an issue with incorrect version string exported from call-home module.
- Fixed an incorrect value related to clusterIp in Helm chart to make it compatible with Talos distribution.
Known behavioural limitations
- As with the previous versions, the Mayastor IO engine makes full utilisation of the allocated CPU cores even when there is less or no I/O load. This is the poller operating at full speed, waiting for I/O.
- As with the previous versions, a Mayastor disk pool is limited to a single block device and cannot span across more than one block device.
Known issues
- Mayastor does not support creation of thin-provisioned volumes as of v2.0.1. This a work-in-progress feature.
- Mayastor does not support creation of volume snapshots and clones as of v2.0.1. This is a work-in-progress feature.
- Mayastor does not support capacity expansion for volumes v2.0.1.
- Mayastor does not support capacity expansion of disk pools as of v2.0.1.
- Under heavy IO and constant scaling up-down of volume replicas, the io-engine pod has been observed to restart occasionally.
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.5_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
- 1.22.10
- 1.21.13
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrading from previous versions of Mayastor is not supported at this time. To use Mayastor version 2.0.1, a fresh install is required.
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments
v2.0.0
Release v2.0.0
Release Date: 8th February 2023
Summary
The 2.0.0 release is a major release, primarily focusing on enhancing the availability, stability and observability of Mayastor, the lightning fast NVMe-based block storage solution for Kubernetes stateful workloads.
Features
1. Availability features
On-demand nexus switch-over on failure detection
A nexus is a data structure created for every Mayastor volume, within the Mayastor instance, which acts as an NVMe controller and performs IO operations for that volume. Each Mayastor volume comprises of a single nexus and one or more replicas. Prior to the 2.0.0 version, nexus had been a single point of failure. To mitigate this, a switch-over(and fail-over) logic has been added in 2.0.0. With this Nexus switch-over/fail-over logic in place, if a nexus is unavailable either as a consequence of failure scenarios associated with software errors, faulty nodes/network paths/underlying storage, or planned events like upgrades, the nexus is recreated on a node that’s most optimal and applications are reconnected to the newly created nexus instances automatically, to ensure I/O continuity.
It should be noted that the switch-over will not happen if the node housing the nexus also contains the last (or only) healthy replica of the volume and that replica is currently inaccessible.
Cordon a node
In 2.0.0, Mayastor introduces the ability to cordon a node, effectively preventing any resources from being provisioned on the said node. Cordoning a node is an indication to the control plane that the node is going to have something done to it which is outside of its usual mode of operation, therefore the control plane should temporarily omit it from its scheduling logic.
Node cordoning is desirable when performing operations where the creation of new resources on the node would be problematic, such as during:
- Maintenance
- Upgrades
- Debugging
Drain a node
In 2.0.0, Mayastor introduces the ability to drain a node, effectively removing/deleting resources from the said node, such that the node ends up in an “empty“ state. Only nexus draining is supported in this release. Node draining helps lay the foundation to support non-disruptive upgrades in the future.
2. Observability features
Pool metrics exporter
A metrics exporter runs as a container in every IO-engine pod and exposes pool capacity metrics in the Prometheus format.
Volume stats exporter
Implemented NodeGetVolumeStats RPC service in Mayastor CSI node-plugin. These metrics are file-system statistics and are exposed by the kubelet on each node.
3. Stability features
- Fixed a possible deadlock scenario with NVMe controller events in IO-engine.
- Fixed a possible data corruption issue when a rebuilding replica encounters a fault, under heavy load.
- Fixed a panic in the nexus_destroy codepath.
- Fixed an issue in CSI node_unstage codepath.
- Added a Kubernetes watcher to process deletion events of Persistent Volumes, left behind by the deletion of PVCs with the reclaim policy set to “retain“.
- Fixed an issue with de-registration of gRPC service on receiving sigterm event.
- Fixed a memory corruption issue during parallel shutdown of multiple nexuses.
- Added a fix to prevent file-system errors during shutdown.
- Fixed NVMF sub-system issues with error handling during nexus unshare.
- Added allowed-hosts control to NVMe targets to prevent access from outdated nodes, as part of switchover.
- Added persistent NVMe reservations on replicas to prevent split-brain scenarios.
4. Supportability features
The 2.0.0 release provides an option in the Mayastor kubectl plug-in to collect supportability information from the cluster for better debugging purposes. The information is collected as a bundle and includes (not limited to) Mayastor component logs, command line and system outputs, Kubernetes resource specs of Mayastor resources and dump of Mayastor etcd data. For storing and retrieving historical logs, the 2.0.0. installer configures and installs a Loki stack alongside the Mayastor components.
5. Ease-of-use features
Single Helm chart-based installation
Mayastor installation process has been simplified in 2.0.0 with a single Helm chart, an enhancement from the prior releases where there were separate Helm charts for control-plane and data-plane.
NodePort dependency removal
Prior to 2.0.0, it was required to configure the etcd and api-rest as services of NodePort type in addition to opening up these ports on the firewall for accessing these services from the Mayastor kubectl-plugin, from outside the cluster. This issue has now been resolved in 2.0.0 with an enhancement to the plug-in to use HTTP and TCP port-forwarding via the kube-proxy crate.
6. Other notable changes
Component nomenclature changes
With this release, Mayastor has moved towards a more consistent and taxonomical scheme of naming for components. This scheme follows {release name}-{class}-{object} for every component. The release name is considered from the Helm chart (default is mayastor). For eg, the rest api component is now named as mayastor-api-rest, the engine mayastor has now become mayastor-io-engine and so on.
Schema changes in etcd
Mayastor uses etcd as a persistent store for storing configuration and state information of its resources like pool, volume, nexus, replica, etc. The etcd cluster is created as a part of Mayastor installation. Prior to 2.0.0, a Mayastor installation was designed to exclusively use the etcd cluster, expecting no interference from other users of etcd. This could lead to data inconsistency if a user-driven standalone etcd cluster is offered to more than one active Mayastor installations.
In 2.0.0, a separate namespace is allotted for every Mayastor cluster.
IO-engine gRPC versioning
An enhanced API scheme with versioning v1 has been introduced in release 2.0.0 for all IO-engine RPC operations, with backward-compatible support to older v0 versioning APIs.
Only NVMe, no more iSCSI
With 2.0.0, Mayastor completely removes the support for iSCSI transport.
NATS deprecated
NATS is no longer needed for communication between agent-core and IO-engine components. This communication is over gRPC transport.
Known behavioural limitations
- As with the previous versions, the Mayastor IO engine makes full utilisation of the allocated CPU cores even when there is less or no I/O load. This is the poller operating at full speed, waiting for I/O.
- As with the previous versions, a Mayastor disk pool is limited to a single block device and cannot span across more than one block device.
Known issues
- Mayastor does not support creation of thin-provisioned volumes as of v2.0.0. This a work-in-progress feature.
- Mayastor does not support creation of volume snapshots and clones as of v2.0.0. This is a work-in-progress feature.
- Mayastor does not support capacity expansion for volumes v2.0.0.
- Mayastor does not support capacity expansion of disk pools as of v2.0.0.
- Under heavy IO and constant scaling up-down of volume replicas, the io-engine pod has been observed to restart occasionally.
Testing
Mayastor is subject to extensive unit, component and system-level testing throughout the development and release cycle. Resources for system-level (E2E) testing are currently provided by DataCore Software.
At this time, personnel and hardware resource limitations constrain testing by the maintainers to linux builds on x86. This reflects the primary use-case which the maintainers are currently targeting with the OpenEBS Mayastor project. Therefore, the use of Mayastor with other operating systems and/or architectures, if even possible, should be considered serendipitous and wholly experimental.
This release has been subject to End-to-End testing under Ubuntu 20.04.5_LTS (kernel: ubuntu-5.15.0-50-generic)
- Tested k8s versions
- 1.23.7
- 1.22.10
- 1.21.13
Getting Started
Mayastor user documentation, including a quick deployment guide, can be found here
Upgrade
Upgrading from 1.0.x is not supported at this time. To use Mayastor version 2.0.0, a fresh install is required.
Support
If you are having issues during installation, configuration or upgrade, you can contact us via:
- OpenEBS on Kubernetes Slack community
- Already signed up? Head to our discussions at #openebs
- Raising an issue
"Unsupported" Architectures and Operating Systems (inc. ARM, Raspberry Pi, MacOS)
As described in the section on software testing above, the maintainers build and test Mayastor only on linux, on x86-64. The use of Mayastor in other environments is therefore not necessarily possible, at least without modification. Where possible, this is currently largely coincidental - it is not "fully" tested and therefore this should be considered an entirely experimental use-case.
The maintainers will be pleased to receive contributions in this area, with the following understanding:
- Such PR's will be reviewed for correctness, good practice, licensing compliance and general quality
- PR's will be accepted on the basis that testing by the maintainers is restricted to demonstrating no negative affect on the stability of x86-64 builds
- The maintainers will not perform acceptance testing or "positive release" of such functionality on any other OS or architecture, which is in accordance with their designation of these environments as experimental use cases at this time.
- The maintainers will not provide build artifacts or container images for these environments