Include temporary warning of stability on RHEL platforms

[landlessreptile]

Update documentation to include warning that ZFS is currently unfit for production use on RHEL systems. Closes #550

[namazso]

Not unfit, unsupported: openzfs/zfs#17360 (comment)

Since the stable version is unsupported and the testing version is recommended against on production systems as per same doc, overall it seems production EL is just not supported at all. Maybe remove the whole section / move to developer docs?

[landlessreptile]

Not unfit, unsupported: openzfs/zfs#17360 (comment)

Since the stable version is unsupported and the testing version is recommended against on production systems as per same doc, overall it seems production EL is just not supported at all. Maybe remove the whole section / move to developer docs?

I don't believe its necessary to remove the entire RHEL section, plus it would cause even more confusion - A simple warning should be sufficient. We really just want to make sure future RHEL users are aware openzfs in its current state not only does not build on RHEL, but does not match RHEL users' expectations of a trouble-free, resilient file system they can simply throw on a box and call it a day. Although this is implied in all open source volunteer-driven projects, it should be made crystal clear for all end users - especially for something as critical as a file system, and even more so for one heavily marketed for its reliability.

This is of no fault of the volunteers, and I applaud all their efforts to continue development. But it does not change the fact openzfs is, at best, not ready for production on rhel. Should that change, and minor revisions don't break builds, then obviously this can be removed.

Do you believe I should change the wording? I think this is good to start with, and then can be adjusted accordingly.

[macthecadillac]

From the docs, it is quite clear that OpenZFS is often behind on supporting new EL minor releases. In fact, if you try the command given in the documentation

dnf list available --showduplicates --releasever=9.6 kmod-zfs

or go to the repository at http://download.zfsonlinux.org/epel/, it is obvious that version 9.6 is not supported by OpenZFS yet. In addition, if you are using kmods, you can ask DNF to list the contents of the ZFS package

dnf repoquery -l kmod-zfs

which clearly shows that the current ZFS kernel module does not support patch level 570.

Some of the issues linked in your bug report seem irrelevant, since if you looked closely, you should see the latest OpenZFS version on RHEL 9.x for the moment remains on the 2.1.x branch. Whining about the 2.2.x branch not working on EL 9.6 when it is not even officially released into the EL 9.6 repo (which by the way doesn't even exist yet) is silly.

We really just want to make sure future RHEL users are aware openzfs in its current state not only does not build on RHEL, but does not match RHEL users' expectations of a trouble-free, resilient file system they can simply throw on a box and call it a day.

It seems to me you have a fundamental misunderstanding of how the EL lifecycle works. EL is stable within each minor release, but from one minor release to the next, the kernel API can change because of how Red Hat backports features and patches to their kernel, which breaks out of tree kernel modules. This is simply the nature of this release model. This is very clearly explained in the OpenZFS docs:

When updating to a new EL minor release the existing kmod packages may not work due to upstream kABI changes in the kernel. The configuration of the current release package may have already made an updated package available, but the package manager may not know to install that package if the version number isn’t newer. When upgrading, users should verify that the kmod-zfs package is providing suitable kernel modules, reinstalling the kmod-zfs package if necessary.

In my opinion, folks using EL in conjunction with third party kernel modules in production either have a Red Hat subscription and will be staying on the previous minor release using Extended Update Support (EUS) until such third party software has caught up, or they have a team that vets updates before rolling out upgrades. With this in mind, I don't think the relevant parts of the OpenZFS documentation is in any way inadequate. If you want to be able to just upgrade whenever updates arrive and expect things to simply work without having to deal with the EL lifecycle, I think you might be better off using a distribution that supports ZFS out of the box, such as Debian or Ubuntu.

[svennd]

When updating to a new EL minor release the existing kmod packages may not work due to upstream kABI changes in the kernel. The configuration of the current release package may have already made an updated package available, but the package manager may not know to install that package if the version number isn’t newer. When upgrading, users should verify that the kmod-zfs package is providing suitable kernel modules, reinstalling the kmod-zfs package if necessary.

As a user, this could be more prominent in the docs. As I simply read over this. We just upgrade and forget to validate this.

[macthecadillac]

I suppose changing that block in the documentation from a note to a warning does not hurt. However, I still think it is the user's responsibility to understand the distribution they are using. In particular, that means understanding what Red Hat using semver on a distribution level truly means and how that release model is radically different from that of most other distributions.

While this sort of defeats the purpose of using an EL derivative, one workaround is to use a third party LTS kernel and use DKMS to install ZFS. The CentOS kmods SIG maintains a few LTS kernels for EL in their repositories and they are, as far as I know, always up-to-date. There is also quite a popular set of copr repositories maintained by kwizart that provide up-to-date LTS kernels and they support EL, although from the download numbers it seems the main audience is Fedora.

[landlessreptile]

@macthecadillac it is clear to me that you have a fundamental misunderstanding of what is considered a production quality file system, and it is alarming that you consider that an acceptable answer.

As a reminder, as of this post, the RHEL build has been broken for more than two weeks. Switching to the zfs-testing branch is unacceptable because, first of all, the documentation says so, and second, the 2.2.x and 2.3.x branches have multiple issues that impact RHEL users. While fixes are in the works, and a promising compromise for RHEL users is being discussed, it should be obvious that this is completely unacceptable for production environments, and the documentation should reflect this clear change of state.

Even once zfs is able to build on EL kernels again, what is stopping this from happening again?

If you are unhappy with the wording of my warning, kindly suggest an edit. If you truly believe that "it is the user's responsibility to understand .. [RHEL's] release model is radically different from that of most other distributions", and that inherently causes problems, then that at the very least should be documented. Not documenting breaking issues, especially for something as critical as a file system, and especially for one as heavily marketed for its reliability, is quite frankly, reckless.

Or perhaps I could go with @namazso's suggestion of removing the entire RHEL section all together, as it is clear the project is struggling to support the distro, and future RHEL users should know to either use a different distribution (Debian as you kindly suggested), or avoid ZFS all together.

As a long time ZFS user, I sincerely hope you can prove me wrong.

[macthecadillac]

@landlessreptile From the docs:

When updating to a new EL minor release the existing kmod packages may not work due to upstream kABI changes in the kernel.

Perhaps RTFM?

As a reminder, as of this post, the RHEL build has been broken for more than two weeks...it should be obvious that this is completely unacceptable for production environments, and the documentation should reflect this clear change of state.

So? Stay on the previous minor release and wait for the third party kernel modules to catch up with the new minor release before upgrading. This is what EUS is for, and that's precisely the point of a Red Hat subscription. The 2.1.x branch available in the repo isn't broken on RHEL 9.5 after all, and RHEL 9.5 is still getting security updates for another 24 months or so. If you use third party software in a production environment and you don't read the docs thoroughly and you don't check compatibility of said software before upgrading your system from one release to the next then you're an idiot.

Even once zfs is able to build on EL kernels again, what is stopping this from happening again?

Oh it certainly will. I am well aware that OpenZFS lags behind Red Hat's release cadence but it's not like their documentation didn't mention it in the relevant section.