Kubernetes v1.24+ container runtime on Fedora CoreOS #767
Comments
|
👀 cri-o doesn't list Fedora CoreOS as a supported OS and the only mention of how to hack it in is a year ago from @dustymabe (thx). And then cri-o creates a new need for https://github.com/cri-o/cri-o/blob/master/install.md |
This is the main reason why cri-o is not shipped as part of the base FCOS. The daemon isn't a generic container runtime that can be freely upgraded by the OS, but it is instead interlocked with the higher level application/service (k8s control plane). This strict interlock is an explicit design, highlighted at https://github.com/cri-o/cri-o. To that extent, for k8s distributions the whole "kubelet plus cri-o" is effectively a single "node runtime" component. |
|
@mrunalp @haircommander PTAL |
|
@mrunalp @haircommander Why isn't Fedora a supported platform for CRI-O? Is this just an oversite? |
|
fedora is listed https://github.com/cri-o/cri-o/blob/master/install.md#fedora-31-or-later |
I agree with @lucab here, kubelet and cri-o should be installed together with matching versions. If there's a way we (the cri-o team) can minimize the friction with this in FCOS, I'd love to hear about it. |
|
As I see it, the issue here is more that it's a bit difficult to install the cri-o RPM because it's distributed as a module and rpm-ostree can't handle it. |
|
how is the kubelet packaged for OKD now? Could the appropriate cri-o version be tossed in that package? |
|
In OKD we grab Similar steps to fetch CRI-O RPM |
|
For comparison, docker+shim (and soon containerd on Flatcar Linux) provide a suitable container runtime that meets Kubernetes CRI minimum needs, out of the box. Kublet is bundled as a container image. Runs. Conformant. We roll forward within days of Kubernetes releases and do need to not wait on changes from the base OS or any package ecosystem. If we move toward |
in practice, not terribly strict. It's the safest bet, though. the cri, while generally stable, does change. We don't backport new cri changes to older versions of cri-o. We also don't attempt to test any kubelet/cri-o skew. Basically: we make no support claims for anything other than matching versions. Generally, folks don't have much trouble with having mismatched versions (I've never heard anyone complain about it). But it's theoretically possible
out of curiosity, do you run the kubelet inside a container, or do you use the image to package it easily? |
|
@vrutkovs @haircommander it looks to me as though we are running the kubelet through the hyperkube binary (see https://github.com/openshift/installer/blob/master/data/data/bootstrap/systemd/units/kubelet.service.template#L15-L16), |
Yes, and we had this option in 3.x days. AFAIK some distros (Rancher) do that too - but node SIG doesn't officially support running kubelet (and container engine) in container, as it complicates volumes. Perhaps we should be extracting binaries to |
|
If we're just sideloading @haircommander Kubelet in a container with podman, like other on-host services (e.g. etcd). With v1.22, container runtime experience might become a differentiator for users picking from base OSes. I want both to be strong choices and don't weigh in on my users choice. |
|
@dghubble coreos/rpm-ostree#1435 would probably be the cleanest solution to this then |
|
I agree @LorbusChris , I think that's the best not-hacky way to guarantee cri-o can be installed (and kubelet can be installed with a corresponding version). we could even couple a kubernetes module with kubelet, cri-o and crictl |
How about Fedora CoreOS shipping |
What reasons are there for shipping a general container runtime? If the modules thing is figured out, and cri-o can be installed with kubernetes seamlessly, I don't see a need for containerd |
|
Some of the original value behind Container Linux being packageless were shipping a minimal OS suitable for cluster uses cases (i.e. container runtime is "new enough"). Adding RPMs is more a step toward traditional approaches and slower cadence. What happens when Kubernetes has a release, but cri-o doesn't have an RPM yet if they're in lock-step? For example, how would we test Kubernetes v1.21-beta.1 right now? We'd need to release without Fedora CoreOS, and I can see users gravitating toward Flatcar/containerd if this just isn't even a factor there. To be clear, I don't care which container runtime is chosen. I have no horse in this race (thank you CRI). Just that it work well in these cases going forward. |
I see this as mostly a syncing of requirements. Up until now, we haven't had a request to test a yet-to-be-released cri-o in FCOS. I see no reason we couldn't ship the module early--assuming we can set expectation about stability before the .0 release. I'm happy to work together to setup cri-o packaging better on FCOS |
|
CRI-O is also being built upstream in non-module RPMs, rpm-ostree can install those - see crio 1.20 |
Yes, this is #681. We definitely need to improve the UX on package layering. (Modularity is something that this sugar will need to consider as well.)
Agreed we need to discuss this (IMO, yes this should be supported). Essentially, I think we need to:
I don't think rpm-ostree natively supporting modules is a blocker for all this, but it definitely would make it easier. |
I think supporting the three releases k8s upstream supports should work
I am happy to go forward with either, though my preference is proper module support (it feels more idiomatic for fedora)
big +1
I suppose if we go the extensions route, we could use the existing OBS infrastructure for this (enable the OBS repo instead of some special cri-o one) |
|
Related: openshift/os#498 |
Fedora CoreOS is meant to be a general platform for running containers. If there's a container runtime that people want to use, and there are no technical blockers to shipping it, we should probably ship it. We already ship Moby and podman, and would already be shipping cri-o if not for the version-skew issue. Is there some reason we shouldn't ship containerd? |
|
By the way there is already a |
|
Hello everyone
Unfortunately, I am no longer active in the RPM Fedora area. The Kubernetes rpms in Fedora has been lacking maintenance for some time. I recall there was an effort to make Kubernetes modular. However, I was not part of the effort. If anyone is interested in proceeding here and keeping the Kubernetes rpms up2date, I can share the access rights. |
|
Commands for |
I have not worked directley with Fedora RPMs but familiar of the process this is something I would be interested in. I'm on matrix if you wanted to speak directley @anthr76:mozilla.org |
|
I have the time, interest, and (some) skills to also contribute where I can add value. Brad |
|
So it sounds like our eventual ideal situation is having modular go, cri-o and kubernetes. We have modular cri-o, so ideally someone could add and maintain go and kubernetes. Do either of those tasks interest either of you @buckaroogeek @anthr76 ? |
|
Sure I, for one, can give it a go. :) Actually, yes I am very interested in helping. Reviewing the package maintainer web pages now. And will follow through on the needed steps. I might also suggest helm as that seems to also be lagging in packaging support. It would seem that getting some solid traction around go and a go module would be a first step? After getting the package maintainers introductions done. |
It looks like the golang maintainers started a discussion about golang and different versions recently on Fedora Devel. It looks like the consensus there was to use golang compat packages (not a module) if they wanted to support different versions in Fedora. Which IIUC should be fine for what we need. I think the status is:
@buckaroogeek @anthr76 - does that sound good? |
|
@dustymabe - Yes, it does for me. |
|
These efforts sound like the right direction to getting |
|
I can help with the non-modular |
|
@buckaroogeek @anthr76 feel free to share you fedora usernames so I can add you as committers to the kubernetes project. Checking the previous commits might give you some hints on how to update the kubernetes to the next version. I have not installed the kubernetes rpms for ages so it might be challenging at the beginning to make it work. Though learning from mistakes will eventually yield the right results. |
|
@ingvagabund - buckaroogeek is the name I use in FAS. Thanks for the tips and assist. |
|
When adding you as a committer, I am getting |
|
@ingvagabund - I do not want to further side track this issue. May I contact you directly? A ticket needs to be filed with the package sponsor pagure instance by a current sponsor. |
My fas is anthr76 I can be reached on matrix as @anthr76:mozilla.org Thanks! |
@buckaroogeek @anthr76 what channel do you prefer? IRC, email, ...? |
I'm on matrix as @anthr76:mozilla.org I'm in #fedora-coreos and #fedora-golang which is also bridged with IRC |
|
Also on matrix as buckaroogeek. Fedora-coresos and fedora-arm mostly
…On Thu, Jan 13, 2022, 05:34 Anthony Rabbito ***@***.***> wrote:
I do not want to further side track this issue. May I contact you directly?
@buckaroogeek <https://github.com/buckaroogeek> @anthr76
<https://github.com/anthr76> what channel do you prefer? IRC, email, ...?
I'm on matrix as @anthr76 <https://github.com/anthr76>:mozilla.org I'm in
#fedora-coreos and #fedora-golang which is also bridged with IRC
—
Reply to this email directly, view it on GitHub
<#767 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKDYHXXXO4OWXZUJD2RHBTUV3IERANCNFSM4ZBN6ONA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@ingvagabund @anthr76 Please file a request for sponsoring @anthr76 through to co-maintain kubernetes here: https://pagure.io/packager-sponsors |
dghubble
changed the title
|
I think this one is going to intersect with https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md for many people. |
This is already in the OS as a dependency of moby-engine, but Typhoon is now also relying on this and likely other people deploying Kubernetes on top of FCOS. Unlike cri-o, the versioning of containerd wrt CRI is less tight, which makes baking it into the host easier. Things may change here in the future, and it's likely we will recommend cri-o as an alternative runtime for k8s eventually. But for now, let's at least be more explicit that we're shipping containerd. For more information, see: coreos/fedora-coreos-tracker#767 poseidon/typhoon#899 (comment)
This is already in the OS as a dependency of moby-engine, but Typhoon is now also relying on this and likely other people deploying Kubernetes on top of FCOS. Unlike cri-o, the versioning of containerd wrt CRI is less tight, which makes baking it into the host easier. Things may change here in the future, and it's likely we will recommend cri-o as an alternative runtime for k8s eventually. But for now, let's at least be more explicit that we're shipping containerd. For more information, see: coreos/fedora-coreos-tracker#767 poseidon/typhoon#899 (comment)
This is already in the OS as a dependency of moby-engine, but Typhoon is now also relying on this and likely other people deploying Kubernetes on top of FCOS. Unlike cri-o, the versioning of containerd wrt CRI is less tight, which makes baking it into the host easier. Things may change here in the future, and it's likely we will recommend cri-o as an alternative runtime for k8s eventually. But for now, let's at least be more explicit that we're shipping containerd. For more information, see: coreos/fedora-coreos-tracker#767 poseidon/typhoon#899 (comment)
|
From @jlebon's memory (which is much better than my memory) in the video community meeting today he think's the remaining steps here are:
|
|
Please let me know how I can help. At one point, Anthony (@anthr76) and I were considering modules for kubernetes, but the approach taken by the nodejs and haskell maintainers on multiple versions seems to be more approachable. I have been negligent in getting a change request submitted to outline the path forward. The version of go supported in a given Fedora release would shape the version(s) of kubernetes available from Fedora repositories. For example, F37 has go 1.19.x. Kubernetes 1.25, 1.24, and 1.23 would be available allowing Fedora, including CoreOS, to function as a node for any supported version of Kubernetes and as a workstation hosting kubectl to manage those clusters. Regardless of the approach, I would like to see (and help produce) good documentation for end users around the installation of vanilla kubernetes on all suitable variants of Fedora. |
This is already in the OS as a dependency of moby-engine, but Typhoon is now also relying on this and likely other people deploying Kubernetes on top of FCOS. Unlike cri-o, the versioning of containerd wrt CRI is less tight, which makes baking it into the host easier. Things may change here in the future, and it's likely we will recommend cri-o as an alternative runtime for k8s eventually. But for now, let's at least be more explicit that we're shipping containerd. For more information, see: coreos/fedora-coreos-tracker#767 poseidon/typhoon#899 (comment)
This is already in the OS as a dependency of moby-engine, but Typhoon is now also relying on this and likely other people deploying Kubernetes on top of FCOS. Unlike cri-o, the versioning of containerd wrt CRI is less tight, which makes baking it into the host easier. Things may change here in the future, and it's likely we will recommend cri-o as an alternative runtime for k8s eventually. But for now, let's at least be more explicit that we're shipping containerd. For more information, see: coreos/fedora-coreos-tracker#767 poseidon/typhoon#899 (comment)
Kubernetes intends to drop drop support for docker-shim as a container runtime in v1.22. Currently, Fedora CoreOS
33.20210217.3.0ships docker19.03.13. docker-shim remains the most stable, tested, available out-of-the-box runtime, but this will end soon. I'd like some kind of clarity on Fedora CoreOS's intentions, such as shipping a compatiblecontainerdorcri-o. With Kubernetes cutting v1.22 alpha releases (likely pretty soon) in the time frame of Fedora 34, as Kubernetes distros we'll want to start evaluating and conformance testing the selection.Overall, I'd like to know there is some plan. Ideally a documented one. Flatcar Linux has published their intentions to ship containerd in time and already has a mechanism to test it (docs).
cri-o
It sounds like
cri-ocan only be installed by downloading an RPM (from where?) directly and rpm-ostree installing it (unverified). dnf and yumdownloader used by an openshift script aren't present. #292 (comment). I have some maturity concerns about that. Is this really the recommended path? For a container optimized distro to not have a better path to getting a container runtime?Releases are also pinned to Kubernetes versions and seem to lag by quite some time, so I have some velocity concerns (the runtime needs to be fairly stable, we roll forward when Kubernetes does, think hours).
containerd
I haven't seen Fedora CoreOS plans to make this the runtime of choice.
Or is the plan something else?
The text was updated successfully, but these errors were encountered: