support building node images from release tarballs

[BenTheElder]

We should implement kind build node-image --type=release-tar /path/to/release.tar and use this for CI testing to dedupe building the binaries etc.

/kind feature
/priority important-soon
/assign

[BenTheElder]

punting to 0.4, but just because 0.3 was loaded down with breaking image change improvements and we need to get those out the door. this is an important feature.

[BenTheElder]

punting to 0.5, 0.4 is focused primarily on networking enhancements / fixes, general bugs / cleanup / minor features, and initial IPv6 support.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[BenTheElder]

xref: #197
I've been sketching up a revamp of the entire build, intend to ship in v0.9.0

[dims]

cc @jiatongw

[jiatongw]

@BenTheElder What is the target date of v0.9.0?

[BenTheElder]

Sorry I missed this message. After some thought v0.9.0 was moved to match Kubernetes 1.19 (not everything we wanted is ready, but it's getting long in the tooth since 0.8.1), which is scheduled for tomorrow.

I'm sweeping through issues and will try to finish up patches for most things, but this one will probably slip to early in the next version.

Some refactors related to this are done, but it's not complete, and there are a couple bugs that should get fixed first.

[rosti]

/remove-lifecycle stale

[BenTheElder]

@BenTheElder should kind fetch files directly from the release tar? Isn't it more flexible WRT workflow and possible future changes (e.g. somebody decides to use xz instead of gzip, or the paths in the tar file change)?

I think most users are not going to want to even deal with wget and downloading isn't the challenging part.

They'll just want to specify a release or CI version, OR specify their source URL | directory.

Isn't it more flexible WRT workflow and possible future changes (e.g. somebody decides to use xz instead of gzip, or the paths in the tar file change)?

The k8s release may not do this without warning, there are other consumers and the release format is "an API".
I will personally raise another stink if SIG release goes to break this again without following a KEP etc. (think the hash file debacle).

---

What I've actually wanted to do here is introduce a flexible spec format for control over what goes into a node image.
Unfortunately no features are shipping right now (including feature review) because all our time is going to bug hunting or non-kind related work, otherwise we'd have cut a release already :( (e.g. #1969 is alarming and needs fixing).

[BenTheElder]

aside: as a more immediate measure, I'm sure you could get a lot of kudos from people with arm macs in the coming weeks if you publish an image like this to dockerhub or similar :+))

we're obviously going to need a more permanent solution, but as it stands we've grown a backlog reviewing features due to the bug / support load 😞

I hope to improve that soon, I've just sunk some time into https://kind.sigs.k8s.io/, particularly the contributor / developer docs, and I hope to announce some improvements to how the project is managed soon 😅

[rosti]

I uploaded 1.18.13, 1.19.5, and 1.20.0 images here.

P.S. Docker on M1 Macs seems to be a long way off, plus I think that it wouldn't be that big deal to recompile Kubernetes from source there as it is for Raspberry Pi 4 (hence this effort).

[BenTheElder]

We got pretty far with working cross-builds, upstream is also taking longer than expected to make dockerless + providerless a default reality, and we won't be able to use pre-built tarballs for our releases until then (too much binary bloat added).

Without that, this has slipped. It's still a good idea. I would be happy to see more discussion outlining the UX.

What I have shipped is dropping --type (no more bazel anyhow) and deprecating --kube-root in favor of kind create cluster <kube-root> in preparation for:

Tentatively I think we should make kind build node-image [source-path] with usage like:

1. kind build node-image / unset
   • default to attempting to auto-detect your local kubernetes source checkout (as today, and the primary use case for kind build node-image / kind currently per project scope doc)
2. kind build node-image /path/to/k8s.io/kubernetes / kubernetes source-checkout path
   • absolute and relative paths that are directories => build from source (1. but user supplied path, not auto-detected)
3. kind build node-image /path/to/release-archive.tar / kubernetes release tarball path
   • if the path is a file instead of a directory, check for a release tarball and build from the binaries/images in it
   • possibly instead this needs to be a locally downloaded directory matching the upstream release layout, we can do a form of 2. instead and sniff the path's contents for these directories vs say some well known file like OWNERS in k8s sources
4. kind build node-image ci/latest / kubernetes versions (from upstream builds).
   • download the named version from upstream builds and build from that.
   • should this have caching? how is the cache invalidated? (probably we can add this later)
5. kind build node-image https://dl.k8s.io/foo/bar / kubernetes pre-built versions in explicit download bucket path (to be more specific than named versions hosted there)
   • same as 4) but with user supplied base URL / GCS bucket instead of well-known version => identify download location, for custom user uploaded builds or being more explicit.

[aojea]

on hindsight, I don't think kind users will care about the details and we should hide urls or tarballs, just kind build node-image <kubernetes-tag>

[BenTheElder]

on hindsight, I don't think kind users will care about the details and we should hide urls or tarballs, just kind build node-image

two reasons they will off the top of my head:

1. some people don't have access to GCS (china) / upstream release binary hosting
2. some people will want to re-use their own k8s builds (think EKS distro)

EDIT: this one is also very easy to support, it's just 4) but skip the indirection of named version <-> download base path

[aojea]

• some people don't have access to GCS (china) / upstream release binary hosting
• some people will want to re-use their own k8s builds (think EKS distro)

those are implementation details needed for 4, we can always expose them

[BenTheElder]

Yes, exposing skipping the indirection is 5)

[aojea]

right, my rationale is that power users are able to build images and use them directly using some registry.
My feeling is that most requests we receive in slack are from users that just want to try the latest versions without having to build them.
My suggestion is to implement just 4 for them, and iterate based on that, if there is more demand we can always expose the internals, however, if we start exposing the internals is hard to remove them ... and we are going to expose a much larger area with all the supportability, time, ... consequences

[BenTheElder]

fixed in #3614