✨ Add kubemark provider #3847

benmoss · 2020-10-22T14:03:52Z

What this PR does / why we need it:
Adds the kubemark provider to test/infrastructure to live alongside its BFF, CAPD.

Right now it lives in https://github.com/benmoss/cluster-api-provider-kubemark which is less than ideal.

Alternately we can move it to a new kubernetes-sigs repo of its own, but figured I would propose this first since we already have CAPD as precedent for these development-minded providers living in this repo.

I think the release Makefile bits will need some twiddling, it's mostly based on CAPA's Makefile but probably should converge more with CAPD's.

k8s-ci-robot · 2020-10-22T14:04:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ncdc after the PR has been reviewed.
You can assign the PR to them by writing /assign @ncdc in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

test/infrastructure/kubemark/Dockerfile

test/infrastructure/kubemark/PROJECT

test/infrastructure/kubemark/api/v1alpha3/condition_consts.go

detiber · 2020-10-22T19:05:03Z

test/infrastructure/kubemark/controllers/kubemarkmachine_controller.go

+	mgr, err := certificate.NewManager(&certificate.Config{
+		BootstrapCertificatePEM: cfg.AuthInfos[TokenUser].ClientCertificateData,
+		BootstrapKeyPEM:         cfg.AuthInfos[TokenUser].ClientKeyData,
+		CertificateStore:        certificateStore,
+		Template: &x509.CertificateRequest{
+			Subject: pkix.Name{
+				CommonName:   fmt.Sprintf("system:node:%s", kubemarkMachine.Name),
+				Organization: []string{"system:nodes"},
+			},
+		},
+		Usages: []certificates.KeyUsage{
+			certificates.UsageDigitalSignature,
+			certificates.UsageKeyEncipherment,
+			certificates.UsageClientAuth,
+		},
+		ClientsetFn: newClientFn,
+	})
+	if err != nil {
+		logger.Error(err, "error creating cert manager")
+		return ctrl.Result{}, err
+	}
+
+	mgr.Start()


We already have access to the cluster CA (through the secrets), I suspect quite a bit of this could be simplified by using that CA rather than using the bootstrap token to request a certificate to be signed by the cluster CA

That said, it might be good to keep this workflow if we intend to support control plane providers other than Kubeadm.

Yeah, I reverse-engineered this by figuring out how both the kubeadm join and kubelet bootstrap phases converted the token into a full-fledged kubeconfig. I am sure it can be made simpler, and yes some head-scratching might have to go into figuring out how we could support other bootstrap providers, right now it's quite tied to CAPBK.

If that's the case, then I'd say you can likely just rip out a good chunk of this by snagging the CA secret and creating the node cert/kubeconfig directly. We can always add additional support later if need be.

test/infrastructure/kubemark/controllers/kubemarkmachine_controller.go

test/infrastructure/kubemark/controllers/copy_pasta.go

benmoss · 2020-10-22T21:25:07Z

One important note with all this: there's no official kubemark images published right now.

We have two options:

publish our own under some sigs.k8s.io
wait until we can convince sig-scalability / sig-release to publish them under k8s.gcr.io

I've tried to start the conversation (today) with sig-scalability, but so far no movement

neolit123 · 2020-10-22T21:35:13Z

test/infrastructure/kubemark/README.md

+   / //_/_  __/ /_  ___  ____ ___  ____ ______/ /__
+  / ,< / / / / __ \/ _ \/ __ `__ \/ __ `/ ___/ //_/
+ / /| / /_/ / /_/ /  __/ / / / / / /_/ / /  / ,<
+/_/ |_\__,_/_.___/\___/_/ /_/ /_/\__,_/_/  /_/|_|


neolit123 · 2020-10-22T21:39:28Z

SGTM, thanks for the contribution @benmoss !

neolit123 · 2020-10-22T22:15:57Z

One important note with all this: there's no official kubemark images published right now.

We have two options:

publish our own under some sigs.k8s.io

wait until we can convince sig-scalability / sig-release to publish them under k8s.gcr.io

I've tried to start the conversation (today) with sig-scalability, but so far no movement

is this about a test image for kubemark or the CAPI provider for kubemark?

for kubemark itself there is already a gcr.io/k8s-testimages/kubemark.
(but there is also this issue to migrate everything away from k8s-testimages:
kubernetes/k8s.io#1312)

for provider images, my assumption is this is where this should go:
https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images

e.g. there is a:
https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-capi-docker

i don't think we should include kubemark* as part of the root k8s.gcr.io/ release, thus #sig-release redirected to scale.

k8s-ci-robot · 2020-10-23T14:11:42Z

@benmoss: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-cluster-api-verify	`fe84dd1`	link	`/test pull-cluster-api-verify`
pull-cluster-api-test-main	`fe84dd1`	link	`/test pull-cluster-api-test-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

benmoss · 2020-10-23T14:22:50Z

is this about a test image for kubemark or the CAPI provider for kubemark?

for kubemark itself there is already a gcr.io/k8s-testimages/kubemark.
(but there is also this issue to migrate everything away from k8s-testimages:
kubernetes/k8s.io#1312)

Yes, this is about the kubemark image itself, not the CAPK image. I didn't know about the k8s-testimages one. It appears to be broken as far as I know, it has an entrypoint of /kubemark but that file doesn't appear to be runnable. I can't get anything to run from it, everything just returns this error:

standard_init_linux.go:211: exec user process caused "no such file or directory"

I extracted the kubemark from the image tarball and given the presence of glog in the error logs I can assume it was built in 2018 at latest.

for provider images, my assumption is this is where this should go:
https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images

agreed

neolit123 · 2020-10-23T14:54:37Z

Ok, sig scale should know more about the broken image.

detiber · 2020-10-23T15:34:35Z

One important note with all this: there's no official kubemark images published right now.

We have two options:

publish our own under some sigs.k8s.io

Definitely worth pursuing at least as a stopgap/short term solution to unblock this work.

wait until we can convince sig-scalability / sig-release to publish them under k8s.gcr.io

I do think this is definitely the preferred long term solution, especially if it can be standardized using the image promotion process.

I've tried to start the conversation (today) with sig-scalability, but so far no movement

Might be worth seeing if it's something they are interested in having the work for this contributed, in general sig-scalability tends to have more work to do than time.

detiber · 2020-10-23T15:38:37Z

test/infrastructure/kubemark/config/samples/infrastructure_v1alpha3_kubemarkmachine.yaml

@@ -0,0 +1,7 @@
+apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4


The whole samples directory can likely be deleted, I believe we already do that for the other generated content as well.

benmoss · 2020-10-23T19:53:50Z

I think we are going to move this to an outside repository, I sent out an email to sig-cluster-lifecycle for the approval. Thanks for the reviews, this will all make it into the other repo for sure!

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 22, 2020

k8s-ci-robot requested review from justinsb and ncdc October 22, 2020 14:04

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 22, 2020

benmoss force-pushed the kubemark-provider branch 2 times, most recently from 0991602 to e7b5a21 Compare October 22, 2020 14:29

detiber reviewed Oct 22, 2020

View reviewed changes

Add kubemark provider

a18e95a

benmoss force-pushed the kubemark-provider branch from e7b5a21 to a7a76f8 Compare October 22, 2020 21:29

neolit123 reviewed Oct 22, 2020

View reviewed changes

Address feedback

fe84dd1

benmoss force-pushed the kubemark-provider branch from 4f1ed6d to fe84dd1 Compare October 23, 2020 14:06

detiber reviewed Oct 23, 2020

View reviewed changes

benmoss closed this Oct 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add kubemark provider #3847

✨ Add kubemark provider #3847

benmoss commented Oct 22, 2020

k8s-ci-robot commented Oct 22, 2020

detiber Oct 22, 2020

benmoss Oct 22, 2020 •

edited

Loading

detiber Oct 22, 2020

benmoss commented Oct 22, 2020 •

edited

Loading

neolit123 Oct 22, 2020

neolit123 commented Oct 22, 2020

neolit123 commented Oct 22, 2020 •

edited

Loading

k8s-ci-robot commented Oct 23, 2020

benmoss commented Oct 23, 2020 •

edited

Loading

neolit123 commented Oct 23, 2020 via email

detiber commented Oct 23, 2020

detiber Oct 23, 2020

benmoss commented Oct 23, 2020

		@@ -0,0 +1,7 @@
		apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4

✨ Add kubemark provider #3847

✨ Add kubemark provider #3847

Conversation

benmoss commented Oct 22, 2020

k8s-ci-robot commented Oct 22, 2020

detiber Oct 22, 2020

Choose a reason for hiding this comment

benmoss Oct 22, 2020 • edited Loading

Choose a reason for hiding this comment

detiber Oct 22, 2020

Choose a reason for hiding this comment

benmoss commented Oct 22, 2020 • edited Loading

neolit123 Oct 22, 2020

Choose a reason for hiding this comment

neolit123 commented Oct 22, 2020

neolit123 commented Oct 22, 2020 • edited Loading

k8s-ci-robot commented Oct 23, 2020

benmoss commented Oct 23, 2020 • edited Loading

neolit123 commented Oct 23, 2020 via email

detiber commented Oct 23, 2020

detiber Oct 23, 2020

Choose a reason for hiding this comment

benmoss commented Oct 23, 2020

benmoss Oct 22, 2020 •

edited

Loading

benmoss commented Oct 22, 2020 •

edited

Loading

neolit123 commented Oct 22, 2020 •

edited

Loading

benmoss commented Oct 23, 2020 •

edited

Loading