Enable different kubernetes distribution support

[gyliu513]

This is an issue migrate from kubernetes-retired/multi-tenancy#1479

The virtualcluster only support vanila Kubernetes now, but the real use case is that many customers are using some different Kubernetes distributions, it would be great to enable the virtualcluster can support OpenShift and other Kubernetes clusters which has passed the conformance test at https://github.com/cncf/k8s-conformance/tree/master/v1.20

What is the plan to enable multiple kubernetes distribution support? Anyone is working for this?

@Fei-Guo @christopherhein

[christopherhein]

@gyliu513 There isn't anyone looking at this, this issue COULD conflict with #44 if we end up in a KCP route it would be harder for you to implement this, that being said if you wanted to experiment with this idea you could like @Fei-Guo has said in that other issue use the NestedAPIServer, NestedControllerManager, NestedEtcd` to templatize and mount your manifests in the container where it expects them. https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/controlplane/nested/controllers/consts.go#L23-L27

Basically, the flow would be copy the files in - https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/controlplane/nested/component-templates change the statefulset specs to deploy whatever distro you'd like to use, and then you'd mount those new go templatized manifests into your image and it should work. The area this breaks down is the higher-level objects that that distro need to be deployed that aren't apiserver, controller manager or etcd, which you'd have try using the ClusterResourceSet which we have an issue about testing #41

[christopherhein]

/kind support

[gyliu513]

Awesome, thanks @christopherhein for the detailed explanation. So how about enabling those parameters as dynamic https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/controlplane/nested/controllers/consts.go#L23-L27 , so that we the customer can decide which k8s distribution they want to use?

defaultEtcdStatefulSetURL = "/nested-etcd/{k8s distribution}/nested-etcd-statefulset-template.yaml"	
defaultEtcdServiceURL     = "/nested-etcd/{k8s distribution}/nested-etcd-service-template.yaml"	
defaultKASStatefulSetURL  = "/nested-apiserver/{k8s distribution}/nested-apiserver-statefulset-template.yaml"	
defaultKASServiceURL      = "/nested-apiserver/{k8s distribution}/nested-apiserver-service-template.yaml"	
defaultKCMStatefulSetURL  = "/nested-controllermanager/{k8s distribution}/nested-controllermanager-statefulset-template.yaml"

[christopherhein]

IMO maintaining multiple distributions is out of the scope for what CAPN should support. We should be enabling you to overwrite and implement your own but not providing those from CAPN. This will lead to unnecessary churn and releases of this codebase to vendor those spec.

@Fei-Guo @charleszheng44 wdyt?

[gyliu513]

@christopherhein +1, let me fork this repo and maintain by ourselves first, and we can check how to enable the upstream support in future.

[charleszheng44]

@gyliu513 Yes, I agree with @christopherhein. CAPN is still in the early stage, it can be hard to support multiple distros before the codebase becomes stable. But feel free to try out the OpenShift with CAPN and we would like to discuss any issues you run into.

[gyliu513]

Close it for now, we can re-visit this issue if any new requirement. Thanks @christopherhein and @charleszheng44 !

[christopherhein]

Thank you, glad we're all on the same page. Also 🙏 if you fork and make changes that aren't distro-specific we'd love to the help and will gladly accept patches!

[gyliu513]

Sure thing, thanks @christopherhein

[gyliu513]

@wangjsty ^^