Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Kubernetes Cluster controller and add support for meta.externalName #43

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

vladfr
Copy link

@vladfr vladfr commented Feb 17, 2023

I ran into this problem that cluster updates are never performed. While trying to debug I actually made good headway.

The problem is that the Kubernetes Civo API changed quite a bit, and now only works via Cluster ID (which makes a ton of sense). The current controller for clusters has two problems:

  • does not use the api properly
  • cannot remember the Civo ID of the cluster, so it never finds the clusters it creates

Changes:

  • it's now using the proper civo API client
  • the cluster k8s object now remembers the Civo ID via meta
  • the controller uses the Civo ID in every API operation
  • Updates work too!

Fixes #41

I have:

  • Read and followed Crossplane's contribution process.
  • Run make reviewable test to ensure this PR is ready for review.

Tests

  • Added tests for create and update.

Open issues

  • currently, the API doesn't allow updating the firewall ID. The code handles this update, but the underlying resource doesn't actually get updated.
  • applications aren't updated or removed.

NOTE: I've bumped /build to latest, since I couldn't run make generate anymore.

@vladfr vladfr changed the title Added civoclient and fixed cluster update via meta.externalname Added civoclient and fixed cluster update via meta.externalName Feb 20, 2023
@vladfr vladfr marked this pull request as ready for review March 21, 2023 13:34
@vladfr
Copy link
Author

vladfr commented Mar 21, 2023

Applications are being installed on Create, but not added on Update.

App name is civo-cluster-autoscaler. If it's specified when Creating, it works and the app appears on the CIVO Dashboard. If you then do an Update, this is the API error:

{"level":"warning","msg":"update error:Error: Unknown error response - status: 404 Not Found, code: 404, reason: {\"code\":\"database_kubernetes_aplication_not_found\",\"reason\":\"Failed to find the Kubernetes application for installation\"}\n ","time":"2023-03-21T16:56:28+02:00"}

@vladfr
Copy link
Author

vladfr commented Mar 21, 2023

I'd like to leave a note here. The Observe() cycle should not trigger any modifications. It should return an ExternalObservation, and the reconciler will call Update when needed.

Previously the code would call Update and trigger resource updates. This made it so that the reconciler never issued proper log messages. With this change, you can clearly see when the reconciler requests a Create or Update; or whenever it sees an error.

@@ -43,11 +43,16 @@ type CivoKubernetesConnectionDetails struct {
type CivoKubernetesSpec struct {
xpv1.ResourceSpec `json:",inline"`
Name string `json:"name"`
Region string `json:"region,omitempty"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The destination region is set in the ProviderConfig, so we're not setting in the spec of the resource, so I would keep as it is

// +required
// +immutable
// NOTE: This can only be set at creation time. Changing this value after creation will not move the cluster into another network..
NetworkID *string `json:"networkId"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting addition, I would set is as optional to be backward compatible and I would add the optional firewallID and the optional firewallRule because civo API use these. If they're absent (as is situation), we still use the default ones.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, the network is actually required in the API, but the Terraform module has this field as optional. This is because when the network is missing, it queries the Civo API to find out the default network ID by doing client.GetDefaultNetwork().

I'll add the same thing here, and it should work to get the network back to optional, and make it consistent with Terraform.

pools:
- id: "8382e422-dcdd-461f-afb4-2ab67f171c3e"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's very uneasy to generate UUID and pass to the manifest, but it doesn't mean we can sent friendly names like these, because they must be unique in our database. I would left like it is. In the API we can auto generate those, but then it means we cannot update through crossplane if we don't provide a reference at all.

Comment on lines -87 to 89
civoClient, err := civocli.NewCivoClient(string(s.Data["credentials"]), providerConfig.Spec.Region)
civoClient, err := civogo.NewClient(string(s.Data["credentials"]), providerConfig.Spec.Region)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why removing the civocli wrapper object?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

civocli looks to be an old package committed to this repo. In the meantime, I see that the new API is implemented in the civogo client. I think this is the right way to go, perhaps the API team can confirm?

Comment on lines -106 to +107
return managed.ExternalObservation{ResourceExists: false}, err
return managed.ExternalObservation{}, nil
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if err != nil why removing the error return and return nil error?

Comment on lines -173 to +182
civoCluster, err := e.civoClient.GetK3sCluster(cr.Spec.Name)

// at the first call, this id will be the cluster name; civo should return 404
existingClusterID := meta.GetExternalName(cr.GetObjectMeta())
civoCluster, err := e.civoClient.GetKubernetesCluster(existingClusterID)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: not sure what meta.GetExternalName would return for a crossplane resource: it should fetch the annotation crossplane.io/external-name but this doesn't correspond to the clusterID as it's requested by the GetKubernetesCluster method (that now, as I see, it's never used).

Comment on lines +194 to +199

clusterRegion := cr.Spec.Region
if clusterRegion == "" {
clusterRegion = e.civoClient.Region
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't use region in the spec, it's implicit in the provider configs

Comment on lines -181 to +221
err = e.civoClient.CreateNewK3sCluster(cr.Spec.Name, cr.Spec.Pools, cr.Spec.Applications, cr.Spec.CNIPlugin, cr.Spec.Version)
kc := &civogo.KubernetesClusterConfig{
Name: cr.Spec.Name,
Region: clusterRegion,
NetworkID: *cr.Spec.NetworkID,
Pools: cr.Spec.Pools,
Applications: strings.Join(cr.Spec.Applications, ","),
Tags: strings.Join(cr.Spec.Tags, " "),
}

if cr.Spec.CNIPlugin != nil {
kc.CNIPlugin = *cr.Spec.CNIPlugin
}
if cr.Spec.Version != nil {
kc.KubernetesVersion = *cr.Spec.Version
}

if cr.Spec.FirewallID != nil {
kc.InstanceFirewall = *cr.Spec.FirewallID
}

newCluster, err := e.civoClient.NewKubernetesClusters(kc)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would readd the wrapper function and move the creation of the createCluster request in the originary class. Besides that, if you added optional networkID, firewallID, firewallRule I would pass those to the request OR, in their absence, I would pass the default network ID as it is now, without removing that logic.

Comment on lines +227 to +228
meta.SetExternalName(cr, newCluster.ID)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is interesting, effectively then you can use the function you proposed above to directly fetch the cluster by ID :claps

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, external-name is Crossplane magic to allow mapping Cloud IDs to Kubernetes IDs. During Create, a controller is supposed to fetch it and write it here.

Civo API uses cloud-generated IDs for resource identification, so this maps to using external-name in Crossplane. It looks like the Civo API always generates IDs, so this should be the main way of doing things. Because you can reference this ID directly to the API in most cases, and you would get the resource back. This is exactly how Upjet providers implement this functionality, so the main AWS/GCP/Azure Crossplane providers all store the Cloud IDs via external-name.

Your comment above about pools IDs is interesting, I need to look more into that. Right now, I don't use the pools api to directly update pools, since there is no pools CRD. So this works for pool updates, just because it uses a cluster-scoped Civo API.

if err := e.civoClient.UpdateK3sCluster(desiredCivoCluster, remoteCivoCluster, providerConfig); err != nil {
desiredClusterConfig := &civogo.KubernetesClusterConfig{
Pools: desiredCivoCluster.Spec.Pools,
Region: desiredCivoCluster.Spec.Region,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RealHarshThakur thoughts if we should introduce Region in the crossplane resource's spec? Now we don't have those.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Region is required by the API, so this is why it needs to be passed here. I think the CRD should have it too, it would be good to follow the API as close as possible. And right now, it still picks up the default Region from the ProviderConfig.

Another argument in favour of the region is that if you want to deploy in multiple regions, you can use the same ProviderConfig.

@vladfr vladfr changed the title Added civoclient and fixed cluster update via meta.externalName Update Kubernetes Cluster controller and add support for meta.externalName Apr 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't create new pool in civo with crossplane
2 participants