Jhony/mcdaniel integrating Karpenter

[jfavellar90]

This is a duplicate of #24 with a couple of changes on top to adjust:

• Code rebase
• commit-lint errors
• Changes to the new way of managing the helm chart.

I copy the original PR description as a reference:

For #7: refactor the new infra-example folder into infra-examples, and add Terraform modules for AWS reference resources.
Adds generic AWS Virtual Private Cloud and Elastic Kubernetes Service modules, both of which are preconfigured as necessary to support use of Karpenter for node auto-scaling and pod bin packing.

[openedx-webhooks]

Thanks for the pull request, @jfavellar90! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[itsjeyd]

@lpm0073 @jfavellar90 Thank you for this contribution :)

[gabor-boros]

@jfavellar90 I still need to create a cluster with Karpenter and deploy the helm chart, but looks good to me! Nice work @lpm0073 @jfavellar90! 🎉

[cmltaWt0]

@jfavellar90 @lpm0073 great job!

I've started testing it.

Issues I've found so far:

1. Hashicorp/Template v2.2.0 is deprecated (archived) and has no distribution for M1:

template v2.2.0 does not have a package available for your current platform, darwin_arm64

Workaround for M1 users is following (tested):

brew install kreuzwerker/taps/m1-terraform-provider-helper
m1-terraform-provider-helper activate
m1-terraform-provider-helper install hashicorp/template -v v2.2.0

2. k8s-cluster terraforming doesn't work for me from the first try (repeated twice to confirm):

│ Error: waiting for Security Group (sg-021189bbba3cc2252) Rule (sgrule-2633141658) create: couldn't find resource
│
│   with module.eks.aws_security_group_rule.node["port_8443"],
│   on .terraform/modules/eks/node_groups.tf line 207, in resource "aws_security_group_rule" "node":
│  207: resource "aws_security_group_rule" "node" {
│
╵

I've repeated terraform apply in aws/k8s-cluster folder since the SG was actually created in AWS.
Second try completed successfully.

Eventually I have a working Cluster, I can update my kube config and connect to it 🎉

---

Next, I'll try to deploy our Chart and couple of openedx installations.
Also we need to rebase the PR on the latest main and re-test.

[lpm0073]

confirming that i ran into both issues for template last week on an unrelated project. comments:

• the M1 incompatibility seems to be permanent. contributors didn't get around to addressing the apple silicon problem until after the M2 had been released. it seems that the Terraform community has moved on and doesn't have plans to address the problem for M1
• the deprecation can be resolved with the following version bump

`terraform {
required_version = "~> 1.3"

required_providers {
template = {
source = "hashicorp/template"
version = "~> 2.2"
}
}
}`

[gabor-boros]

I just found out that I didn't post my pending review comments for the past two weeks. 🤦 GitHub UI tricked me. Sorry about that. Posting now.

@cmltaWt0 and @lpm0073 thank you for helping out in functional reviewing!

[gabor-boros]

Wow. These seems to be really high limits. I don't mind, but it may not make sense. I did not see a 100 node cluster for a while.

[jfavellar90]

I reduced the number of nodes to 50, is it a better number?

[itsjeyd]

Hey @jfavellar90, a friendly reminder to follow up on @gabor-boros's comments above.

[jfavellar90]

@gabor-boros I think this is ready for a second review. I'll be adding documentation to the main README file

[gabor-boros]

@jfavellar90 Should I wait for the readme change or shall I proceed? 🚀

[itsjeyd]

Hey @jfavellar90, a friendly reminder to get back to @gabor-boros's question above.

[jfavellar90]

@gabor-boros sorry for the late answer, I already added the docs to install Karpenter in the example infrastructure. However, there are changes we need to apply to the instructions in order to test the PR. In step 4 the Helm chart is installed in the cluster. Since the new version of the chart including Karpenter has not been released, step 4 must be replaced by the following tasks to install the chart from source:

• Update the dependencies:

# From the repo root path
cd charts/harmony-chart
helm dependency update

• Install the helm chart from source:

# From the repo root path
helm install --namespace harmony --create-namespace -f values-example.yaml harmony ./charts/harmony-chart --wait

Once the Helm chart is released, the instructions indicated in step 4 will work as expected.

[gabor-boros]

Ah! Thank you @jfavellar90 I'll review this after the meeting we have now. 😇

[gabor-boros]

FTR: The PR is reviewed, I just want to check one more thing before approving.

[gabor-boros]

Hey @jfavellar90,

I just approved this PR, though I was not able to setup the kubernetes cluster. The hashicorp/template provider is deprecated and we are still depending on it (not just directly, but probably through dependencies as well?). Hence it cannot be installed on my system.

That should be a task of a different PR to fix that as the scope of the current PR is already huge. This is a good point to raise during the meeting today.

I read through the changes many times, so I can confidently approve this.

[itsjeyd]

@gabor-boros Based on your review above it sounds like this PR is ready for merge. Or are there any blockers that are keeping you from hitting the button? :)

[gabor-boros]

@jfavellar90 the latest commit fixed the issue, though there is some other issues during EKS install with CBS and other things. That needs a separate issue to fix and waaaaay out of scope for this PR, so I'm merging this one now.

[openedx-webhooks]

@jfavellar90 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.