-
Notifications
You must be signed in to change notification settings - Fork 14
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Integrates Karpenter and adds an infrastructure example for AWS users. --------- Co-authored-by: lpm0073 <[email protected]> Co-authored-by: Gábor Boros <[email protected]>
- Loading branch information
1 parent
71f517f
commit 3d3bc53
Showing
32 changed files
with
1,892 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15 changes: 15 additions & 0 deletions
15
charts/harmony-chart/templates/karpenter/node-template.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
{{- if .Values.karpenter.enabled -}} | ||
apiVersion: karpenter.k8s.aws/v1alpha1 | ||
kind: AWSNodeTemplate | ||
metadata: | ||
name: {{ .Values.karpenter.nodeTemplate.name }} | ||
annotations: | ||
"helm.sh/hook": post-install,post-upgrade | ||
spec: | ||
subnetSelector: | ||
karpenter.sh/discovery: {{ .Values.karpenter.settings.aws.clusterName }} | ||
securityGroupSelector: | ||
karpenter.sh/discovery: {{ .Values.karpenter.settings.aws.clusterName }} | ||
tags: | ||
karpenter.sh/discovery: {{ .Values.karpenter.settings.aws.clusterName }} | ||
{{- end }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
{{- if .Values.karpenter.enabled -}} | ||
apiVersion: karpenter.sh/v1alpha5 | ||
kind: Provisioner | ||
metadata: | ||
name: {{ .Values.karpenter.provisioner.name }} | ||
annotations: | ||
"helm.sh/hook": post-install,post-upgrade | ||
spec: | ||
{{- if .Values.karpenter.provisioner.spec.requirements }} | ||
requirements: {{ toYaml .Values.karpenter.provisioner.spec.requirements | nindent 4 }} | ||
{{- end }} | ||
{{- if .Values.karpenter.provisioner.spec.limits.resources }} | ||
limits: | ||
resources: | ||
{{- range $key, $value := .Values.karpenter.provisioner.spec.limits.resources }} | ||
{{ $key }}: {{ $value | quote }} | ||
{{- end }} | ||
{{- end }} | ||
providerRef: | ||
name: {{ .Values.karpenter.nodeTemplate.name }} | ||
ttlSecondsUntilExpired: {{ .Values.karpenter.provisioner.spec.ttlSecondsUntilExpired }} | ||
ttlSecondsAfterEmpty: {{ .Values.karpenter.provisioner.spec.ttlSecondsAfterEmpty }} | ||
{{- end }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
# Reference Architecture for AWS | ||
|
||
This module includes Terraform modules to create AWS reference resources that are preconfigured to support Open edX as well as [Karpenter](https://karpenter.sh/) for management of [AWS EC2 spot-priced](https://aws.amazon.com/ec2/spot/) compute nodes and enhanced pod bin packing. | ||
|
||
## Virtual Private Cloud (VPC) | ||
|
||
There are no explicit requirements for Karpenter within this VPC defintion. However, there *are* several requirements for EKS which might vary from the VPC module defaults now or in the future. These include: | ||
|
||
- defined sets of subnets for both private and public networks | ||
- a NAT gateway | ||
- enabling DNS host names | ||
- custom resource tags for public and private subnets | ||
- explicit assignments of AWS region and availability zones | ||
|
||
See additional details here: [AWS VPC README](./vpc/README.rst) | ||
|
||
## Elastic Kubernetes Service (EKS) | ||
|
||
AWS EKS has grown more complex over time. This reference implementation is preconfigured as necessary to ensure that a.) you and others on your team can access the Kubernetes cluster both from the AWS Console as well as from kubectl, b.) it will work for an Open edX deployment, and c.) it will work with Karpenter. With these goals in mind, please note the following configuration details: | ||
|
||
- requirements detailed in the VPC section above are explicitly passed in to this module as inputs | ||
- cluster endpoints for private and public access are enabled | ||
- IAM Roles for Service Accounts (IRSA) is enabled | ||
- Key Management Service (KMS) is enabled, encrypting all Kubernetes Secrets | ||
- cluster access via aws-auth/configMap is enabled | ||
- a karpenter.sh/discovery resource tag is added to the EKS instance | ||
- various AWS EKS add-ons that are required by Open edX and/or Karpenter and/or its supporting systems (metrics-server, vpa) are included | ||
- additional cluster node security configuration is added to allow node-to-node and pod-to-pod communication using internal DNS resolution | ||
- a managed node group is added containing custom labels, IAM roles, and resource tags; all of which are required by Karpenter | ||
- adds additional resources required by AWS EBS CSI Driver add-on, itself required by EKS since 1.22 | ||
- additional EC2 security groups are added to enable pod shell access from kubectl | ||
|
||
See additional details here: [AWS EKS README](./k8s-cluster/README.rst) |
Oops, something went wrong.