Skip to content

Commit

Permalink
docs: Major updates to the README
Browse files Browse the repository at this point in the history
  • Loading branch information
@bradenmacdonald
bradenmacdonald committed Apr 12, 2023
1 parent 3552df6 commit 5df8d52
Showing 1 changed file with 63 additions and 40 deletions.
103 changes: 63 additions & 40 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,48 +1,70 @@
# Harmony Chart
# Harmony Project - Open edX on Kubernetes

This repository contains a Helm Chart and [Tutor](https://docs.tutor.overhang.io/) plugin that you can use to install
the necessary components onto a Kubernetes cluster so that the cluster can be used to host multiple instances of Open
edX, where each instance is provisioned and managed by Tutor.
This project is focused on making it easy to set up a standardized, scalable, secure Kubernetes environment that can host **multiple instances** of [Open edX](https://www.openedx.org). See [Motivation](#motivation) below.

## Central Load Balancing
Specifically, this repository contains:
* A Helm Chart that can install necessary shared resources into your cluster (a load balancer / ingress controller, autoscaling infrastructure, monitoring tools, databases, etc.)
* A [Tutor](https://docs.tutor.overhang.io/) plugin that configures Tutor to build images that will use the shared resources deployed by the Helm chart.

The problem: Tutor uses Caddy, which requires a central configuration file listing all the hostnames to route HTTP(S)
traffic to. Deploying a separate Caddy load balancer per Open edX instance also deploys a corresponding network load
balancers at the Cloud Provider, which can become expensive and are an inefficient use of resources.
This repository does not set up the Kubernetes cluster or provision individual Open edX instances.

Instead, we want to use this Helm chart to deploy a single load balancer onto the cluster along with other shared
resources. The load balancer should **auto-detect** any Open edX instances that get deployed onto the cluster, without
a central configuration file.
See [technology stack and architecture](#technology-stack-and-architecture) below for more details.

Currently, this initial version uses [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) as a load balancer
alongside [cert-manager](cert-manager.io/) to provide automatic SSL certificates. Given that this chart serves as an
umbrella chart for additional dependencies cert-manager will be installed on the same namespace as the parent chart,
you should take special care not to install cert-manager twice due to it installing several non-namespaced resources.
In addition, you would be in charge of
[managing the CRDs used by cert-manager](https://cert-manager.io/docs/installation/upgrading/#crds-managed-separately)
always upgrading to the corresponding version before updating the chart. This is due to the some limitations
in the management of CRDs by Helm.
If you already installed cert-manager by different means, make sure set `cert-manager.enabled: false` for this chart.
## Motivation

## Central Database/Monitoring/etc
Many Open edX providers and users have a need to deploy multiple instances of Open edX onto Kubernetes, but there is currently no standardized way to do so and each provider must build their own tooling to manage that. This project aims to provide an easy and standardized approach that incorporates industry best practices and lessons learned.

In the future, this chart could also be made to install monitoring, databases, ElasticSearch, or other shared resources
that can be shared by all the instances in the cluster, following all the best practices. For now it's just a proof of
concept that focuses on load balancing and integration with the existing Tutor plugins/ecosystem.
In particular, this project aims to provide the following benefits to Open edX operators:
* **Ease of use** and **rapid deployment**: This project aims to provide an Open edX hosting environment that just works out of the box, that can be easily upgraded, and that follows best practices for monitoring, security, etc.
* **Lower costs** by sharing resources where it makes sense. For example, by default Tutor's k8s feature will deploy a separate load balancer and ingress controller for each Open edX instance, instead of a shared ingress controller for all the instances in the cluster. Likewise for MySQL, MongoDB, ElasticSearch, and other resources. By using shared resources by default, costs can be dramatically reduced and operational monitoring and maintenance is greatly simplified.
- For setups with many small instances, this shared approach provides a huge cost savings with virtually no decrease in performance.
- For larger instances on the cluster that need dedicated resources, they can easily be configured to do so.
* **Scalable hosting** for instances of any size. This means for example that the default configuration includes autoscaling of LMS pods to handle increased traffic.
* **Flexibility**: this project aims to be "batteries included" and to support setting up all the resources that you need, with useful default configurations, but it is carefully designed so that operators can configure, replace, or disable any components as needed.

Alternately, many of those shared databases can be provisioned outside of the cluster using Terraform, and configured to
work with the instances that Tutor deploys. That approach is outside the scope of this Helm chart but see
[Grove](https://grove.opencraft.com/) for an open source implementation.
## Technology stack and architecture

1. At the base is a Kubernetes cluster, which you must provide (e.g. using Terraform to provision Amazon EKS).
* Any cloud provider such as AWS or Digital Ocean should work. There is an example Terraform setup in `infra-example` but it is just a starting point and not recommended for production use.
2. On top of that, this project's helm chart will install the shared resources you need - an ingress controller, monitoring, database clusters, etc. The following are included but can be disabled/replaced if you prefer an alternative:
* Ingress controller: [ingress-nginx](https://kubernetes.github.io/ingress-nginx/)
* Automatic HTTPS cert provisioning: [cert-manager](https://cert-manager.io/)
* Autoscaling: `metrics-server` and `vertical-pod-autoscaler`
* Search index: ElasticSearch (support for OpenSearch is planned)
* Monitoring: TODO
* Database clusters: TODO (for now we recommend provisioning managed MySQL/MongoDB database clusters from your cloud provider using Terraform or a tool like [Grove](https://grove.opencraft.com/).)
* Where possible, we try to configure these systems to **auto-detect** newly deployed Open edX instances and adapt to them automatically; where that isn't possible, Tutor plugins are used so that the instances self-register or self-provision the shared resources as needed.
3. [Tutor](https://docs.tutor.overhang.io/) is used to build the container images that will be deployed onto the cluster.
* This project's Tutor plugin is required to make the images compatible with the shared resources deployed by the Helm chart.
* The
[pod-autoscaling plugin](https://github.com/eduNEXT/tutor-contrib-pod-autoscaling) is required for autoscaling.
* You can use the `tutor k8s` commands directly (as documented below) or you can use a CI/CD tool like [Grove](https://grove.opencraft.com/) to deploy instances/images.

<br><br><br>
## FAQ

### Is this ready to use in production?

We are tracking that in [issue 26](https://github.com/openedx/openedx-k8s-harmony/issues/26), so check that issue for the current status.

### This project aims to support many small/medium instances deployed onto a cluster; is it also suitable for deploying one really high traffic instance?

Supporting one really large instance is not a core design goal, but it should work well and we may consider including this as a goal in the future. Please reach out to us or get involved with this project if you have this requirement.

How to use:
### What are the gotchas related to cert-manager?

## Step 1: Use Helm to provision a kubernetes cluster using this chart
This helm chart uses [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) as a load balancer alongside [cert-manager](https://cert-manager.io/) to provide automatic SSL certificates. Because of how Helm works, the cert-manager sub-chart will be installed into the same namespace as the parent harmony chart. But if you already have cert-manager on your cluster, this will create a conflict. You should take special care not to install cert-manager twice due to it installing several non-namespaced resources. If you already installed cert-manager by different means, make sure set `cert-manager.enabled: false` for this chart.

### Option 1a: Setting up Harmony Chart on a cloud-hosted Kubernetes Cluster (recommended)
In addition, [the cert-manager Helm charts do not install the required CRDs used by cert-manager](https://cert-manager.io/docs/installation/upgrading/#crds-managed-separately), so you will need to manually install and upgrade them to the correct version as described in the instructions below. This is due to the some limitations in the management of CRDs by Helm.



<br><br><br>

## Usage Instructions

### Step 1: Use Helm to provision a kubernetes cluster using this chart

#### Option 1a: Setting up Harmony Chart on a cloud-hosted Kubernetes Cluster (recommended)

For this recommended approach, you need to have a Kubernetes cluster in the cloud **with at least 12GB of usable
memory** (that's enough to test 2 Open edX instances).
Expand All @@ -69,7 +91,7 @@ Note: in the future, if you make any changes to `values.yaml`, then run this com
helm upgrade --namespace harmony -f values.yaml harmony ./harmony-chart
```

### Option 1b: Setting up Harmony Chart locally on Minikube
#### Option 1b: Setting up Harmony Chart locally on Minikube

*Note: if possible, it's preferred to use a cloud-hosted cluster instead (see previous section). But if you don't have a
cluster available in the cloud, you can use minikube to try this out locally. The minikube version does not support
Expand All @@ -88,7 +110,7 @@ HTTPS and is more complicated due to the need to use tunnelling.*
to include the port numbers shown above when accessing the instances.


## Step 2: Get the external IP
### Step 2: Get the external IP

The [ingress NGINX Controller](https://kubernetes.github.io/ingress-nginx/) is used to automatically set up an HTTPS
reverse proxy for each Open edX instance as it gets deployed onto the cluster. There is just one load balancer with a
Expand All @@ -101,7 +123,7 @@ kubectl get svc -n harmony harmony-ingress-nginx-controller
To test that your load balancer is working, go to `http://<the external ip>/cluster-echo-test` .
You may need to ignore the HTTPS warnings, but then you should see a response with some basic JSON output.

## Step 3: Deploying an Open edX instance using Tutor
### Step 3: Deploying an Open edX instance using Tutor

Important: First, get the load balancer's IP (see "external IP" above), and set the DNS records for the instance you
want to create to be pointing to the load balancer (Usually if you want the LMS at `lms.example.com`, you'll need to set
Expand All @@ -110,7 +132,7 @@ two A records for `lms.example.com` and `*.lms.example.com`, pointing to the ext
You also will need to have the tutor-contrib-harmony-plugin installed into Tutor:

```
pip install -e 'git+https://github.com/open-craft/tutor-contrib-harmony.git#egg=tutor-contrib-harmony-plugin&subdirectory=tutor-contrib-harmony-plugin'
pip install -e 'git+https://github.com/openedx/openedx-k8s-harmony.git#egg=tutor-contrib-harmony-plugin&subdirectory=tutor-contrib-harmony-plugin'
```

Next, create a Tutor config directory unique to this instance, and configure it:
Expand All @@ -137,11 +159,10 @@ of the box.
**You can repeat step 3 many times to install multiple instances onto the cluster.**


<br><br><br><br>

----------------
<br><br><br>

## Configuration
## Configuration Reference

### Multi-tenant Elasticsearch

Expand All @@ -165,11 +186,13 @@ RUN_ELASTICSEARCH: false

In order for SSL to work without warnings the CA certificate needs to be mounted in the relevant pods. This is not yet implemented as due to an [outstanding issue in tutor](https://github.com/overhangio/tutor/issues/791) that had not yet been completed at the time of writing.

## Appendix : how to uninstall this chart
## Extended Documentation

### How to uninstall this chart

Just run `helm uninstall --namespace harmony harmony` to uninstall this.

## Appendix B: how to create a cluster for testing on DigitalOcean
### How to create a cluster for testing on DigitalOcean

If you use DigitalOcean, you can use Terraform to quickly spin up a cluster, try this out, then shut it down again.
Here's how. First, put the following into `infra-tests/secrets.auto.tfvars` including a valid DigitalOcean access token:
Expand Down

0 comments on commit 5df8d52

Please sign in to comment.