| title | parent | thumbnail | tags | date | description | MRM | ali-mukadam | xredirect | slug | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Hosting a private Helm Repository on OCI (Oracle Cloud Infrastructure) with ChartMuseum and OCI Object Storage |
|
assets/helm-network-diagram.png |
|
2021-12-15 12:21 |
This guide walks you through how to host Helm Chart privately with ChartMuseum, an open source Helm Chart Repository server with support for various cloud storage backends, including OCI Object Storage. |
WWMK211213P00054 |
|
helm-repository-chartmuseum-object |
In another article, we deployed a helm chart (Redis) into Oracle Container Engine (OKE) which we pulled from the stable repository.
We've already described how to host the container images privately on Oracle Registry Service (OCIR). What if we want to do the same for our helm charts?
Fortunately, there is a solution for that too.
In this article we'll introduce you to chartmuseum, an open source Helm Chart Repository server with support for various cloud storage backends, including OCI Object Storage.
By default, the stable repo is hosted at here.
We can also manually add the incubator repository if we want to:
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/This should respond with something similar to:
"incubator" has been added to your repositoriesThe container images in those repositories are also publicly available (for example, the default image for redis is bitnami/redis in bitnami’s container redis repository on Dockerhub).
- An Oracle Cloud Infrastructure Free Tier account. [Start for Free]({{ site.urls.always_free }}).
- A MacOS, Linux, or Windows computer with
sshsupport installed. - OCI Cloud Shell - It provides a great platform for quickly working with Terraform as well as a host of other OCI interfaces and tools.
In this tutorial, we'll be using the terraform-oci-oke project to provision the OKE cluster. For more details, see this post on provisioning OKE using Terraform.
Below is an architecture of what we’ll build:
{% imgx assets/helm-network-diagram.png "Deploying chartmuseum on OCI" %}
-
Install DNS - Follow this post to install ExternalDNS.
-
nginx - Install nginx-controller.
We’ll use this to access chartmuseum later.In the command below, replace
chartmuseum.acme.comby your preferred hostname:helm install --name nginxcontroller stable/nginx-ingress \ --set controller.name=controller \ --set defaultBackend.enabled=true \ --set defaultBackend.name=defaultbackend \ --set rbac.create=true \ --set controller.service.annotations."external-dns\.alpha\.kubernetes\.io/hostname"=chartmuseum.acme.com
-
Verify that an OCI Load Balancer has been created and a DNS
Arecord has been inserted in the DNS Zone.
In the OCI Console, navigate to Object Storage and create a bucket called chartmuseum:
{% imgx assets/helm-chartmuseum-bucket.png "chartmuseum bucket window" %}
We want a similar experience to the stable or incubator repos:
- anonymous
GETbut protectedPOSTrequests - Basic Auth for authentication purposes
We'll need to create 2 secrets to accomplish this.
-
secret 1 - Let’s start by creating a secret to hold the Basic Auth username and password:
kubectl create secret generic chartmuseum-auth --from-literal=user=curator --from-literal=pass=password -
secret 2 - You'll also need to create a secret that will allow chartmuseum to communicate with the OCI APIs.
-
Temporarily, copy your API keys to the bastion. You can also do this locally if you have
kubectland local access to the kubeconfig. -
Create a file
configand enter your user and tenancy OCIDs, api key fingerprint, and region value. Thekey_filevalue has to be/home/chartmuseum/.oci/oci.key.[DEFAULT] user=<USER_OCID> fingerprint=<API_KEY_FINGERPRINT> key_file=/home/chartmuseum/.oci/oci.key tenancy=<TENANCY_OCID> region=<REGION>
-
Create the secret using appropriate absolute or relative path to the config file and private API key.
kubectl create secret generic chartmuseum-secret --from-file=config="/path/to/config" --from-file=key_file="/path/to/apikey.pem"
-
With the secrets created, we're ready to start deploying chartmuseum.
-
Download the chartmuseum values file:
curl -o values.yaml https://raw.githubusercontent.com/helm/charts/master/stable/chartmuseum/values.yaml -
Edit the
values.yamlfile and replace the parameters as follows:env: open: STORAGE: oracle STORAGE_ORACLE_COMPARTMENTID:<COMPARTMENT_OCID> STORAGE_ORACLE_BUCKET: chartmuseum STORAGE_ORACLE_PREFIX: chartmuseum DISABLE_API: false AUTH_ANONYMOUS_GET: true AUTH_REALM: chartmuseumexistingSecret: chartmuseum-auth existingSecretMappings: BASIC_AUTH_USER: user BASIC_AUTH_PASS: passingress: enabled: true labels: dns: "ocidns"annotations: kubernetes.io/ingress.class: nginxhosts: - name: chartmuseum.acme.com path: / tls: falseoracle: secret: enabled: true name: chartmuseum-secret config: config key_file: key_file
-
Install chartmuseum:
helm install --name=chartmuseum -f values.yaml stable/chartmuseum -
Wait for the pod to run:
kubectl get pods -wThis should give output similar to:
NAME READY STATUS RESTARTS AGE chartmuseum-chartmuseum-748c8dbbd8-7nctc 1/1 Running 0 5m20s
-
Verify that the ingress has been created:
kubectl get ingThis should give output similar to:
NAME HOSTS ADDRESS PORTS AGE chartmuseum-chartmuseum chartmuseum.acme.com 80 7m9s
-
And that it also maps to the chartmuseum service:
k describe ingThis should give output similar to:
chartmuseum-chartmuseum Name: chartmuseum-chartmuseum Namespace: default Address: Default backend: default-http-backend:80 () Rules: Host Path Backends ---- ---- -------- chartmuseum.acme.com / chartmuseum-chartmuseum:8080 () Annotations: kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 6m20s nginx-ingress-controller Ingress default/chartmuseum-chartmuseum Normal UPDATE 5m31s nginx-ingress-controller Ingress default/chartmuseum-chartmuseum
-
Verify whether you can reach chartmuseum publicly with your browser.
{% imgx assets/helm-chartmuseum-welcome-msg.png "Chartmuseum welcome message" %}
-
Install the helm
pushplugin:helm plugin install https://github.com/chartmuseum/helm-push -
Add the repo:
helm repo add --username curator --password password cm http://chartmuseum.acme.com/ -
Create a basic test chart,
mycharthelm create mycharthelm package mychartThis should respond with something similar to:
Successfully packaged chart and saved it to: /home/opc/chart/mychart-0.1.0.tgz -
Push the chart:
helm push mychart cmThis should respond with something similar to:
Pushing mychart-0.1.0.tgz to cm... Done.
If we search for
mychartat this point, we'll only find the local copy:helm search mychartThis will respond with something similar to:
NAME CHART VERSION APP VERSION DESCRIPTION local/mychart 0.1.0 1.0 A Helm chart for Kubernetes
-
We'll need to do a few things to get
mypartto appear.-
First, do a
repo update:helm repo update cmThis should respond with something similar to:
Hang tight while we grab the latest from your chart repositories... ...Skip local chart repository ...Successfully got an update from the "cm" chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈ Happy Helming!⎈
-
And then, conduct a new search:
helm search mychartThis should respond with something similar to:
NAME CHART VERSION APP VERSION DESCRIPTION cm/mychart 0.1.0 1.0 A Helm chart for Kubernetes local/mychart 0.1.0 1.0 A Helm chart for Kubernetes
Our test chart (
mychart) appears!
-
At this point, we could test authentication by removing the repo and adding it again without the username and password. Instead, we'll try something a little more elegant and use a combination of Postman and chartmuseum’s APIs to test.
These are the behaviors we're looking for:
GET: No Auth RequiredPOST: Auth RequiredDELETE: Auth Required
{% imgx assets/helm-get-request-w-chartmuseum.png "GET (without authentication)" %}
From this, you can see we're able to get a list of charts using GET without authentication.
However, a DELETE or POST without authentication fails:
{% imgx assets/helm-chartmuseum-unauth-request.png "Unauthorized chart deletion" %}
Yet another twist: when we enter the credentials, the chart deletion succeeds:
{% imgx assets/helm-chartmuseum-good-request.png "Successful chart deletion" %}
If we now check OCI Object Storage, we can see our chart there:
{% imgx assets/helm-oci-dash.png "OCI dashboard displaying Chartmuseum details " %}
Now that chartmuseum is up and running, we'll also want to secure its usage by encrypting traffic. To do this, we'll use Let’s Encrypt and cert-manager. cert-manager is an “add-on to automate the management and issuance of TLS certificates from various issuing sources.”
At this point, you can follow along with the cert-manager installation guide or read on.
-
Create the
CustomResourceDefinitionsnamespace for cert-manager and disable resource validation on the namespace:kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yamlkubectl create namespace cert-managerkubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true -
Add the Jetpack Helm repo and update:
helm repo add jetstack https://charts.jetstack.io helm repo update jetstack
-
Install cert-manager using its helm chart:
helm install --name cert-manager --namespace cert-manager \ -
Verify the installation:
kubectl get pods --namespace cert-managerThis should respond with something similar to:
NAME READY STATUS RESTARTS AGE cert-manager-776cd4f499-98vsh 1/1 Running 0 3h14m cert-manager-cainjector-744b987848-pkk5s 1/1 Running 0 3h14m cert-manager-webhook-645c7c4f5f-4mbjd 1/1 Running 0 3h14m
-
Test that the webhook works by creating a
test-resources.yamlfile:apiVersion: v1 kind: Namespace metadata: name: cert-manager-test --- apiVersion: certmanager.k8s.io/v1alpha1 kind: Issuer metadata: name: test-selfsigned namespace: cert-manager-test spec: selfSigned: {} --- apiVersion: certmanager.k8s.io/v1alpha1 kind: Certificate metadata: name: selfsigned-cert namespace: cert-manager-test spec: commonName: example.com secretName: selfsigned-cert-tls issuerRef: name: test-selfsigned
-
Create the test resources:
kubectl create -f test-resources.yaml -
Check the status of the newly-created certificate:
kubectl describe certificate -n cert-manager-testName: selfsigned-certIt should respond with something similar to:
Namespace: cert-manager-test Labels: Annotations: API Version: certmanager.k8s.io/v1alpha1 Kind: Certificate Metadata: Creation Timestamp: 2019-06-25T00:45:25Z Generation: 1 Resource Version: 116448 Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/cert-manager-test/certificates/selfsigned-cert UID: 8576413b-96e2-11e9-b5fc-0a580aed39b8 Spec: Common Name: example.com Issuer Ref: Name: test-selfsigned Secret Name: selfsigned-cert-tls Status: Conditions: Last Transition Time: 2019-06-25T00:45:26Z Message: Certificate is up to date and has not expired Reason: Ready Status: True Type: Ready Not After: 2019-09-23T00:45:26Z Events:
Note: You can delete the test resources after testing:
kubectl delete -f test-resources.yaml
{:notice}
For reference, you can follow the quick-start guide for using cert-manager with Nginx Ingress.
-
Let’s start with creating a
staging issuerby creating astaging-issuer.yamland using your email address:apiVersion: certmanager.k8s.io/v1alpha1 kind: Issuer metadata: name: cm-staging spec: acme: # The ACME server URL server: https://acme-staging-v02.api.letsencrypt.org/directory # Email address used for ACME registration email: your_email_address # Name of a secret used to store the ACME account private key privateKeySecretRef: name: cm-staging # Enable the HTTP-01 challenge provider http01: {}
-
Create the
staging issuer:kubectl create -f staging-issuer.yamlThis will respond with something similar to:
issuer.certmanager.k8s.io/cm-staging created -
Repeat the above for a production environment by creating a
production-issuer.yaml, once again using your email address:apiVersion: certmanager.k8s.io/v1alpha1 kind: Issuer metadata: name: cm-prod spec: acme: # The ACME server URL server: https://acme-v02.api.letsencrypt.org/directory # Email address used for ACME registration email: your_email_address # Name of a secret used to store the ACME account private key privateKeySecretRef: name: cm-prod # Enable the HTTP-01 challenge provider http01: {}
-
Create the issuer:
kubectl create -f production-issuer.yamlCheck the status of the staging issuer:
kubectl describe issuer cm-stagingName: cm-stagingThis should respond with something similar to:
Namespace: default Labels: Annotations: API Version: certmanager.k8s.io/v1alpha1 Kind: Issuer Metadata: Creation Timestamp: 2019-06-25T03:28:16Z Generation: 1 Resource Version: 143856 Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/issuers/cm-staging UID: 452908da-96f9-11e9-b5fc-0a580aed39b8 Spec: Acme: Email: your_email_address Http 01: Private Key Secret Ref: Name: cm-staging Server: <https://acme-staging-v02.api.letsencrypt.org/directory> Status: Acme: Uri: <https://acme-staging-v02.api.letsencrypt.org/acme/acct/9718789> Conditions: Last Transition Time: 2019-06-25T03:28:17Z Message: The ACME account was registered with the ACME server Reason: ACMEAccountRegistered Status: True Type: Ready Events:
Edit your values.yaml file for chartmuseum and look for the Ingress section. Pay special attention to the added configuration commands.
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
certmanager.k8s.io/issuer: "cm-staging"
certmanager.k8s.io/acme-challenge-type: http01
## Chartmuseum Ingress hostnames
## Must be provided if Ingress is enabled
##
hosts:
- name: chartmuseum.acme.com
path: /
tls: true
tlsSecret: cm-tls-
Upgrade your helm chart:
helm upgrade chartmuseum stable/chartmuseum -f values.yamlThis will respond with something similar to:
Release "chartmuseum" has been upgraded. Happy Helming! -
cert-manager will read the annotations and create a certificate:
kubectl get certificateThis will respond with something similar to:
NAME READY SECRET AGE cm-tls True cm-tls 45m
-
Take a quick peek at the certificate:
kubectl describe certificate cm-tlsName: cm-tlsThis will respond with something similar to:
Namespace: default Labels: app=chartmuseum chart=chartmuseum-2.3.1 dns=ocidns heritage=Tiller release=chartmuseum Annotations: API Version: certmanager.k8s.io/v1alpha1 Kind: Certificate Metadata: Creation Timestamp: 2019-06-25T03:33:47Z Generation: 1 Owner References: API Version: extensions/v1beta1 Block Owner Deletion: true Controller: true Kind: Ingress Name: chartmuseum-chartmuseum UID: d5a9cee1-9673-11e9-a790-0a580aed1020 Resource Version: 152567 Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/cm-tls UID: 0ac806f6-96fa-11e9-8836-0a580aed4b3e Spec: Acme: Config: Domains: chartmuseum.acme.com Http 01: Ingress Class: nginx Dns Names: chartmuseum.acme.com Issuer Ref: Kind: Issuer Name: cm-staging Secret Name: cm-tls Status: Conditions: Last Transition Time: 2019-06-25T04:19:27Z Message: Certificate is up to date and has not expired Reason: Ready Status: True Type: Ready Not After: 2019-09-23T03:19:27Z Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Cleanup 44m cert-manager Deleting old Order resource "cm-tls-3817114402" Normal OrderCreated 43m (x2 over 44m) cert-manager Created Order resource "cm-tls-3433596774" Normal OrderComplete 43m cert-manager Order "cm-tls-3433596774" completed successfully Normal Cleanup 2m21s cert-manager Deleting old Order resource "cm-tls-3433596774" Normal Generated 2m14s (x3 over 47m) cert-manager Generated new private key Normal GenerateSelfSigned 2m14s (x3 over 47m) cert-manager Generated temporary self signed certificate Normal OrderCreated 2m14s (x3 over 47m) cert-manager Created Order resource "cm-tls-3817114402" Normal OrderComplete 2m13s (x2 over 47m) cert-manager Order "cm-tls-3817114402" completed successfully Normal CertIssued 2m13s (x3 over 47m) cert-manager Certificate issued successfully
Once complete, cert-manager will have created a secret with the details of the certificate based on the secret used in the ingress resource. You can use the describe command as well to see some details:
kubectl describe secret cm-tlsThis should respond with something similar to:
Name: cm-tls
Namespace: default
Labels: certmanager.k8s.io/certificate-name=cm-tls
Annotations: certmanager.k8s.io/alt-names: chartmuseum.acme.com
certmanager.k8s.io/common-name: chartmuseum.acme.com
certmanager.k8s.io/ip-sans:
certmanager.k8s.io/issuer-kind: Issuer
certmanager.k8s.io/issuer-name: cm-staging
Type: kubernetes.io/tls
Data
====
ca.crt: 0 bytes
tls.crt: 3578 bytes
tls.key: 1679 bytesIf you access chartmuseum now, you’ll see a warning:
{% imgx assets/helm-warning-msg.png "Warning message in browser" %}
If you ignore the warning and go ahead anyway, you'll be able to access chartmuseum except that now you'll be accessing it over HTTPS. Your browser will also warn you that it’s added an exception to this site.
Let's take a look at how to correct this:
-
Edit the
values.yamlfor your chartmuseum again and this time change theissuer annotationto cm-prod:certmanager.k8s.io/issuer: "cm-prod" -
Run a helm upgrade again:
helm upgrade chartmuseum stable/chartmuseum -f values.yamlThis should respond with something similar to:
Release "chartmuseum" has been upgraded. Happy Helming! -
Delete the secret:
kubectl delete secret cm-tlsThis will cause cert-manager to get a new certificate. You can verify this:
kubectl describe certificate cm-tlsThis will respond with something similar to:
. . .Normal Generated 33s (x4 over 61m) cert-manager Generated new private key Normal GenerateSelfSigned 33s (x4 over 61m) cert-manager Generated temporary self signed certificate Normal OrderCreated 33s (x4 over 58m) cert-manager Created Order resource "cm-tls-3433596774" Normal CertIssued 31s (x5 over 61m) cert-manager Certificate issued successfully Normal OrderComplete 31s (x3 over 57m) cert-manager Order "cm-tls-3433596774" completed successfully
If you now access chartmuseum, you will be able to access it with HTTPS and will not be prompted with the security warning/exception.
{% imgx assets/helm-chartmuseum-welcome.png "Chartmuseum welcome message" %}
We've covered a lot of ground in this article. Hopefully, you've found this useful!
At this point, you should be ready to explore all of the features that chartmuseum has to offer. Chartmuseum enhances your CI/CD capabilities by:
- enabling you to host your helm charts privately and securely
- integrating with your CI/CD deployment tool chain and pipelines
- supporting multiple teams and organizations with multitenant capabilities
- using a variety of storage capabilities including local file system, Oracle Object Storage, OpenStack Object storage, and others.
Running chartmuseum outside of your Kubernetes cluster:
If you'd like to run it outside your Kubernetes cluster (like on a VM), you can follow this guide instead.
Additional information:
To explore more information about development with Oracle products: