manifest-template

A GitOps manifest template to get you started with ArgoCD and validation on GitHub Actions.

See also: infra-template

Directory structure

ArgoCD's "root" Application loads ./dev/.applications directory. ApplicationSet discovers applications in ./dev/*, where directory names correspond to application name and namespace name.

manifest
├── dev
│   ├── .applications
│   │   ├── application-set.yaml
│   │   └── kustomization.yaml
│   └── my-app
│       ├── resource-1.yaml
│       ├── resource-2.yaml
│       └── kustomization.yaml
└── prod
    ├── .applications
    │   ├── application-set.yaml
    │   └── kustomization.yaml
    └── my-app
        ├── resource-1.yaml
        ├── resource-2.yaml
        └── kustomization.yaml

Setup

Setup k8s cluster

This repository assumes that you already own a Kubernetes cluster. See infra-template to get started. Make sure you can run kubectl apply to deploy initial resources.

Create your repository from this template

To get started, press the "Use this template" button on the top right of this page.

We recommend importing recommended-ruleset.json to your repository (Settings -> Rules -> Rulesets -> Import a ruleset). Adjust rulesets to your liking.

We intentionally allow "Deploy keys" to bypass the ruleset. This is to allow other CI/CD systems to directly push to main branch of this repository.

Setup sops / age

1. Install sops and age locally.
2. Run age-keygen -o key.txt to generate a key pair.
   • key.txt is the private key, so you want to keep this secret.
   • Public key is printed to stdout. Set this to .sops.yaml.
3. Create a Secret named age-key in argocd namespace.
   • kubectl create ns argocd && kubectl create secret generic age-key -n argocd --from-file=key.txt
   • This is referenced from ./dev/argocd manifest.
4. Place key.txt somewhere safe, or delete if you don't need it.
   • macOS recommended location: $HOME/Library/Application Support/sops/age/keys.txt
   • https://github.com/getsops/sops?tab=readme-ov-file#23encrypting-using-age

How to handle secrets

See scripts/secret-*.sh for various utility scripts to manipulate sops-encrypted files.

Most basic ones:

• scripts/secret-encrypt.sh: Encrypt a file.
• scripts/secret-set-key.sh: Set a key-value pair in an encrypted file.
• scripts/secret-set-key-base64.sh: Use this instead if your value has special characters.

You will likely NOT need to use secret-decrypt.sh or secret-edit.sh NOR need the private key locally, given the fact that age encryption is asymmetric. You could even commit encrypted files to a public repository.

After encrypting a file with scripts/secret-encrypt.sh (usually placed at ./dev/*/secrets/*.yaml), reference them via ksops file (usually placed at ./dev/*/ksops.yaml).

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: ksops
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops

files:
  - ./secrets/argocd-secret.yaml
  - ./secrets/notifications.yaml

Note

Files included in this template are already encrypted with an example key. Remove the sops key in YAML, replace the contents with your actual values, then run scripts/secret-encrypt.sh <filename> to encrypt the file. Update ksops.yaml as needed.

Install Karpenter

If you have created your cluster with infra-template, it can't schedule normal nodes / pods yet because there are no associated node groups. We will use Karpenter to provision nodes for normal pods.

1. Configure placeholders in ./dev/karpenter/values.yaml and ./dev/karpenter/default.yaml.
2. Create karpenter namespace: kubectl create ns karpenter
3. Deploy Karpenter: scripts/build.sh ./dev/karpenter | kubectl apply -f -

Note that the Karpenter controller itself is meant to be scheduled on Fargate nodes. The corresponding Fargate profile is created in infra-template.

Install ArgoCD

See ./dev/argocd/values.yaml to configure values. You will likely need to change:

• global.domain: Your ArgoCD host.
• configs.cm."oidc.config": OIDC configuration.
• configs.rbac: RBAC configuration.

Then:

1. Deploy ArgoCD: scripts/build.sh ./dev/argocd | kubectl apply -f -
2. Get admin password: kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 --decode && echo
3. Port-forward to access temporarily at localhost:8080: kubectl port-forward -n argocd svc/argocd-server 8080:8080

Connect the manifest repository

1. Create a GitHub App.
   • Minimal permission required: Contents: Read-Only
     • If you want to discover Applications based on PRs, you will also need Pull-Requests: Read-Only
   • Then, install this app into your manifest repository.
   • Take note of: GitHub App ID, installation ID (check the URL in your browser after installation!), and private key.
2. Add repository from ArgoCD UI.
3. Configure webhook on push from GitHub to https://<your-argocd-url>/api/webhook.
   • Set webhook "Content Type" to application/json.
   • https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/

Sync applications

You can start adding applications from the ArgoCD UI. Check if you need other applications under ./dev/*, and replace placeholders as needed.

Recommended first applications to sync (in order):

• kyverno (adds pull-through-cache image rewrite)
• aws-load-balancer-controller
• traefik (includes ALB and Ingress Controller for use in other apps)

After you're done creating some essential applications and done deleting unnecessary application directories, you can add a "root" application to automatically discover and sync all applications under ./dev/*.

To create a "root" Application, replace <your-org>/<your-repo> in ./dev/.application/application-set.yaml with your actual repository names. Then, add an Application pointing to ./dev/.applications directory.

Note

To prevent accidental deletion of Applications' resources, the following syncPolicy is in place for ApplicationSet:

syncPolicy:
  # Prevents ArgoCD from deleting Application resources when the Application is deleted.
  # https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Application-Deletion/
  preserveResourcesOnDeletion: true
  # Prevents ApplicationSet from deleting Application when it is no longer discovered.
  # https://argo-cd.readthedocs.io/en/latest/operator-manual/applicationset/Controlling-Resource-Modification/
  applicationsSync: create-update

(And applicationsetcontroller.enable.policy.override is set to true in argocd-cmd-params-cm.)

To properly delete an Application itself and its resources, first delete the directory from the git repository and then delete the Application using cascading deletion from the UI.

Setup ArgoCD notifications (Slack)

https://argo-cd.readthedocs.io/en/stable/operator-manual/notifications/services/slack/

1. Create a Slack App

   • Example app manifest:

     display_information:
       name: ArgoCD (dev)
       description: ArgoCD (dev) notifications
       background_color: "#a36d00"
     features:
       bot_user:
         display_name: ArgoCD (dev)
         always_online: false
     oauth_config:
       scopes:
         bot:
           - chat:write
           - chat:write.customize
     settings:
       org_deploy_enabled: false
       socket_mode_enabled: false
       token_rotation_enabled: false

2. Obtain an OAuth Token (xoxp-...) and set it to argocd-notifications-secret (./dev/argocd/secrets/notifications.yaml). Encrypt the secret with scripts/secret-encrypt.sh, and update ksops.yaml accordingly.

   apiVersion: v1
   kind: Secret
   metadata:
     name: argocd-notifications-secret
   
   stringData:
     slack-token: "replace-me-with-actual-token"

Best practices

Use repository-local helm chart

When you need to share a set of manifest definitions between environments / applications, place a helm chart under ./base (or somewhere else). ./base/traefik-forward-auth is one such example.

Creating repository-local helm chart and simply rendering from kustomization.yaml is much simpler and easier to maintain than using traditional "overlays" and "patches" using kustomize.

At first, kustomize patch seems to be simpler than Helm's template engine to patch a few locations - but as your manifests grow, the patch set quickly blows up and it becomes almost impossible for newcomers to understand what's going on.

Helm template engine gives you much more control and flexibility for managing common manifest definitions. values.yaml is the "interface" between the common definition and your actual use-case. Try to "design" these values.yaml to be as agnostic as possible to your actual use-cases. There is no need to make every value configurable via values.yaml - this chart is only used locally after all - try to keep values.yaml as simple as possible.

Use third-party helm charts whenever possible

Similarly, you will likely want to rely on third-party helm charts as much as possible.

At first, you might want to reference a resource URL (e.g. https://raw.githubusercontent.com/argoproj/argo-cd/refs/heads/master/manifests/install.yaml) from kustomization.yaml and maintain a few kustomize patches along, but for similar reasons as above, kustomize patches and extra resources quickly become unmaintainable.

Try to rely on well-maintained helm charts whenever possible, so you write less code and get more work done.

Examples:

• ArgoCD: https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd
• victoria-metrics-k8s-stack (all-in-one monitoring solution): https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack
• Self-host Sentry (in case you prefer self-hosting): https://github.com/sentry-kubernetes/charts
• OpenCost (for tracking resource costs): https://github.com/opencost/opencost-helm-chart/tree/main/charts/opencost

Do not hesitate to push to main frequently

This repository is intended to be used with ArgoCD with GitOps principles. Create a PR, wait for CI to pass, merge, and repeat quickly. Even if you break something, just do git revert && git push and you're all good (for most cases).

When you have a commit history and this GitOps repository as the "single source of truth" for your infrastructure, rollbacks and post-mortem investigations become much easier.

Do not use kubectl to manipulate cluster states

For similar reasons as well - keep this repository as the "single source of truth" for your infrastructure.

Use kubectl or k9s only for inspecting cluster states. alias k9s-ro='k9s --readonly' is one good alias to use. When you need to make changes to the cluster, always make changes via this repository.

Use centralized SSO (single-sign-on)

./dev/auth and ./dev/traefik gives you centralized SSO and allow apps to use header authentication via traefik ForwardAuth middlewares.

traPtitech/traefik-forward-auth gives you a simple, yet powerful SSO and header authentication solution via OAuth2 or OIDC. This assumes you already have an identity provider (IdP). We recommend using Google's OAuth2 App for starters.

Configure "groups" in ./base/traefik-forward-auth/values.yaml. (see the actual templates for more details for how this is done)

groups:
  admin:
    - [email protected]

This will generate auth-group-${group_name} middlewares for each group, so reference them in your IngressRoute definitions.

    - kind: Rule
      match: Host(`traefik.example.com`)
      middlewares:
        - name: auth-group-admin
          namespace: auth
      services:
        - kind: TraefikService
          name: dashboard@internal

This will also pass X-Forwarded-User and X-Forwarded-Sub headers to your application. If your application supports header authentication (a.k.a "proxy authentication"), configure them accordingly.

For example, Grafana supports proxy authentication.