From 751b6ee30429a572ed0ebc0b9ae890037cf325ab Mon Sep 17 00:00:00 2001 From: Eron Wright Date: Thu, 20 Feb 2025 01:56:40 -0800 Subject: [PATCH 1/4] Update to PKO docs for v2.0 --- .../pulumi-kubernetes-operator.md | 557 +++++++++++------- 1 file changed, 341 insertions(+), 216 deletions(-) diff --git a/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md b/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md index c7ed795993a6..b3bcafb249e3 100644 --- a/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md +++ b/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md @@ -17,102 +17,186 @@ aliases: --- This page details how to use the [Pulumi Kubernetes -Operator](https://github.com/pulumi/pulumi-kubernetes-operator) to manage deploying -[Stacks][stack]. The Pulumi program for a Stack can come from a [Program resource][], from git, or from a [Flux source][flux-source]. +Operator](https://github.com/pulumi/pulumi-kubernetes-operator) (PKO) to automate the deployment of Pulumi [stacks][stack]. The Pulumi program for a stack can come from a [Program resource][], from a Git repository, or from a [Flux source][flux-source], and may be authored in any supported Pulumi language (TypeScript, Python, Go, .NET, Java, YAML). [Program resource]: https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/programs.md [flux-source]: https://fluxcd.io/flux/components/source/ ## Overview -The Pulumi Kubernetes Operator is an [extension pattern][k8s-ext-pattern] that -enables Kubernetes users to create a `Stack` as a first-class API -resource, and use the `StackController` to drive the updates of the Stack until -success. +The Pulumi Kubernetes Operator provides [custom resources][k8s-ext-pattern] to: +- Provision a workspace (an execution environment) for a Pulumi project +- Keep a Pulumi stack up-to-date using gitops +- Write [Pulumi YAML][] programs as Kubernetes objects +- Run Pulumi deployment operations -Deploying Pulumi Stacks in Kubernetes provides the capability to build -out CI/CD and automation systems into your clusters, creating native support to manage your infrastructure alongside your Kubernetes workloads. +Deploying Pulumi stacks using Kubernetes provides the capability to build +out CI/CD and other automation systems, and to manage your infrastructure alongside your Kubernetes workloads or in dedicated control-plane clusters. To work with the operator, we'll need to follow these steps. - [Overview](#overview) -- [Deploy the Pulumi Kubernetes Operator](#deploy-the-pulumi-kubernetes-operator) -- [Create a Stack CustomResource](#create-a-stack-customresource) - - [Using a git repository](#using-a-git-repository) +- [Install the Pulumi Kubernetes Operator](#install-the-pulumi-kubernetes-operator) + - [Using Helm](#using-helm) + - [Dev Install](#dev-install) +- [Create a Service Account](#create-a-service-account) +- [Configure Pulumi Cloud Access](#configure-pulumi-cloud-access) +- [Create a Stack Resource](#create-a-stack-resource) + - [Using a Git repository](#using-a-git-repository) - [Using a Flux source](#using-a-flux-source) - - [Using Argo CD](#using-argo-cd) - [Using a Program object](#using-a-program-object) - - [Stack Settings](#stack-settings) - - [Extended Examples](#extended-examples) -- [Concurrent Updates on the Same Stack](#concurrent-updates-on-the-same-stack) +- [Configure your Stack Resource](#configure-your-stack-resource) + - [Stack Configuration Values](#stack-configuration-values) + - [Environment Variables](#environment-variables) + - [Drift Detection](#drift-detection) + - [State Refresh](#state-refresh) + - [Stack Cleanup](#stack-cleanup) + - [Stack Prerequisites](#stack-prerequisites) +- [Use With Argo CD](#use-with-argo-cd) +- [More Information](#more-information) + - [Examples](#examples) + - [Getting Help](#getting-help) [k8s-ext-pattern]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ [stack]: /docs/concepts/stack/ +[Pulumi YAML]: /docs/iac/languages-sdks/yaml/ -## Deploy the Pulumi Kubernetes Operator +## Install the Pulumi Kubernetes Operator -The operator configuration is composed of: +### Using Helm -- A ServiceAccount for identity, -- A Role and RoleBinding to the ServiceAccount for RBAC, and -- A Deployment to manage the controller. +Use [Helm 3.x][helm] to install the Pulumi Kubernetes Operator into your cluster. -To create the operator, choose an installation preference using Pulumi -with a supported programming language or `kubectl` with YAML. +```bash +helm install --create-namespace -n pulumi-kubernetes-operator pulumi-kubernetes-operator \ + oci://ghcr.io/pulumi/helm-charts/pulumi-kubernetes-operator --version 2.0.0 +``` + +[helm]: https://helm.sh/ + +### Dev Install + +A simple "quickstart" installation manifest is provided for non-production environments. + +Install with `kubectl`: + +```bash +kubectl apply -f https://raw.githubusercontent.com/pulumi/pulumi-kubernetes-operator/refs/tags/v2.0.0/deploy/quickstart/install.yaml +``` + +Note: the installation manifest creates a usable Kubernetes service account named `default/pulumi` +for your convenience. + +## Create a Service Account + +The operator uses Kubernetes pods as the execution environment for Pulumi stack operations, +with each Stack having a dedicated pod. A pod service account is needed to serve as the stack's identity +and to authenticate users. + +Create a `ServiceAccount` named `default/pulumi` and grant the `system:auth-delegator` cluster role: + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + namespace: default + name: pulumi +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: default:pulumi:system:auth-delegator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:auth-delegator # permissions: TokenReview, SubjectAccessReview +subjects: +- kind: ServiceAccount + namespace: default + name: pulumi +``` + + If your Pulumi program uses the [Kubernetes Provider][] to manage resources within the cluster, the stack’s service account will need extra permissions, e.g. a `ClusterRoleBinding` to the `cluster-admin` cluster role. + + See [“Kubernetes: Service Accounts”][service-accounts] for more information. + +[Kubernetes Provider]: https://www.pulumi.com/registry/packages/kubernetes/ +[service-accounts]: https://kubernetes.io/docs/concepts/security/service-accounts/ + + +## Configure Pulumi Cloud Access -- [Installing the Operator with Pulumi in Typescript, Python, C#, and Go][use-pulumi] -- [Installing the Operator with kubectl][use-kubectl] +By default, the operator uses Pulumi Cloud as the state backend for your stacks. +Please create a `Secret` containing a Pulumi access token to be used to authenticate to Pulumi Cloud. Follow [these instructions][tokens] to create a personal, organization, or team access token. -[use-pulumi]: https://github.com/pulumi/pulumi-kubernetes-operator#using-pulumi -[use-kubectl]: https://github.com/pulumi/pulumi-kubernetes-operator#using-kubectl +Here’s an easy way to create a secret named `default/pulumi-api-secret`: + +```bash +kubectl create secret generic -n default pulumi-api-secret \ + --from-literal=accessToken=$PULUMI_ACCESS_TOKEN +``` + +In the Stack specification, use `spec.envRefs` to reference the secret: + +```yaml +spec: + envRefs: + PULUMI_ACCESS_TOKEN: + type: Secret + secret: + name: pulumi-api-secret + key: accessToken +``` + +To use a DIY state backend, set the `spec.backend` field to a storage endpoint URL. +Use `spec.envRefs` to attach credentials and to set environment variables for the backend as necessary. + +See ["States & Backends"][states-backends] for more information. + +[tokens]: https://www.pulumi.com/docs/pulumi-cloud/access-management/access-tokens/ +[states-backends]: https://www.pulumi.com/docs/iac/concepts/state-and-backends/ -When launched, the operator invokes the `StackController` to manages updates to -`Stack` CustomResources created, updated, or deleted in Kubernetes. -These updates are run in the form of reconciliation loops that attempt to update a Stack until success -is reached for the Git commit SHA provided, also known as the `desired state`. +## Create a Stack Resource -## Create a Stack CustomResource +The `Stack` Resource encapsulates a Pulumi project to provision infrastructure resources such as cloud VMs, object storage, and Kubernetes clusters and their workloads. -The `Stack` CustomResource (CR) encapsulates a Pulumi project that creates any set of -infrastructure resources such as cloud VMs, object storage, Kubernetes -clusters, or Kubernetes workloads through API resources. +Set the `spec.serviceAccountName` field to the name of a `ServiceAccount` with the requisite permissions. -### Using a git repository +Set the `spec.stack` field to a unique Pulumi stack name, using a [supported format][]. -In this scenario, the Stack points at an existing Git repo, and checks out the repo to deploy a `pulumi up`. The Stack configuration can specify a specific commit SHA or a reference to a branch or tag to track. If a commit is specified, the operator will try to reify the desired state of the stack in the commit until it succeeds. If a branch reference is specified, the Pulumi Kubernetes Operator will periodically poll the branch for any new commits and roll out updates as they are found. +[supported format]: https://www.pulumi.com/docs/iac/concepts/stacks/#create-stack -In the example below, we're creating a Stack for an existing Pulumi project that provisions -a simple [NGINX][nginx-stack] Deployment in Kubernetes. +### Using a Git repository -When the Stack is processed and deployed by the operator, NGINX will be created -in the same cluster as the operator. This is because the NGINX Pulumi program does not -explicitly use a [Kubernetes Provider resource][k8s-provider], and the Operator -makes its ServiceAccount credentials available to Stacks that rely on -the [default, ambient kubeconfig credentials][default-kubeconfig]. +In this scenario, the stack draws on a Git repository for the program source code. -The role permissions for the operator can be adjusted to control what in-cluster API resources are allowed. +The `Stack` specification can specify a commit SHA (`spec.commit`) or a branch reference (`spec.branch`). The repository URL is specified with `spec.projectRepo` plus an optional `spec.repoDir`. + +If a branch reference is specified, the operator will periodically poll the branch for any new commits +and roll out updates as they are found. Use the `spec.resyncFrequencySeconds` field to set the polling frequency. + +Specify Git authentication options with the `spec.gitAuth` field. + +In the example below, we're creating a `Stack` for a Pulumi project called `kubernetes-ts-nginx` to deploy a simple [NGINX][nginx-stack] server to your cluster. Without any configuration, the [Kubernetes Provider][k8s-provider] uses the in-cluster Kubernetes context. [nginx-stack]: https://github.com/pulumi/examples/blob/master/kubernetes-ts-nginx/index.ts [k8s-provider]: /registry/packages/kubernetes/api-docs/provider/ [default-kubeconfig]: /registry/packages/kubernetes/installation-configuration/#setup -Choose your preferred language below, or check out [Create Pulumi Stacks using kubectl][stacks-use-kubectl]. - {{< chooser language "typescript,python,go,csharp" >}} {{% choosable language typescript %}} ```typescript import * as pulumi from "@pulumi/pulumi"; import * as k8s from "@pulumi/kubernetes"; -import * as kx from "@pulumi/kubernetesx"; // Get the Pulumi API token. const pulumiConfig = new pulumi.Config(); const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken") // Create the API token as a Kubernetes Secret. -const accessToken = new kx.Secret("accesstoken", { +const accessToken = new k8s.core.v1.Secret("accessToken", { stringData: { accessToken: pulumiAccessToken }, }); @@ -121,6 +205,7 @@ const mystack = new k8s.apiextensions.CustomResource("my-stack", { apiVersion: 'pulumi.com/v1', kind: 'Stack', spec: { + serviceAccountName: "pulumi", envRefs: { PULUMI_ACCESS_TOKEN: { type: "Secret", @@ -132,9 +217,9 @@ const mystack = new k8s.apiextensions.CustomResource("my-stack", { }, stack: "/k8s-nginx/dev", projectRepo: "https://github.com/pulumi/examples", - repoDir: "kubernetes-ts-nginx", - commit: "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1", - // branch: "refs/heads/master", // Alternatively, track master branch. + repoDir: "kubernetes-ts-nginx/", + commit: "03658b5514f08970f350618a6e6fdf1bd75f45d0", + // branch: "master", // Alternatively, track master branch. destroyOnFinalize: true, } }); @@ -152,13 +237,14 @@ pulumi_config = pulumi.Config() pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken") # Create the API token as a Kubernetes Secret. -access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token }) +access_token = core.v1.Secret("accessToken", string_data={ "access_token": pulumi_access_token }) # Create an NGINX deployment in-cluster. my_stack = apiextensions.CustomResource("my-stack", api_version="pulumi.com/v1", kind="Stack", spec={ + "serviceAccountName": "pulumi", "envRefs": { "PULUMI_ACCESS_TOKEN": { "type": "Secret", @@ -170,9 +256,9 @@ my_stack = apiextensions.CustomResource("my-stack", }, "stack": "/k8s-nginx/dev", "projectRepo": "https://github.com/pulumi/examples", - "repoDir": "kubernetes-ts-nginx", - "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1", - # branch: "refs/heads/master", # Alternatively, track master branch. + "repoDir": "kubernetes-ts-nginx/", + "commit": "03658b5514f08970f350618a6e6fdf1bd75f45d0", + # branch: "master", # Alternatively, track master branch. "destroyOnFinalize": True, } ) @@ -199,6 +285,9 @@ class StackArgs : CustomResourceArgs class StackSpecArgs : ResourceArgs { + [Input("serviceAccountName")] + public Input? ServiceAccountName { get; set; } + [Input("accessTokenSecret")] public Input? AccessTokenSecret { get; set; } @@ -224,7 +313,7 @@ class MyStack : Stack var pulumiAccessToken = config.RequireSecret("pulumiAccessToken"); // Create the API token as a Kubernetes Secret. - var accessToken = new Secret("accesstoken", new SecretArgs + var accessToken = new Secret("accessToken", new SecretArgs { StringData = { @@ -237,13 +326,14 @@ class MyStack : Stack { Spec = new StackSpecArgs { + ServiceAccountName = "pulumi", AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name), Stack = "/k8s-nginx/dev", InitOnCreate = true, ProjectRepo = "https://github.com/pulumi/examples", - RepoDir = "kubernetes-ts-nginx", - Commit = "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1", - // branch: "refs/heads/master", // Alternatively, track master branch. + RepoDir = "kubernetes-ts-nginx/", + Commit = "03658b5514f08970f350618a6e6fdf1bd75f45d0", + // branch: "master", // Alternatively, track master branch. DestroyOnFinalize = true, } }); @@ -273,7 +363,7 @@ func main() { pulumiAccessToken := c.Require("pulumiAccessToken") // Create the API token as a Kubernetes Secret. - accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{ + accessToken, err := corev1.NewSecret(ctx, "accessToken", &corev1.SecretArgs{ StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)}, }) if err != nil { @@ -286,6 +376,7 @@ func main() { Kind: pulumi.String("Stack"), OtherFields: kubernetes.UntypedArgs{ "spec": map[string]interface{}{ + "serviceAccountName": "pulumi", "envRefs": pulumi.Map{ "PULUMI_ACCESS_TOKEN": pulumi.Map{ "type": pulumi.String("Secret"), @@ -297,9 +388,9 @@ func main() { }, "stack": "/k8s-nginx/dev", "projectRepo": "https://github.com/pulumi/examples", - "repoDir": "kubernetes-ts-nginx", - "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1", - // "branch": "refs/heads/master", // Alternatively, track master branch. + "repoDir": "kubernetes-ts-nginx/", + "commit": "03658b5514f08970f350618a6e6fdf1bd75f45d0", + // "branch": "master", // Alternatively, track master branch. "destroyOnFinalize": true, }, }, @@ -312,40 +403,40 @@ func main() { {{% /choosable %}} {{% /chooser %}} +G +### Using a Flux source -[stacks-use-kubectl]: https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/create-stacks-using-kubectl.md +[Flux][] offers a powerful alternative for fetching Pulumi program source code from +a variety of sources, including OCI repositories and cloud storage buckets. +Flux also supports some advanced Git options. Flux sources are specified as Custom Resources in a Kubernetes cluster; examples of sources are `GitRepository`, `OCIRepository`, and `Bucket` resources. -### Using a Flux source +To refer to a [Flux source object][flux-source], use the `spec.fluxSource` field. Use `spec.fluxSource.dir` to refer to a program directory within the source artifact. -To refer to a [Flux source][flux-source] rather than polling git directly, use the field -`.spec.fluxSource` and leave empty all of `.spec.projectRepo`, `.spec.commit`, `.spec.branch`, and -`.spec.gitAuth`. Here is the TypeScript example from above, adjusted to create a Flux source for the -git repo, then refer to it from the Stack object. The example assumes you have installed Flux into -the cluster beforehand. +Here is the TypeScript example from above, adjusted to create a Flux source for the +Git repo and then use it in the Stack specification. This example assumes you've already installed Flux into your cluster (see ["Flux installation"][flux-install]). ```typescript import * as pulumi from "@pulumi/pulumi"; import * as k8s from "@pulumi/kubernetes"; -import * as kx from "@pulumi/kubernetesx"; // Get the Pulumi API token. const pulumiConfig = new pulumi.Config(); const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken") // Create the API token as a Kubernetes Secret. -const accessToken = new kx.Secret("accesstoken", { +const accessToken = new k8s.core.v1.Secret("accessToken", { stringData: { accessToken: pulumiAccessToken }, }); // Create a GitRepository const gitrepo = new k8s.apiextensions.CustomResource("nginx-repo", { - apiVersion: "toolkit.source.fluxcd.io/v1beta2", + apiVersion: "source.toolkit.fluxcd.io/v1", kind: "GitRepository", metadata: {}, spec: { interval: '5m0s', url: "https://github.com/pulumi/examples", - ref: { commit: "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1" }, + ref: { commit: "03658b5514f08970f350618a6e6fdf1bd75f45d0" }, }, }); @@ -354,6 +445,7 @@ const mystack = new k8s.apiextensions.CustomResource("my-stack", { apiVersion: 'pulumi.com/v1', kind: 'Stack', spec: { + serviceAccountName: "pulumi", envRefs: { PULUMI_ACCESS_TOKEN: { type: "Secret", @@ -366,7 +458,7 @@ const mystack = new k8s.apiextensions.CustomResource("my-stack", { stack: "/k8s-nginx/dev", fluxSource: { sourceRef: { - apiVersion: "source.toolkit.fluxcd.io/v1beta2", + apiVersion: "source.toolkit.fluxcd.io/v1", kind: "GitRepository", name: gitrepo.metadata.name, }, @@ -376,65 +468,211 @@ const mystack = new k8s.apiextensions.CustomResource("my-stack", { }); ``` -### Using Argo CD +[flux]: https://fluxcd.io/ +[flux-install]: https://fluxcd.io/flux/installation/ -Much like Flux, we can use ArgoCD to manage the lifetime of the Stack via a GitOps paradigm. This will give us the advantage of using the ArgoCD UI or CLI to interact with the Stack, and allowing ArgoCD to reconcile changes to the repo. +### Using a Program object + +With the `Program` resource, you can define a Pulumi YAML program directly as a Kubernetes resource. +The reference docs for the [Program +Custom Resource][program-crd] details the wrapping; the [reference docs for Pulumi YAML][pulumi-yaml-ref] +gives all the fields that are part of the program code. + +[program-crd]: https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/programs.md +[pulumi-yaml-ref]: /docs/languages-sdks/yaml/yaml-language-reference/ + +Here is an example as a YAML manifest file: -First, we need to define a Pulumi stack as a Kubernetes manifest that ArgoCD can understand. We assume here that this manifest lives in the same repository as the Pulumi program, in the subfolder `deploy/`. However, this manifest could live in a separate repository, such as an "app-of-apps" repo. In this example, the manifest declares a service account and cluster role bindings to allow the stack to create resources in the cluster. Additionally, we expect a Secret to exist on the cluster containing a valid Pulumi access token +```yaml +--- +apiVersion: pulumi.com/v1 +kind: Program +metadata: + name: staticwebsite +program: + resources: + my-bucket: + type: aws:s3:BucketV2 + my-bucket-ownership-controls: + type: aws:s3:BucketOwnershipControls + properties: + bucket: ${my-bucket.id} + rule: + objectOwnership: ObjectWriter + my-bucket-acl: + type: aws:s3:BucketAclV2 + properties: + bucket: ${my-bucket.bucket} + acl: public-read + options: + dependsOn: + - ${my-bucket-ownership-controls} + my-bucket-public-access-block: + type: aws:s3:BucketPublicAccessBlock + properties: + bucket: ${my-bucket.id} + blockPublicAcls: false + my-bucket-website: + type: aws:s3:BucketWebsiteConfigurationV2 + properties: + bucket: ${my-bucket.bucket} + indexDocument: + suffix: index.html + index.html: + type: aws:s3:BucketObject + properties: + bucket: ${my-bucket} + source: + fn::stringAsset:

Hello, world!

+ acl: public-read + contentType: text/html + outputs: + bucketEndpoint: http://${my-bucket-website.websiteEndpoint} +``` -Note specifically that the Stack's `projectRepo` and `branch` point to the location of the Pulumi program to be executed by the Pulumi Kubernetes Operator. +You can then create a Stack object to deploy the program, by referring to it in the `spec.programRef` field: + +```yaml +--- +apiVersion: pulumi.com/v1 +kind: Stack +metadata: + name: staticwebsite +spec: + serviceAccountName: pulumi + stack: /staticwebsite/dev + programRef: + name: staticwebsite + destroyOnFinalize: true + config: + aws:region: us-east-1 +``` + + +## Configure your Stack Resource + +Here's some advanced options provided by the `Stack` resource. +Detailed documentation on the Stack API is available [here][pko-stacks]. + +[pko-stacks]: https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/stacks.md + +### Stack Configuration Values + +In many cases, different stacks for a single project will need differing values. +For instance, you may want to use a different size for your AWS EC2 instance, or a different number of replicas +for a particular Kubernetes deployment. Pulumi offers a configuration system for managing such differences; see ["Configuration"][iac-config] for more information. + +Use the `spec.config` block to set stack configuration values. The values are merged +into your project’s stack settings file. + +Use the `spec.secretsRef` block to set configuration values containing secrets. +The value may be a literal value or may be a reference to a Kubernetes `Secret`. + +Use the `spec.secretsProvider` field to use an alternative encryption provider. +See ["Initializing a stack with alternative encryption"][iac-secrets-provider] for more information. + +[iac-config]: https://www.pulumi.com/docs/iac/concepts/config/ + +[iac-secrets-provider]: https://www.pulumi.com/docs/intro/concepts/secrets/#initializing-a-stack-with-alternative-encryption + +### Environment Variables + +Use the `spec.envRefs` field to set environment variables for the Pulumi program, +such as `PULUMI_ACCESS_TOKEN` or `AWS_SECRET_ACCESS_KEY`. + +Values may be literals or based on the contents of a `ConfigMap` or `Secret` object. + +### Drift Detection + +Drift detection means to detect unwanted changes to your provisioned infrastructure. +The operator supports drift detection and remediation by periodically running `pulumi up`. This is referred to as re-synchronization. + +Use the `spec.continueResyncOnCommitMatch` field to enable periodic resyncs. Use the `spec.resyncFrequencySeconds` field to set the resync frequency. + +### State Refresh + +Use the `spec.refresh` field to refresh the state of the stack's resources before each update. + +{{< notes type="info" >}} + It is recommended that `spec.refresh` be enabled. +{{< /notes >}} + +### Stack Cleanup + +Use the `spec.destroyOnFinalize` field to automatically destroy the Pulumi stack (i.e. `pulumi destroy -f`) +when the `Stack` object is deleted. Enable this option to link the lifecycle of the Pulumi stack, and the resources it contains, to its `Stack` object. + +{{< notes type="info" >}} + Stack object deletion is slower when this option is enabled, because a Pulumi deployment operation + must be run during object finalization. +{{< /notes >}} + +### Stack Prerequisites + +It is possible to declare that a particular `Stack` be dependent on another `Stack`. +The dependent stack waits for the other stack to be successfully deployed. +Use the `succeededWithinDuration` field to set a duration within which the prerequisite must have reached success; otherwise the dependency is automatically re-synced. + + +## Use With Argo CD + +We can use ArgoCD in combination with PKO to manage the lifetime of the Stack via the GitOps paradigm. This gives you the ability to use the ArgoCD UI or CLI to interact with the Stack, and to allow ArgoCD to reconcile changes to the Stack specification. The Pulumi Kubernetes Operator handles the details. + +First, we need to define a Pulumi stack as a Kubernetes manifest that ArgoCD can deploy. We assume here that this manifest lives in the same repository as the Pulumi program, in the subfolder `deploy/`. However, this manifest could live in a separate repository, such as an "app-of-apps" repo. In this example, the manifest declares a service account and cluster role bindings to allow the stack to create resources in the cluster. Additionally, we expect a Secret to exist on the cluster containing a Pulumi access token. + +Note that the Stack's `projectRepo` and `branch` point to the location of the Pulumi program to be executed by the Pulumi Kubernetes Operator. ```yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: pulumi-application:system:auth-delegator + name: my-app:system:auth-delegator annotations: argocd.argoproj.io/sync-wave: "2" labels: - app.kubernetes.io/instance: pulumi-application + app.kubernetes.io/instance: my-app roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount - name: pulumi-application + name: my-app namespace: some-namepace --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: - name: pulumi-application:cluster-admin + name: my-app:cluster-admin annotations: argocd.argoproj.io/sync-wave: "2" labels: - app.kubernetes.io/instance: pulumi-application + app.kubernetes.io/instance: my-app roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount - name: pulumi-application + name: my-app namespace: some-namepace --- apiVersion: pulumi.com/v1 kind: Stack metadata: - name: pulumi-application-dev + name: my-app-dev namespace: some-namepace labels: - app.kubernetes.io/instance: pulumi-application + app.kubernetes.io/instance: my-app annotations: argocd.argoproj.io/sync-wave: "3" pulumi.com/reconciliation-request: "before-first-update" link.argocd.argoproj.io/external-link: http://app.pulumi.com/my-org/my-prject/dev spec: - serviceAccountName: pulumi-application - stack: my-org/my-prjoect/dev - projectRepo: "https://github.com/my-repo/my-pulumi-application.git" + serviceAccountName: my-app + stack: my-org/my-project/dev + projectRepo: "https://github.com/my-repo/my-app.git" branch: main refresh: true resyncFrequencySeconds: 60 @@ -447,19 +685,19 @@ spec: key: PULUMI_ACCESS_TOKEN workspaceTemplate: spec: - image: pulumi/pulumi:3.134.1-nonroot` + image: pulumi/pulumi:3.134.1-nonroot ``` -Next we instantiate an ArgoCD Application kind using the following manifest, `application.yaml`: +Next we create an ArgoCD `Application` object: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: - name: pulumi-application + name: my-app namespace: argocd finalizers: - # note: the finalizer must be background to allow Pulumi to invoke the shutdown of the workspace (pulumi down) + # best practice: use background cascading deletion when destroyOnFinalize is enabled. - resources-finalizer.argocd.argoproj.io/background spec: destination: @@ -470,138 +708,25 @@ spec: prune: true project: default source: - repoURL: "https://github.com/my-repo/my-pulumi-application.git" + repoURL: "https://github.com/my-repo/my-app.git" path: "./deploy" # the location of the Stack maifest targetRevision: main ``` -Deploy in the usual way `kubectl apply -f application.yaml -n default`. ArgoCD will take care of instantiation of the Stack object which will in turn invoke the PKO Workspace. The result will look something like this in the ArgoCD UI: - -![ArgoCD PKO Example](/images/docs/reference/argocd/pko-example.png) - -### Using a Program object - -It is also possible to supply a Pulumi YAML program directly as a Kubernetes resource. A Program -resource is a Pulumi YAML program, wrapped up as a Kubernetes object. The reference for the [Program -custom resource definition][] details the wrapping; the [reference for Pulumi YAML][] gives all the -fields that are part of the program itself. -[Program custom resource definition]: https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/programs.md -[reference for Pulumi YAML]: /docs/languages-sdks/yaml/yaml-language-reference/ +ArgoCD will sync the `Application` by applying the `Stack` object, +which will in turn effect a Pulumi deployment. The result will look something like this in the ArgoCD UI: -Here is an example as a YAML file: - -```yaml ---- -apiVersion: pulumi.com/v1 -kind: Program -metadata: - name: staticwebsite -program: - resources: - my-bucket: - type: aws:s3:BucketV2 - my-bucket-ownership-controls: - type: aws:s3:BucketOwnershipControls - properties: - bucket: ${my-bucket.id} - rule: - objectOwnership: ObjectWriter - my-bucket-acl: - type: aws:s3:BucketAclV2 - properties: - bucket: ${my-bucket.bucket} - acl: public-read - options: - dependsOn: - - ${my-bucket-ownership-controls} - my-bucket-public-access-block: - type: aws:s3:BucketPublicAccessBlock - properties: - bucket: ${my-bucket.id} - blockPublicAcls: false - my-bucket-website: - type: aws:s3:BucketWebsiteConfigurationV2 - properties: - bucket: ${my-bucket.bucket} - indexDocument: - suffix: index.html - index.html: - type: aws:s3:BucketObject - properties: - bucket: ${my-bucket} - source: - fn::stringAsset:

Hello, world!

- acl: public-read - contentType: text/html - outputs: - bucketEndpoint: http://${my-bucket-website.websiteEndpoint} -``` - -You can then create a Stack object to deploy the program, by referring to it in the field -`programRef`: - -```yaml ---- -apiVersion: pulumi.com/v1 -kind: Stack -metadata: - name: staticwebsite -spec: - stack: /staticwebsite/dev - programRef: - name: staticwebsite - destroyOnFinalize: true - config: - aws:region: us-east-1 -``` - -### Stack Settings - -Stack CustomResources provide the following properties to configure the Stack update run: - -- The first is the access token secret (`PULUMI_ACCESS_TOKEN`), which is required to authenticate with pulumi.com to - perform the update. You can create a new [Pulumi access token](/docs/pulumi-cloud/accounts#access-tokens) - specifically for your CI/CD job on your [Pulumi Account page](https://app.pulumi.com/account/tokens). -- Environment variables for the Stack that are sourced from Kubernetes ConfigMaps and/or Secrets. Examples include - cloud provider credentials and other application settings. -- Pulumi Stack configs and secrets that can complement or override settings - in the repo for use within the Stack. -- Project repo settings like the repo URL, the commit to deploy, and a repo - access token for private repos or rate-limiting. -- Lifecycle control such as creating the stack if it does not exist, - issuing a refresh before the update, and destroying the Stack's resources - and stack itself upon deletion of the CR. -- Switching to an open source backend. - -Detailed documentation on Stack CR configuration is available [here](https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/stacks.md). - -### Extended Examples - -Check out how to [manage a Kubernetes Blue/Green Deployment][blue-green-demo], -or how to create [AWS S3 buckets][aws-s3-demo] using the Operator and a Stack CR. - -You can watch a demo below for a complete walkthrough. - -{{< youtube "nQZr3uquc-c" >}} - -[blue-green-demo]: https://github.com/pulumi/pulumi-kubernetes-operator/tree/master/examples/blue-green -[aws-s3-demo]: https://github.com/pulumi/pulumi-kubernetes-operator/tree/master/examples/aws-s3 - -## Concurrent Updates on the Same Stack +![ArgoCD PKO Example](/images/docs/reference/argocd/pko-example.png) -Operators, by definition, will invoke a reconciliation loop for the creation, update, or deletion of a Stack CR. +## More Information -To avoid conflicting resource updates or corrupting resource state, Pulumi only -runs one update at a time per stack. By default, the operator checks for updates -already in progress, and will not spawn another reconciliation loop if one is already -running, blocking that stack. We strongly advice against running external updates on stacks controlled by the operator. +### Examples -You can optionally choose to retry on update conflicts by using the -`RetryOnUpdateConflict` field in the Stack. +More examples are available in the [pulumi/pulumi-kubernetes-operator](pko-examples) repository. -> Note: This is only recommended if you are sure that the stack updates are idempotent, and if you are willing to accept retry loops until -> all spawned retries succeed. This will also create a more populated, and randomized activity timeline for the stack in the Pulumi Cloud. +[pko-examples]: https://github.com/pulumi/pulumi-kubernetes-operator/tree/master/examples +### Getting Help Check out [troubleshooting](https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/troubleshooting.md) for more details, look at [known issues](https://github.com/pulumi/pulumi-kubernetes-operator/issues/) or open a [new issue](https://github.com/pulumi/pulumi-kubernetes-operator/issues/new) in GitHub. From 1b763b2116158b38df8476c048179c214127e36e Mon Sep 17 00:00:00 2001 From: Eron Wright Date: Thu, 20 Feb 2025 02:15:25 -0800 Subject: [PATCH 2/4] linting --- .../pulumi-kubernetes-operator.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md b/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md index b3bcafb249e3..65b05c3d9467 100644 --- a/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md +++ b/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md @@ -25,6 +25,7 @@ Operator](https://github.com/pulumi/pulumi-kubernetes-operator) (PKO) to automat ## Overview The Pulumi Kubernetes Operator provides [custom resources][k8s-ext-pattern] to: + - Provision a workspace (an execution environment) for a Pulumi project - Keep a Pulumi stack up-to-date using gitops - Write [Pulumi YAML][] programs as Kubernetes objects @@ -117,13 +118,12 @@ subjects: ``` If your Pulumi program uses the [Kubernetes Provider][] to manage resources within the cluster, the stack’s service account will need extra permissions, e.g. a `ClusterRoleBinding` to the `cluster-admin` cluster role. - + See [“Kubernetes: Service Accounts”][service-accounts] for more information. [Kubernetes Provider]: https://www.pulumi.com/registry/packages/kubernetes/ [service-accounts]: https://kubernetes.io/docs/concepts/security/service-accounts/ - ## Configure Pulumi Cloud Access By default, the operator uses Pulumi Cloud as the state backend for your stacks. @@ -156,7 +156,6 @@ See ["States & Backends"][states-backends] for more information. [tokens]: https://www.pulumi.com/docs/pulumi-cloud/access-management/access-tokens/ [states-backends]: https://www.pulumi.com/docs/iac/concepts/state-and-backends/ - ## Create a Stack Resource The `Stack` Resource encapsulates a Pulumi project to provision infrastructure resources such as cloud VMs, object storage, and Kubernetes clusters and their workloads. @@ -403,7 +402,7 @@ func main() { {{% /choosable %}} {{% /chooser %}} -G + ### Using a Flux source [Flux][] offers a powerful alternative for fetching Pulumi program source code from @@ -548,7 +547,6 @@ spec: aws:region: us-east-1 ``` - ## Configure your Stack Resource Here's some advanced options provided by the `Stack` resource. @@ -558,9 +556,10 @@ Detailed documentation on the Stack API is available [here][pko-stacks]. ### Stack Configuration Values -In many cases, different stacks for a single project will need differing values. -For instance, you may want to use a different size for your AWS EC2 instance, or a different number of replicas -for a particular Kubernetes deployment. Pulumi offers a configuration system for managing such differences; see ["Configuration"][iac-config] for more information. +In many cases, different stacks for a single project will need differing values. +For instance, you may want to use a different size for your AWS EC2 instance, or a different number of replicas +for a particular Kubernetes deployment. Pulumi offers a configuration system for managing such differences; +see ["Configuration"][iac-config] for more information. Use the `spec.config` block to set stack configuration values. The values are merged into your project’s stack settings file. @@ -613,7 +612,6 @@ It is possible to declare that a particular `Stack` be dependent on another `Sta The dependent stack waits for the other stack to be successfully deployed. Use the `succeededWithinDuration` field to set a duration within which the prerequisite must have reached success; otherwise the dependency is automatically re-synced. - ## Use With Argo CD We can use ArgoCD in combination with PKO to manage the lifetime of the Stack via the GitOps paradigm. This gives you the ability to use the ArgoCD UI or CLI to interact with the Stack, and to allow ArgoCD to reconcile changes to the Stack specification. The Pulumi Kubernetes Operator handles the details. @@ -713,7 +711,6 @@ spec: targetRevision: main ``` - ArgoCD will sync the `Application` by applying the `Stack` object, which will in turn effect a Pulumi deployment. The result will look something like this in the ArgoCD UI: @@ -728,5 +725,6 @@ More examples are available in the [pulumi/pulumi-kubernetes-operator](pko-examp [pko-examples]: https://github.com/pulumi/pulumi-kubernetes-operator/tree/master/examples ### Getting Help + Check out [troubleshooting](https://github.com/pulumi/pulumi-kubernetes-operator/blob/master/docs/troubleshooting.md) for more details, look at [known issues](https://github.com/pulumi/pulumi-kubernetes-operator/issues/) or open a [new issue](https://github.com/pulumi/pulumi-kubernetes-operator/issues/new) in GitHub. From e17e3e5e65b9c9af358de28f88045f20f2937704 Mon Sep 17 00:00:00 2001 From: Eron Wright Date: Thu, 20 Feb 2025 09:38:50 -0800 Subject: [PATCH 3/4] minor updates to the "preview" blog post --- .../blog/pulumi-kubernetes-operator-2-0/index.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/content/blog/pulumi-kubernetes-operator-2-0/index.md b/content/blog/pulumi-kubernetes-operator-2-0/index.md index bca8c187234b..a6058dd45c79 100644 --- a/content/blog/pulumi-kubernetes-operator-2-0/index.md +++ b/content/blog/pulumi-kubernetes-operator-2-0/index.md @@ -7,8 +7,10 @@ meta_desc: "Pulumi Kubernetes Operator 2.0: Horizontal Scaling, Multi-Tenancy" meta_image: operator.png --- +_Update: ["Pulumi Kubernetes Operator 2.0 is Now Generally Available!"](/blog/pko-2-0-ga/)_ + A few years ago we released the [Pulumi Kubernetes Operator](/blog/pulumi-kubernetes-operator-1-0/), a cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. We've heard your feedback about limitations related to scalability and isolation. -Today, we're excited to announce version [v2.0 (beta 2)](https://github.com/pulumi/pulumi-kubernetes-operator/releases/tag/v2.0.0-beta.2) of the Pulumi Kubernetes Operator. +Today, we're excited to announce a preview release of version [v2.0](https://github.com/pulumi/pulumi-kubernetes-operator/releases/tag/v2.0.0) of the Pulumi Kubernetes Operator. We've put a new, horizontally scalable architecture in place along with a variety of new security features and customization options. Let's dig in! @@ -59,7 +61,7 @@ We provide three ways to install the Operator: 1. a Pulumi program (see: `deploy/pulumi-operator-yaml/`) 2. a Helm chart (see: `deploy/helm/pulumi-operator/`) -3. a simple manifest file (see: `deploy/yaml/install.yaml`) +3. a simple manifest file for dev environments (see: `deploy/quickstart/install.yaml`) Please uninstall any earlier version of the Pulumi Kubernetes Operator, then install the new version using one of the above options. Note that this will remove any existing `Stack` objects in your cluster, so be sure to export their manifests or have @@ -89,9 +91,10 @@ fit the needs of your stack by using the `workspaceTemplate` field. If your Kube we recommend increasing the resource requests and limits. See ["Pod Quality of Service Classes"](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) for more information. -The pod is not discarded at the end of a stack operation, instead it remains in-place unless the stack is deleted. -We're considering introducing the option to clean up the workspace pod after each operation to enable you to make a trade-off -decision between performance and efficiency. Please [let us know](https://slack.pulumi.com) if this is something you'd like to see! +Once a stack reaches its desired state, the workspace +pod may be retained or deleted based on the `workspaceReclaimPolicy` field. The default behavior +is to retain the workspace for subsequent updates. This feature enables you to make a trade-off +decision between performance and efficiency. ### Stability @@ -186,6 +189,8 @@ See ["Pod and container logs"](https://kubernetes.io/docs/concepts/cluster-admin If you need to run an interactive Pulumi command for your stack, e.g. `pulumi import`, exec into the workspace pod. Navigate to the `/share/workspace` directory and you should find your program there. +Use the `spec.pulumiLogLevel` field to increase the log verbosity level of the Pulumi CLI. + ## Walkthrough Here's a quick demonstration of the new system. Let's deploy the [random-yaml](https://github.com/pulumi/examples/tree/master/random-yaml) program from the [pulumi/examples](https://github.com/pulumi/examples) repository. From 74d271526df923df7c5a52e4ba87a0daf5f7188e Mon Sep 17 00:00:00 2001 From: Eron Wright Date: Thu, 20 Feb 2025 10:10:20 -0800 Subject: [PATCH 4/4] review feedback --- .../pulumi-kubernetes-operator-2-0/index.md | 2 +- .../pulumi-kubernetes-operator.md | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/content/blog/pulumi-kubernetes-operator-2-0/index.md b/content/blog/pulumi-kubernetes-operator-2-0/index.md index a6058dd45c79..7f40798b0bb1 100644 --- a/content/blog/pulumi-kubernetes-operator-2-0/index.md +++ b/content/blog/pulumi-kubernetes-operator-2-0/index.md @@ -218,7 +218,7 @@ spec: ### Install a Pulumi Access Token Create a `Secret` containing a Pulumi access token to be used by the stack to authenticate to Pulumi Cloud. -Follow [these instructions](https://www.pulumi.com/docs/pulumi-cloud/access-management/access-tokens/) to create a +Follow [these instructions](/docs/pulumi-cloud/access-management/access-tokens/) to create a personal, organization, or team access token. Store the access token into a Kubernetes Secret. Here's an easy way to create a Secret named `pulumi-api-secret`. diff --git a/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md b/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md index 65b05c3d9467..51b22738e7b9 100644 --- a/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md +++ b/content/docs/iac/using-pulumi/continuous-delivery/pulumi-kubernetes-operator.md @@ -46,13 +46,14 @@ To work with the operator, we'll need to follow these steps. - [Using a Git repository](#using-a-git-repository) - [Using a Flux source](#using-a-flux-source) - [Using a Program object](#using-a-program-object) -- [Configure your Stack Resource](#configure-your-stack-resource) +- [Explore other Features](#explore-other-features) - [Stack Configuration Values](#stack-configuration-values) - [Environment Variables](#environment-variables) - [Drift Detection](#drift-detection) - [State Refresh](#state-refresh) - [Stack Cleanup](#stack-cleanup) - [Stack Prerequisites](#stack-prerequisites) + - [External Triggers](#external-triggers) - [Use With Argo CD](#use-with-argo-cd) - [More Information](#more-information) - [Examples](#examples) @@ -547,7 +548,7 @@ spec: aws:region: us-east-1 ``` -## Configure your Stack Resource +## Explore other Features Here's some advanced options provided by the `Stack` resource. Detailed documentation on the Stack API is available [here][pko-stacks]. @@ -612,6 +613,17 @@ It is possible to declare that a particular `Stack` be dependent on another `Sta The dependent stack waits for the other stack to be successfully deployed. Use the `succeededWithinDuration` field to set a duration within which the prerequisite must have reached success; otherwise the dependency is automatically re-synced. +### External Triggers + +It is possible to trigger a stack update for a stack at any time by applying +the `pulumi.com/reconciliation-request` annotation: + +```bash +kubectl annotate stack $STACK_NAME "pulumi.com/reconciliation-request=$(date)" --overwrite +``` + +The value of the annotation is arbitrary, and we recommend using a timestamp. + ## Use With Argo CD We can use ArgoCD in combination with PKO to manage the lifetime of the Stack via the GitOps paradigm. This gives you the ability to use the ArgoCD UI or CLI to interact with the Stack, and to allow ArgoCD to reconcile changes to the Stack specification. The Pulumi Kubernetes Operator handles the details. @@ -720,7 +732,7 @@ which will in turn effect a Pulumi deployment. The result will look something li ### Examples -More examples are available in the [pulumi/pulumi-kubernetes-operator](pko-examples) repository. +More examples are available in the [pulumi/pulumi-kubernetes-operator][pko-examples] repository. [pko-examples]: https://github.com/pulumi/pulumi-kubernetes-operator/tree/master/examples