checking kubernetes:yaml:ConfigGroup resources

[errordeveloper]

I am creating a GKE cluser and using kubernetes:yaml:ConfigGroup to deploy some workload to the cluster.

Before I create any resources at all, my policy that looks for certain specific resources doesn't work, albeit it does later on, if cluster was already deployed.

So, in other words, when preview output look like this, it doesn't work:

     Type                                                               Name                                        Status              Info
     pulumi:pulumi:Stack                                                ilya-dev                      (668s)             2 messages
 +   ├─ gcp:container:Cluster                                           ilya-dev-1                              created (666s)      
 +   ├─ gcp:container:NodePool                                          ilya-dev-1-np-0                         created (111s)      
 +   ├─ pulumi:providers:kubernetes                                     gke-provider                                created (0.40s)     
     └─ kubernetes:yaml:ConfigGroup                                     flux-install                                 (0.00s)       

But it works when the kubernetes:yaml:ConfigGroup resource has dependents in the preview output:

     pulumi:pulumi:Stack                                                ilya-dev                      (668s)             2 messages
 +   ├─ gcp:container:Cluster                                           ilya-dev-1                              created (666s)      
 +   ├─ gcp:container:NodePool                                          ilya-dev-1-np-0                         created (111s)      
 +   ├─ pulumi:providers:kubernetes                                     gke-provider                                created (0.40s)     
     └─ kubernetes:yaml:ConfigGroup                                     flux-install                                 (0.00s)            
 +      ├─ kubernetes:core/v1:ServiceAccount                            flux-system/helm-controller                 created (3s)        
 +      ├─ kubernetes:core/v1:Namespace                                 flux-system                                 created (0.58s)     
 +      ├─ kubernetes:core/v1:ServiceAccount                            flux-system/kustomize-controller            created (1s)        
 +      ├─ kubernetes:core/v1:Service                                   flux-system/notification-controller         created (131s)      
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                flux-system/allow-scraping                  created (1s)        
 +      ├─ kubernetes:core/v1:ServiceAccount                            flux-system/notification-controller         created (1s)        
 +      ├─ kubernetes:core/v1:ServiceAccount                            flux-system/source-controller               created (2s)        
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                flux-system/allow-webhooks                  created (3s)        
 +      ├─ kubernetes:apps/v1:Deployment                                flux-system/notification-controller         created (166s)      
 +      ├─ kubernetes:core/v1:Service                                   flux-system/source-controller               created (149s)      
 +      ├─ kubernetes:apps/v1:Deployment                                flux-system/helm-controller                 created (166s)      
 +      ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy                flux-system/allow-egress                    created (6s)        
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole          crd-controller-flux-system                  created (7s)        
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  receivers.notification.toolkit.fluxcd.io    created (8s)        
 +      ├─ kubernetes:apps/v1:Deployment                                flux-system/kustomize-controller            created (165s)      
 +      ├─ kubernetes:apps/v1:Deployment                                flux-system/source-controller               created (168s)      
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding   cluster-reconciler-flux-system              created (11s)       
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  alerts.notification.toolkit.fluxcd.io       created (11s)       
 +      ├─ kubernetes:core/v1:Service                                   flux-system/webhook-receiver                created (136s)      
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  ocirepositories.source.toolkit.fluxcd.io    created (12s)       
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding   crd-controller-flux-system                  created (12s)       
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  providers.notification.toolkit.fluxcd.io    created (12s)       
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  helmrepositories.source.toolkit.fluxcd.io   created (12s)       
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  buckets.source.toolkit.fluxcd.io            created (13s)       
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  helmcharts.source.toolkit.fluxcd.io         created (13s)       
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  gitrepositories.source.toolkit.fluxcd.io    created (13s)       
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  helmreleases.helm.toolkit.fluxcd.io         created (13s)       
 +      └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition  kustomizations.kustomize.toolkit.fluxcd.io  created (13s) 

It will take some effort to make all of the code sharable, but here is what the policy definition looks like at the moment:

import * as gcp from "@pulumi/gcp";
import * as policy from "@pulumi/policy";

import * as kube from "@pulumi/kubernetes";

const stackPolicy: policy.StackValidationPolicy = {
    name: "validate-stack-resources",
    description: "Validate the stack contains correct resources.",
    enforcementLevel: "mandatory",
    validateStack: async(args: policy.StackValidationArgs, fail: policy.ReportViolation) => {
        let networks = args.resources.filter(r => r.isType(gcp.compute.Network));
        if (networks.length != 1) {
            fail(`expected one network, found ${networks.length}`)
        }

        let subnetworks = args.resources.filter(r => r.isType(gcp.compute.Subnetwork));
        if (subnetworks.length != 1) {
            fail(`expected one subnetwork, found ${networks.length}`)
        }

        const network = networks[0].asType(gcp.compute.Network);
        const subnetwork = subnetworks[0].asType(gcp.compute.Subnetwork);

        if (network && subnetwork) {
            if (subnetwork.network != network.selfLink) {
                fail("subnetwork is linked to an unexpected network")
            }
        } else {
            fail("missing network or subnetwork")
        }

        let clusters = args.resources.filter(r => r.isType(gcp.container.Cluster));
        if (clusters.length !== 1) {
            fail(`expected one GKE Cluster but found ${clusters.length}`);
        }

        const cluster = clusters[0].asType(gcp.container.Cluster);
        if (cluster) {
            if (cluster.network && network) {
                if (!network.selfLink.endsWith(cluster.network)) {
                    fail("cluster is linked to an unexpected network: "+`${cluster.network} != ${network.selfLink}`)
                }
            }

            let addons = cluster.addonsConfig;
            if (!addons.dnsCacheConfig.enabled) {
                fail("DNS cache addon should be enabled")
            }
            if (addons.horizontalPodAutoscaling.disabled) {
                fail("HPA addon should be enabled")
            }
            if (!addons.httpLoadBalancing.disabled) {
                fail("HTTP LB should be disabled")
            }
            if (!addons.configConnectorConfig.enabled) {
                fail("Config Connector should be enabled")
            }

            if (cluster.datapathProvider != "ADVANCED_DATAPATH") {
                fail("datapath provider should be 'ADVANCED_DATAPATH'")
            }

            if (cluster.podSecurityPolicyConfig?.enabled) {
                fail("PSP should be disabled")
            }

            if (cluster.privateClusterConfig.enablePrivateEndpoint) {
                fail("private endpoints should be disabled")
            }

            if (!cluster.privateClusterConfig.enablePrivateNodes) {
                fail("private ndos shoudl be enabled")
            }

            if (!cluster.removeDefaultNodePool) {
                fail("default node pool should be disabled")
            }
        }

        let fluxResourcesStats = new Map<string, number>(Object.entries({
            "kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition": 10,
            "kubernetes:apps/v1:Deployment": 4,
            "kubernetes:core/v1:Namespace": 1,
            "kubernetes:core/v1:Service": 3,
            "kubernetes:core/v1:ServiceAccount": 4,
            "kubernetes:networking.k8s.io/v1:NetworkPolicy": 3,
            "kubernetes:rbac.authorization.k8s.io/v1:ClusterRole": 1,
            "kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding": 2,
        }));

        let fluxConfigGroup = args.resources.filter(r => r.isType(kube.yaml.ConfigGroup))

        if (!fluxConfigGroup) {
            let fluxResources = args.resources.filter(r => (r.parent?.name == "flux-install"))
            let fluxResourceCount = 0;
            for (let [k, v] of fluxResourcesStats) {
                let resources = fluxResources.filter(r => (r.type == k))
                fluxResourceCount += resources.length
                if (resources.length != v) {
                    fail(`unexpected number of flux resources of type ${k}: ${resources.length} != ${v}`)
                }
            }
            if (fluxResourceCount != 28) {
                fail(`unexpected total number of flux resources: ${fluxResourceCount}`)
            }
        } else {
            if (fluxConfigGroup.length != 1) {
                fail(`config group length: ${fluxConfigGroup.length}`)
            }
        }
    },
};

const tests = new policy.PolicyPack("infra-policy", {
    policies: [stackPolicy],
});

It would make some sense if I could at least get the group resource itself, which I'm attempting to this way:

        let fluxConfigGroup = args.resources.filter(r => r.isType(kube.yaml.ConfigGroup))

But it doesn't seem to be present at all.

[RobbieMcKinstry]

Hello @errordeveloper thanks for opening this. Could you share the output of pulumi about in the comments?

Odd...I would expect r => r.isType(kube.yaml.ConfigGroup) to have at least one member. I wish I had more to offer at this time.

For posterity:

• isType docs
• StackValidationArgs docs

[errordeveloper]

Thanks for getting back to me @RobbieMcKinstry!

I've made a few changes and use load manifests dirctly from files instead of using a package.

$ pulumi about --stack dev2                      
CLI          
Version      3.43.1
Go Version   go1.19.2
Go Compiler  gc

Plugins
NAME        VERSION
gcp         6.40.0
gcp         6.40.0
kubernetes  3.21.4
kubernetes  3.21.4
nodejs      unknown
tls         4.6.1

Host     
OS       darwin
Version  12.6
Arch     arm64

This project is written in nodejs: executable='/Users/ilya/Library/Local/Homebrew/bin/node' version='v18.10.0'

Current Stack: dev2

Found no resources associated with dev2

Found no pending operations associated with dev2

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/errordeveloper
User           errordeveloper
Organizations  errordeveloper, atomisthq

Dependencies:
NAME                      VERSION
jest                      29.1.2
ts-node                   10.9.1
@pulumi/gcp               6.40.0
@pulumi/kubernetes        3.21.4
@pulumi/policy            1.5.0
@pulumi/tls               4.6.1
@types/jest               29.1.2
@babel/preset-env         7.19.4
@babel/preset-typescript  7.18.6
@pulumi/pulumi            3.42.0
yaml                      2.1.3

Pulumi locates its logs in /var/folders/gm/dnc405js6nlb275l4v2gh_b40000gn/T/ by default
$ pulumi up --stack dev2 --policy-pack policy
Previewing update (dev2)

View Live: https://app.pulumi.com/errordeveloper/image-build-service/dev2/previews/c15eaa03-60ff-4541-9db9-0195f2c5e148

     Type                              Name                                                   Plan       Info
 +   pulumi:pulumi:Stack               image-build-service-dev2                               create     1 error
 +   ├─ kubernetes:yaml:ConfigGroup    flux-install                                           create     
 +   │  ├─ kubernetes:yaml:ConfigFile  ../configs/shared/flux/flux.yaml                       create     1 message
 +   │  └─ kubernetes:yaml:ConfigFile  ../configs/shared/flux/sources.yaml                    create     1 message
 +   ├─ gcp:projects:Service           logging-service                                        create     
 +   ├─ gcp:projects:Service           iam-service                                            create     
 +   ├─ gcp:projects:Service           monitoring-service                                     create     
 +   ├─ gcp:projects:Service           cloudkms-service                                       create     
 +   ├─ tls:index:PrivateKey           flux-image-build-service                               create     
 +   ├─ gcp:projects:Service           storage-api-service                                    create     
 +   ├─ gcp:projects:Service           compute-service                                        create     
 +   ├─ gcp:projects:Service           container-service                                      create     
 +   ├─ gcp:projects:Service           iamcredentials-service                                 create     
 +   ├─ gcp:projects:Service           storage-component-service                              create     
 +   ├─ gcp:projects:Service           cloudresourcemanager-service                           create     
 +   ├─ gcp:compute:Network            ilya-ibs-dev-2-network                                 create     
 +   ├─ gcp:serviceAccount:Account     ilya-ibs-dev-2-node-sa                                 create     
 +   ├─ gcp:compute:Router             ilya-ibs-dev-2-router                                  create     
 +   ├─ gcp:compute:Subnetwork         ilya-ibs-dev-2-subnetwork                              create     
 +   ├─ gcp:projects:IAMMember         monitoring-metricwriter-ilya-ibs-dev-2-node-sa-member  create     
 +   ├─ gcp:projects:IAMMember         monitoring-viewer-ilya-ibs-dev-2-node-sa-member        create     
 +   ├─ gcp:projects:IAMMember         logging-logwriter-ilya-ibs-dev-2-node-sa-member        create     
 +   ├─ gcp:compute:RouterNat          ilya-ibs-dev-2-router-nat                              create     
 +   ├─ gcp:container:Cluster          ilya-ibs-dev-2                                         create     
 +   ├─ pulumi:providers:kubernetes    gke-provider                                           create     
 +   └─ gcp:container:NodePool         ilya-ibs-dev-2-np-0                                    create     
 
Diagnostics:
  kubernetes:yaml:ConfigFile (../configs/shared/flux/sources.yaml):
    Can't decode yaml config when provider is not fully initialized. This can result in empty previews but should resolve correctly during apply.
 
  pulumi:pulumi:Stack (image-build-service-dev2):
    error: preview failed
 
  kubernetes:yaml:ConfigFile (../configs/shared/flux/flux.yaml):
    Can't decode yaml config when provider is not fully initialized. This can result in empty previews but should resolve correctly during apply.
 
Policy Violations:
    [mandatory]  build-service-infra-policy v0.0.1  validate-stack-resources (pulumi:pulumi:Stack: image-build-service-dev2)
    Validate the stack contains correct resources.
    config group length: 0
    
    [mandatory]  build-service-infra-policy v0.0.1  validate-stack-resources (pulumi:pulumi:Stack: image-build-service-dev2)
    Validate the stack contains correct resources.
    config files length: 0
    
Outputs:
    cluster           : "ilya-ibs-dev-2-4bc8a11"
    fluxPublicKey     : output<string>
    nodeServiceAccount: output<string>
    project           : "sandbox-298914"
    region            : "europe-west2"
$

So now there also Can't decode yaml config when provider is not fully initialized warnings, which is probably to do with CRDs and CRs.
I understand that some things get a bit complex to do with CRDs and CRs... but I still think the object should be visible to the policy.

Here is relevant policy logic:

        let fluxConfigGroup = args.resources.filter(r => r.isType(kube.yaml.ConfigGroup))
        if (fluxConfigGroup.length == 0) {
            fail(`config group length: ${fluxConfigGroup.length}`)
        }

        let fluxConfigFiles = args.resources.filter(r => r.isType(kube.yaml.ConfigFile))
        if (fluxConfigFiles.length == 0) {
            fail(`config files length: ${fluxConfigFiles.length}`)
        }

I also added one kube.core.v1.Secret, but this is dependenat on a private key, so I have this in my code:

const fluxPrivateKey = new PrivateKey("flux-image-build-service", { algorithm: "RSA" });

	let fluxPrivateKeySecret: kube.core.v1.Secret;

	fluxPrivateKey.privateKeyOpenssh.apply(key => {
		fluxPrivateKeySecret = new kube.core.v1.Secret("flux-image-build-service", {
			metadata: {
				// name must be set here to avoid Pulumi auto-naming, as otherwise
				// secret name would have to be passed to a manifest that is currently
				// static (albeit transform could be used for this, it seems unnecessary)
				name: "flux-image-build-service",
				namespace: "flux-system",
			},
			stringData: {
				identity: key.trimEnd(),
			 },
		}, kubeOpts);

I'm trying to write policy logic for this also, and it seems like I'm hitting another bump, probably to do with that the resource is defoined in the promise callback... What happens is that args.resources.filter(r => r.isType(kube.core.v1.Secret)) returns array of one element, but it's undefined.

I guess this leave me wondering wheather I'm really hitting the limitations of how policy is implemented at the moment?