crossplane with k3d; install grafana and temporal

README.md

@@ -1,10 +1,73 @@
-# k3d Cluster
+# Crossplane Demo
 
-## Create
+## Regenerate Certs (Optional) 
 ```
-sudo k3d cluster create crossplane-demo
-sudo k3d kubeconfig merge crossplane-demo --kubeconfig-switch-context -o /home/<username>/.kube/config
-sudo chown <username>:<username> ~/.kube/config 
+./registry/certs/generateCerts.sh
+```
+
+## Create k3d Cluster
+```
+sudo k3d cluster create crossplane-demo \
+--volume "$(pwd)/registry/registries.yaml:/etc/rancher/k3s/registries.yaml" \
+--volume "$(pwd)/registry/certs/self-signed-ca.crt:/etc/ssl/certs/self-signed-ca.crt" \
+-p "8081:80@loadbalancer" // not tested yet
+
+sudo k3d kubeconfig merge crossplane-demo --kubeconfig-switch-context -o ~/.kube/config
+
+USERNAME=$(whoami) && sudo chown ${USERNAME}:${USERNAME} ~/.kube/config 
+```
+
+## DNS
+Many Linux distributions ships with NSS-myhostname, which resolves *.localhost
+
+If not install `nss-myhostname` 
+```
+sudo apt install libnss-myhostname
+```
+
+If not possbile you have to add /etc/hots entries pointing to 127.0.0.1 manually
+
+## Selfhosted Registry
+We can not utilize K3d's build-in registry (`--create-registry`), because we need SSL.
+Furthermore we can also not use build-in image upload mechanism (`k3d image import`), because crossplane needs an OCI compatible registry to pull from.
+
+
+Start selfhosted registry 
+```
+sudo docker-compose up -d [--build]
+```
+
+Connect Cluster to selfhosted registry 
+```
+sudo docker network connect k3d-crossplane-demo registry.k3d.localhost
+```
+
+Modify coredns configmap and add entry: 
+```
+sudo docker inspect registry.k3d.localhost | jq -r '.[0].NetworkSettings.Networks."k3d-crossplane-demo".IPAddress'
+```
+
+<IpOfRegistry> registry.k3d.localhost
+
+retstart codedns pod
+
+### Push image
+```
+sudo docker tag <image>:<tag> crossplane-demo-registry.local:<port>/<image>:<tag>
+sudo docker push crossplane-demo-registry.local:<port>/<image>:<tag>
+```
+
+### Push *.xpkg file 
+
+Start container with crossplane cli + trusted self signed cert
+```
+sudo docker build -t "crossplane-cli:latest" -f ./registry/Dockerfile.crossplane-cli ./registry
+sudo docker run --rm -it --net=host -v $(pwd)/registry/files:/files crossplane-cli:latest bash
+```
+
+push file to OCI registry
+```
+crossplane xpkg push -f /files/<file> registry.k3d.localhost:5000/<name>:<tag>
 ```
 
 ## Start and Stop
@@ -17,6 +80,12 @@ sudo k3d cluster start crossplane-demo
 https://docs.crossplane.io/latest/software/install/
 
 ## Install via Helm
+Create configmap to trust private registry with selfsigned cert
+```
+kubectl -n crossplane-system create cm ca-bundle-config \
+--from-file=ca-bundle=./registry/certs/self-signed-ca.crt
+```
+
 ```
 helm repo add crossplane-stable https://charts.crossplane.io/stable
 helm repo update
@@ -25,7 +94,8 @@ helm repo update
 ```
 helm install crossplane \
 --namespace crossplane-system \
---create-namespace crossplane-stable/crossplane 
+--create-namespace crossplane-stable/crossplane \
+--set registryCaBundleConfig.name=ca-bundle-config,registryCaBundleConfig.key=ca-bundle
 ```
 
 # Crossplane UI
@@ -34,15 +104,11 @@ helm install crossplane \
 ```
 helm repo add komodorio https://helm-charts.komodor.io 
 helm repo update  
-helm install komoplane komodorio/komoplane --set "ingress.enabled=true,ingress.hosts[0].host=komoplane.k8s.local,ingress.hosts[0].paths[0].path=/,ingress.hosts[0].paths[0].pathType=ImplementationSpecific"
-```
-
-Add entry to /etc/hosts
-```
-<LoadBalancerIP> komoplane.k8s.local
+helm install komoplane komodorio/komoplane \
+--set "ingress.enabled=true,ingress.hosts[0].host=komoplane.k8s.localhost,ingress.hosts[0].paths[0].path=/,ingress.hosts[0].paths[0].pathType=ImplementationSpecific"
 ```
 
-Crossplane UI URL: http://komoplane.k8s.local/
+Crossplane UI URL: http://komoplane.k8s.localhost/
 
 # Testing with Grafana
 ## Install Grafana via Helm
@@ -52,24 +118,70 @@ https://github.com/grafana/helm-charts/blob/main/charts/grafana/README.md
 ```
 helm repo add grafana https://grafana.github.io/helm-charts
 helm repo update
-helm install my-release grafana/grafana --set "adminUser=admin,adminPassword=password,ingress.enabled=true,ingress.hosts[0]=grafana.k8s.local"
+helm install my-release grafana/grafana --set "adminUser=admin,adminPassword=password,ingress.enabled=true,ingress.hosts[0]=grafana.k8s.localhost"
 ```
 
-Add entry to /etc/hosts
+## Install Crossplane Grafana Provider
+[Grafana Provider](https://marketplace.upbound.io/providers/grafana/provider-grafana/v0.8.0) was generated by [upjet](https://github.com/upbound/upjet).
+
+Therefore Provider configuration (Kind:ProviderConfig.spec.credentials) can be inferred by:
+https://registry.terraform.io/providers/grafana/grafana/latest/docs#schema
+
 ```
-<LoadBalancerIP> grafana.k8s.local
+kubectl apply -f ./k8s/crossplane/grafana
 ```
 
-Grafana URL: http://grafana.k8s.local/
+## Check
 
-## Install Crossplane Grafana Provider
-[Grafana Provider](https://marketplace.upbound.io/providers/grafana/provider-grafana/v0.8.0) was generated by [upjet](https://github.com/upbound/upjet).
+Grafana URL: http://grafana.k8s.localhost/
 
-Therefore Provider configuration can be inferred by:
-https://registry.terraform.io/providers/grafana/grafana/latest/docs#schema
 
+# Test with Temporal
+## Install Temporal via Helm
+```
+git clone https://github.com/temporalio/helm-charts
+cd helm-charts
+
+helm dependencies update
+
+helm install --set "server.replicaCount=1,cassandra.config.cluster_size=1,prometheus.enabled=false,grafana.enabled=false,elasticsearch.enabled=false,web.ingress.enabled=true,web.ingress.hosts[0]=temporal.k8s.localhost,server.frontend.service.type=LoadBalancer" temporaltest . --timeout 15m
+```
+
+## Push *.xpkg file 
+Start container with crossplane cli + trusted self signed cert
+```
+sudo docker build -t "crossplane-cli:latest" -f ./registry/Dockerfile.crossplane-cli ./registry
+sudo docker run --rm -it --net=host -v $(pwd)/registry/files:/files crossplane-cli:latest bash
+```
 
+push file to OCI registry (The file was built with `make build` in source repo)
 ```
-kubectl apply -f ./k8s/crossplane
+crossplane xpkg push -f /files/provider-temporal-v0.0.0-2.g3abd9d5.dirty.xpkg registry.k3d.localhost:5000/provider-temporal:v13.0.0
 ```
 
+## Install Crossplane Temporal Provider
+```
+kubectl apply -f ./k8s/crossplane/grafana
+```
+
+## Check
+
+Temporal URL: http://temporal.k8s.localhost/
+
+Query namespaces with CLI
+```
+temporal operator namespace list --address temporal.k8s.localhost:7233
+```
+
+## Install Crossplane Temporal Provider
+```
+git clone https://github.com/denniskniep/provider-temporal.git
+make build
+```
+
+push the created xpkg file to oci registry:
+see section `Push *.xpkg file`
+
+```
+kubectl apply -f ./k8s/crossplane/temporal
+```

docker-compose.yaml

@@ -0,0 +1,17 @@
+version: "3.8"
+services:
+  registry.k3d.localhost:
+    container_name: registry.k3d.localhost    
+    image: registry:2
+    environment:
+      - REGISTRY_HTTP_ADDR=0.0.0.0:5000
+      - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/self-signed-ca.crt
+      - REGISTRY_HTTP_TLS_KEY=/certs/self-signed-ca.key      
+    volumes:
+      - ./registry/certs:/certs
+      - registry-data:/var/lib/registry
+    ports:
+      - 5000:5000
+
+volumes:
+  registry-data:      

k8s/crossplane/01-grafana-provider.yaml → k8s/crossplane/grafana/01-grafana-provider.yaml

@@ -3,6 +3,6 @@ kind: Provider
 metadata:
   name: provider-grafana
 spec:
-  package: xpkg.upbound.io/grafana/provider-grafana:v0.8.0
-  packagePullPolicy: IfNotPresent
+  package: grafana/provider-grafana:v0.8.0
+  packagePullPolicy: Never
   revisionActivationPolicy: Automatic

k8s/crossplane/temporal/01-temporal-provider.yaml

@@ -0,0 +1,25 @@
+apiVersion: pkg.crossplane.io/v1beta1
+kind: DeploymentRuntimeConfig
+metadata:
+  name: debug-config
+spec:
+  deploymentTemplate:
+    spec:
+      selector: {}
+      template:
+        spec:
+          containers:
+            - name: package-runtime
+              args:
+                - --debug
+---
+apiVersion: pkg.crossplane.io/v1
+kind: Provider
+metadata:
+  name: provider-temporal
+spec:
+  package: registry.k3d.localhost:5000/provider-temporal:v13.0.0
+  packagePullPolicy: IfNotPresent
+  revisionActivationPolicy: Automatic
+  runtimeConfigRef:
+    name: debug-config

k8s/crossplane/temporal/02-temporal-provider-config-secret.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: provider-temporal-config-creds
+  namespace: crossplane-system
+type: Opaque
+stringData:
+  credentials: |
+    {
+      "HostPort": "temporaltest-frontend.default:7233"
+    }

k8s/crossplane/temporal/03-temporal-provider-config.yaml

@@ -0,0 +1,11 @@
+apiVersion: temporal.crossplane.io/v1alpha1
+kind: ProviderConfig
+metadata:
+  name: local-temporal-instance-config
+spec: 
+  credentials:
+    source: Secret
+    secretRef:
+      namespace: crossplane-system
+      name: provider-temporal-config-creds
+      key: credentials  

k8s/crossplane/temporal/04-temporal-namespace-resources.yaml

@@ -0,0 +1,11 @@
+apiVersion: core.temporal.crossplane.io/v1alpha1
+kind: TemporalNamespace
+metadata:
+  name: ns1
+spec:
+  forProvider:
+    name: "Test 1"
+    description: "Test Desc 1"
+    ownerEmail: "[email protected]"
+  providerConfigRef:
+    name: local-temporal-instance-config

registry/Dockerfile.crossplane-cli

@@ -0,0 +1,15 @@
+FROM ubuntu:22.04
+
+COPY ./certs/ /certs
+
+RUN apt-get update && apt-get -y install curl ca-certificates
+
+RUN curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" | sh
+
+RUN mv crossplane /usr/bin/crossplane
+
+RUN cp /certs/self-signed-ca.crt /usr/local/share/ca-certificates/self-signed-ca.crt
+
+RUN update-ca-certificates
+
+CMD tail -f /dev/null

registry/certs/generateCerts.sh

@@ -0,0 +1,7 @@
+openssl genrsa -out self-signed-ca.key 2048
+
+openssl req -new -subj "/C=DE/ST=/L=/O=Security/OU=IT/CN=k3d-registry.localhost" -key self-signed-ca.key -out self-signed-ca.csr
+
+openssl x509 -req -in self-signed-ca.csr -signkey self-signed-ca.key -out self-signed-ca.crt -days 3650 -sha256 -extfile v3.ext
+
+rm self-signed-ca.csr

registry/certs/self-signed-ca.crt

@@ -0,0 +1,25 @@
+-----BEGIN CERTIFICATE-----
+MIIEMDCCAxigAwIBAgIUEN+RefLDzC7vJA7BNYVJiqv0oVcwDQYJKoZIhvcNAQEL
+BQAwTjELMAkGA1UEBhMCREUxETAPBgNVBAoMCFNlY3VyaXR5MQswCQYDVQQLDAJJ
+VDEfMB0GA1UEAwwWazNkLXJlZ2lzdHJ5LmxvY2FsaG9zdDAeFw0yMzEyMDkyMzQw
+NTlaFw0zMzEyMDYyMzQwNTlaME4xCzAJBgNVBAYTAkRFMREwDwYDVQQKDAhTZWN1
+cml0eTELMAkGA1UECwwCSVQxHzAdBgNVBAMMFmszZC1yZWdpc3RyeS5sb2NhbGhv
+c3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCzrUmtfQEode3Q7TVm
+YtCIPr4QQfo2kX3ywT9kRmaoiPuQakHwkM62F5B00fT0aN+De6CjeWzLX8tnEaOG
+buFaz+iflZzi3FjmKRZ8CqxuD1gaKtVWtecLdxxnJvKTuly3hOTUZdJvzmkLzy36
+geuiJissfqRpoE61XfJVS7dz/oSPDhIpAgv+i08rbf9b/C53x8d4fmPn26Pb+4n2
+BGMvbV3XXkhwfczjfrHSaue2ZMGpXsdJGuag41G5gwbo5tCps9vQiMYnDZmElt+t
+7g3FL3qS9yZ89jtK3cdDn94qghnNdHICpfxGNQrVrU72q3aD24KmIkRHvlb18aM4
+iDQVAgMBAAGjggEEMIIBADAdBgNVHQ4EFgQUSWvj9CcVMWsSOoiG/UylE9z4pfUw
+gYsGA1UdIwSBgzCBgIAUSWvj9CcVMWsSOoiG/UylE9z4pfWhUqRQME4xCzAJBgNV
+BAYTAkRFMREwDwYDVQQKDAhTZWN1cml0eTELMAkGA1UECwwCSVQxHzAdBgNVBAMM
+FmszZC1yZWdpc3RyeS5sb2NhbGhvc3SCFBDfkXnyw8wu7yQOwTWFSYqr9KFXMAwG
+A1UdEwQFMAMBAf8wCwYDVR0PBAQDAgL8MBoGA1UdEQQTMBGCDyouazNkLmxvY2Fs
+aG9zdDAaBgNVHRIEEzARgg8qLmszZC5sb2NhbGhvc3QwDQYJKoZIhvcNAQELBQAD
+ggEBAE+5YIjS9E+UInr6U6XChi9qurxxj1iZdhKFSHWpyUvPUmvkHaFQld8r1PqO
+06ixsPluI0HSyfQDw8pbVDCpVXaGQ8u2WzPCBfvGHDT+hAebLsgcsX3S0IBa+YwE
+LERC2z+SgldTQvZUVl2a2fRQWj3gaL5ZA944Q4xSrDubjZPkQJiDst3LP9rX7wd3
+wkxljT2oPgLgeOUDK2ovkIYnEZ9Oar5760ffqugWY8loOZ94sPqgUW/Jtj3zBoBx
+KS98xjOlNUaEBO+YnAMW8Ga6cn145Oor9jzkmIvsHstXUDLhIRdaZTxDnYcD14a1
+/C+M+ASnnjYTxZYAsTsFZmnDqbs=
+-----END CERTIFICATE-----

registry/certs/self-signed-ca.key

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpQIBAAKCAQEAs61JrX0BKHXt0O01ZmLQiD6+EEH6NpF98sE/ZEZmqIj7kGpB
+8JDOtheQdNH09Gjfg3ugo3lsy1/LZxGjhm7hWs/on5Wc4txY5ikWfAqsbg9YGirV
+VrXnC3ccZybyk7pct4Tk1GXSb85pC88t+oHroiYrLH6kaaBOtV3yVUu3c/6Ejw4S
+KQIL/otPK23/W/wud8fHeH5j59uj2/uJ9gRjL21d115IcH3M436x0mrntmTBqV7H
+SRrmoONRuYMG6ObQqbPb0IjGJw2ZhJbfre4NxS96kvcmfPY7St3HQ5/eKoIZzXRy
+AqX8RjUK1a1O9qt2g9uCpiJER75W9fGjOIg0FQIDAQABAoIBAELvBQlqDepitCsm
+ksQHf6Ohq/Amg+2cGY07TEAwscGK9hwAWXwilb7j3F6IcHY0Mz+p6Fe9vjetAfrt
+aTuFRtIrrX71nzIr37ouCEBBQabmdG2bE0pwDRgKUTxkF+3MeQmsqgotvE0GWb+a
+eIhfqcYH7gFvVBQ9MdUCoGoCAN9mirWh9YN5Z4lD7wIce/iaCYjt3oFjJqEiZAaS
+9n2+QHK/fCSC8ohROereJfxWW6R/GWjhZXv8SbT40uylwIJ/Hc6fHNEApyotzlC5
+Q2nUGGn3NBCxOYY0Mxsh7xTzwKuOF5h94CGNyRjMmXXeEYew84o8ra1z+igbK9hQ
+1FuF8CkCgYEA7DXTKoVT+qZ9Y8UEI6uwxHVJalWD/cLVAGXODoZNJaSKArlQbGVd
+OR9jkLtLyxkLfJZRWEEWdHXVGFbqfYVYaPqJfwqwoUjDqYWRhiD9ygujSZht5kn/
+xK6v78sWTMG0uia7UoRajg0LNuhWGNLHmI2wT3krNhcKFLs9CzAUp2sCgYEAwrrz
+W9lLbYcpfolAq0u8XZWdjfH9/Ca8IqTkV9cZDMCOGN5Epba9e+udoFyYxKFDOsxu
+x6nmOqoIaobut7vEQb/ZZaSBTImt518N3OJRp2xK1/SWcItnG9+CBPLrMdQFm3of
+RN7j/e9zlQdJlhZxOTepIpXzzxX+T1IqYleS8n8CgYEAlZyKAbatOGdXqGfTdVx7
+/BNlysobbhGHpFbXKu/50UvkdlE2FVImt9Wf1L4tBZj5Y5I6N7ISLPWtYVs77wEU
+vHljuWnl8mIKPEdzowcmeGDirey0P1bDf6o9mZ6Co19Vv0pk/6EBUlQEtGR4lAR9
+k6y67buLmRkCs5rc0UdE7b0CgYEAqEec5MPcbhN+5U3BKAa0Cn+v/ymf3pauwJWU
+iuBkUmPYqF70HezUxNExj5JVpp7hlV9zYHJPdmTYDw76HqdSC6u6OB84aacVNLwt
+I7Qk8jUgXI2OsEWX3juBhbtTSuU5z0ly2jZv+lNCpAf/LIfKYrLaI/xIDK6p7WxJ
+GmEpqQsCgYEA3HUrT+BGVdpuXrwF5lhQWL+rot82gtK+SDG+HqX/mlCLw2qPlrV3
+hKDyUeeWv+FqMwm/jRWqYHaVsd5RGHmcToSkAB6Fd32E1zamRoUNQUg9nYElXfZP
+BLjB5P8WEY2BijaT/gKQlwLOEbHl2Isn+mfI1+QZzKtfwL3D2ERTrtA=
+-----END RSA PRIVATE KEY-----

registry/certs/v3.ext

@@ -0,0 +1,6 @@
+subjectKeyIdentifier   = hash
+authorityKeyIdentifier = keyid:always,issuer:always
+basicConstraints       = CA:TRUE
+keyUsage               = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign
+subjectAltName         = DNS:*.k3d.localhost, DNS:host.k3d.internal
+issuerAltName          = issuer:copy

registry/registries.yaml

@@ -0,0 +1,9 @@
+mirrors:
+  "registry.k3d.localhost:5000":
+    endpoint:
+      - https://registry.k3d.localhost:5000
+
+configs:
+  "registry.k3d.localhost:5000":
+    tls:
+      ca_file: "/etc/ssl/certs/self-signed-ca.crt"