diff --git a/README.md b/README.md index 376beba..0f944b1 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,73 @@ -# k3d Cluster +# Crossplane Demo -## Create +## Regenerate Certs (Optional) ``` -sudo k3d cluster create crossplane-demo -sudo k3d kubeconfig merge crossplane-demo --kubeconfig-switch-context -o /home//.kube/config -sudo chown : ~/.kube/config +./registry/certs/generateCerts.sh +``` + +## Create k3d Cluster +``` +sudo k3d cluster create crossplane-demo \ +--volume "$(pwd)/registry/registries.yaml:/etc/rancher/k3s/registries.yaml" \ +--volume "$(pwd)/registry/certs/self-signed-ca.crt:/etc/ssl/certs/self-signed-ca.crt" \ +-p "8081:80@loadbalancer" // not tested yet + +sudo k3d kubeconfig merge crossplane-demo --kubeconfig-switch-context -o ~/.kube/config + +USERNAME=$(whoami) && sudo chown ${USERNAME}:${USERNAME} ~/.kube/config +``` + +## DNS +Many Linux distributions ships with NSS-myhostname, which resolves *.localhost + +If not install `nss-myhostname` +``` +sudo apt install libnss-myhostname +``` + +If not possbile you have to add /etc/hots entries pointing to 127.0.0.1 manually + +## Selfhosted Registry +We can not utilize K3d's build-in registry (`--create-registry`), because we need SSL. +Furthermore we can also not use build-in image upload mechanism (`k3d image import`), because crossplane needs an OCI compatible registry to pull from. + + +Start selfhosted registry +``` +sudo docker-compose up -d [--build] +``` + +Connect Cluster to selfhosted registry +``` +sudo docker network connect k3d-crossplane-demo registry.k3d.localhost +``` + +Modify coredns configmap and add entry: +``` +sudo docker inspect registry.k3d.localhost | jq -r '.[0].NetworkSettings.Networks."k3d-crossplane-demo".IPAddress' +``` + + registry.k3d.localhost + +retstart codedns pod + +### Push image +``` +sudo docker tag : crossplane-demo-registry.local:/: +sudo docker push crossplane-demo-registry.local:/: +``` + +### Push *.xpkg file + +Start container with crossplane cli + trusted self signed cert +``` +sudo docker build -t "crossplane-cli:latest" -f ./registry/Dockerfile.crossplane-cli ./registry +sudo docker run --rm -it --net=host -v $(pwd)/registry/files:/files crossplane-cli:latest bash +``` + +push file to OCI registry +``` +crossplane xpkg push -f /files/ registry.k3d.localhost:5000/: ``` ## Start and Stop @@ -17,6 +80,12 @@ sudo k3d cluster start crossplane-demo https://docs.crossplane.io/latest/software/install/ ## Install via Helm +Create configmap to trust private registry with selfsigned cert +``` +kubectl -n crossplane-system create cm ca-bundle-config \ +--from-file=ca-bundle=./registry/certs/self-signed-ca.crt +``` + ``` helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update @@ -25,7 +94,8 @@ helm repo update ``` helm install crossplane \ --namespace crossplane-system \ ---create-namespace crossplane-stable/crossplane +--create-namespace crossplane-stable/crossplane \ +--set registryCaBundleConfig.name=ca-bundle-config,registryCaBundleConfig.key=ca-bundle ``` # Crossplane UI @@ -34,15 +104,11 @@ helm install crossplane \ ``` helm repo add komodorio https://helm-charts.komodor.io helm repo update -helm install komoplane komodorio/komoplane --set "ingress.enabled=true,ingress.hosts[0].host=komoplane.k8s.local,ingress.hosts[0].paths[0].path=/,ingress.hosts[0].paths[0].pathType=ImplementationSpecific" -``` - -Add entry to /etc/hosts -``` - komoplane.k8s.local +helm install komoplane komodorio/komoplane \ +--set "ingress.enabled=true,ingress.hosts[0].host=komoplane.k8s.localhost,ingress.hosts[0].paths[0].path=/,ingress.hosts[0].paths[0].pathType=ImplementationSpecific" ``` -Crossplane UI URL: http://komoplane.k8s.local/ +Crossplane UI URL: http://komoplane.k8s.localhost/ # Testing with Grafana ## Install Grafana via Helm @@ -52,24 +118,70 @@ https://github.com/grafana/helm-charts/blob/main/charts/grafana/README.md ``` helm repo add grafana https://grafana.github.io/helm-charts helm repo update -helm install my-release grafana/grafana --set "adminUser=admin,adminPassword=password,ingress.enabled=true,ingress.hosts[0]=grafana.k8s.local" +helm install my-release grafana/grafana --set "adminUser=admin,adminPassword=password,ingress.enabled=true,ingress.hosts[0]=grafana.k8s.localhost" ``` -Add entry to /etc/hosts +## Install Crossplane Grafana Provider +[Grafana Provider](https://marketplace.upbound.io/providers/grafana/provider-grafana/v0.8.0) was generated by [upjet](https://github.com/upbound/upjet). + +Therefore Provider configuration (Kind:ProviderConfig.spec.credentials) can be inferred by: +https://registry.terraform.io/providers/grafana/grafana/latest/docs#schema + ``` - grafana.k8s.local +kubectl apply -f ./k8s/crossplane/grafana ``` -Grafana URL: http://grafana.k8s.local/ +## Check -## Install Crossplane Grafana Provider -[Grafana Provider](https://marketplace.upbound.io/providers/grafana/provider-grafana/v0.8.0) was generated by [upjet](https://github.com/upbound/upjet). +Grafana URL: http://grafana.k8s.localhost/ -Therefore Provider configuration can be inferred by: -https://registry.terraform.io/providers/grafana/grafana/latest/docs#schema +# Test with Temporal +## Install Temporal via Helm +``` +git clone https://github.com/temporalio/helm-charts +cd helm-charts + +helm dependencies update + +helm install --set "server.replicaCount=1,cassandra.config.cluster_size=1,prometheus.enabled=false,grafana.enabled=false,elasticsearch.enabled=false,web.ingress.enabled=true,web.ingress.hosts[0]=temporal.k8s.localhost,server.frontend.service.type=LoadBalancer" temporaltest . --timeout 15m +``` + +## Push *.xpkg file +Start container with crossplane cli + trusted self signed cert +``` +sudo docker build -t "crossplane-cli:latest" -f ./registry/Dockerfile.crossplane-cli ./registry +sudo docker run --rm -it --net=host -v $(pwd)/registry/files:/files crossplane-cli:latest bash +``` +push file to OCI registry (The file was built with `make build` in source repo) ``` -kubectl apply -f ./k8s/crossplane +crossplane xpkg push -f /files/provider-temporal-v0.0.0-2.g3abd9d5.dirty.xpkg registry.k3d.localhost:5000/provider-temporal:v13.0.0 ``` +## Install Crossplane Temporal Provider +``` +kubectl apply -f ./k8s/crossplane/grafana +``` + +## Check + +Temporal URL: http://temporal.k8s.localhost/ + +Query namespaces with CLI +``` +temporal operator namespace list --address temporal.k8s.localhost:7233 +``` + +## Install Crossplane Temporal Provider +``` +git clone https://github.com/denniskniep/provider-temporal.git +make build +``` + +push the created xpkg file to oci registry: +see section `Push *.xpkg file` + +``` +kubectl apply -f ./k8s/crossplane/temporal +``` diff --git a/docker-compose.yaml b/docker-compose.yaml new file mode 100644 index 0000000..c2bba0b --- /dev/null +++ b/docker-compose.yaml @@ -0,0 +1,17 @@ +version: "3.8" +services: + registry.k3d.localhost: + container_name: registry.k3d.localhost + image: registry:2 + environment: + - REGISTRY_HTTP_ADDR=0.0.0.0:5000 + - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/self-signed-ca.crt + - REGISTRY_HTTP_TLS_KEY=/certs/self-signed-ca.key + volumes: + - ./registry/certs:/certs + - registry-data:/var/lib/registry + ports: + - 5000:5000 + +volumes: + registry-data: \ No newline at end of file diff --git a/k8s/crossplane/01-grafana-provider.yaml b/k8s/crossplane/grafana/01-grafana-provider.yaml similarity index 57% rename from k8s/crossplane/01-grafana-provider.yaml rename to k8s/crossplane/grafana/01-grafana-provider.yaml index ba912ae..ad1768d 100644 --- a/k8s/crossplane/01-grafana-provider.yaml +++ b/k8s/crossplane/grafana/01-grafana-provider.yaml @@ -3,6 +3,6 @@ kind: Provider metadata: name: provider-grafana spec: - package: xpkg.upbound.io/grafana/provider-grafana:v0.8.0 - packagePullPolicy: IfNotPresent + package: grafana/provider-grafana:v0.8.0 + packagePullPolicy: Never revisionActivationPolicy: Automatic \ No newline at end of file diff --git a/k8s/crossplane/02-grafana-provider-config-secret.yaml b/k8s/crossplane/grafana/02-grafana-provider-config-secret copy.yaml similarity index 100% rename from k8s/crossplane/02-grafana-provider-config-secret.yaml rename to k8s/crossplane/grafana/02-grafana-provider-config-secret copy.yaml diff --git a/k8s/crossplane/03-grafana-provider-config.yaml b/k8s/crossplane/grafana/03-grafana-provider-config.yaml similarity index 100% rename from k8s/crossplane/03-grafana-provider-config.yaml rename to k8s/crossplane/grafana/03-grafana-provider-config.yaml diff --git a/k8s/crossplane/04-grafana-managed-resources.yaml b/k8s/crossplane/grafana/04-grafana-managed-resources.yaml similarity index 100% rename from k8s/crossplane/04-grafana-managed-resources.yaml rename to k8s/crossplane/grafana/04-grafana-managed-resources.yaml diff --git a/k8s/crossplane/temporal/01-temporal-provider.yaml b/k8s/crossplane/temporal/01-temporal-provider.yaml new file mode 100644 index 0000000..ffa0e60 --- /dev/null +++ b/k8s/crossplane/temporal/01-temporal-provider.yaml @@ -0,0 +1,25 @@ +apiVersion: pkg.crossplane.io/v1beta1 +kind: DeploymentRuntimeConfig +metadata: + name: debug-config +spec: + deploymentTemplate: + spec: + selector: {} + template: + spec: + containers: + - name: package-runtime + args: + - --debug +--- +apiVersion: pkg.crossplane.io/v1 +kind: Provider +metadata: + name: provider-temporal +spec: + package: registry.k3d.localhost:5000/provider-temporal:v13.0.0 + packagePullPolicy: IfNotPresent + revisionActivationPolicy: Automatic + runtimeConfigRef: + name: debug-config \ No newline at end of file diff --git a/k8s/crossplane/temporal/02-temporal-provider-config-secret.yaml b/k8s/crossplane/temporal/02-temporal-provider-config-secret.yaml new file mode 100644 index 0000000..90e858c --- /dev/null +++ b/k8s/crossplane/temporal/02-temporal-provider-config-secret.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Secret +metadata: + name: provider-temporal-config-creds + namespace: crossplane-system +type: Opaque +stringData: + credentials: | + { + "HostPort": "temporaltest-frontend.default:7233" + } \ No newline at end of file diff --git a/k8s/crossplane/temporal/03-temporal-provider-config.yaml b/k8s/crossplane/temporal/03-temporal-provider-config.yaml new file mode 100644 index 0000000..7ffbfcb --- /dev/null +++ b/k8s/crossplane/temporal/03-temporal-provider-config.yaml @@ -0,0 +1,11 @@ +apiVersion: temporal.crossplane.io/v1alpha1 +kind: ProviderConfig +metadata: + name: local-temporal-instance-config +spec: + credentials: + source: Secret + secretRef: + namespace: crossplane-system + name: provider-temporal-config-creds + key: credentials \ No newline at end of file diff --git a/k8s/crossplane/temporal/04-temporal-namespace-resources.yaml b/k8s/crossplane/temporal/04-temporal-namespace-resources.yaml new file mode 100644 index 0000000..bdd7d43 --- /dev/null +++ b/k8s/crossplane/temporal/04-temporal-namespace-resources.yaml @@ -0,0 +1,11 @@ +apiVersion: core.temporal.crossplane.io/v1alpha1 +kind: TemporalNamespace +metadata: + name: ns1 +spec: + forProvider: + name: "Test 1" + description: "Test Desc 1" + ownerEmail: "Test@test.local" + providerConfigRef: + name: local-temporal-instance-config diff --git a/registry/Dockerfile.crossplane-cli b/registry/Dockerfile.crossplane-cli new file mode 100644 index 0000000..dd6a71e --- /dev/null +++ b/registry/Dockerfile.crossplane-cli @@ -0,0 +1,15 @@ +FROM ubuntu:22.04 + +COPY ./certs/ /certs + +RUN apt-get update && apt-get -y install curl ca-certificates + +RUN curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" | sh + +RUN mv crossplane /usr/bin/crossplane + +RUN cp /certs/self-signed-ca.crt /usr/local/share/ca-certificates/self-signed-ca.crt + +RUN update-ca-certificates + +CMD tail -f /dev/null \ No newline at end of file diff --git a/registry/certs/generateCerts.sh b/registry/certs/generateCerts.sh new file mode 100755 index 0000000..e9e3ef8 --- /dev/null +++ b/registry/certs/generateCerts.sh @@ -0,0 +1,7 @@ +openssl genrsa -out self-signed-ca.key 2048 + +openssl req -new -subj "/C=DE/ST=/L=/O=Security/OU=IT/CN=k3d-registry.localhost" -key self-signed-ca.key -out self-signed-ca.csr + +openssl x509 -req -in self-signed-ca.csr -signkey self-signed-ca.key -out self-signed-ca.crt -days 3650 -sha256 -extfile v3.ext + +rm self-signed-ca.csr diff --git a/registry/certs/self-signed-ca.crt b/registry/certs/self-signed-ca.crt new file mode 100644 index 0000000..fe32579 --- /dev/null +++ b/registry/certs/self-signed-ca.crt @@ -0,0 +1,25 @@ +-----BEGIN CERTIFICATE----- +MIIEMDCCAxigAwIBAgIUEN+RefLDzC7vJA7BNYVJiqv0oVcwDQYJKoZIhvcNAQEL +BQAwTjELMAkGA1UEBhMCREUxETAPBgNVBAoMCFNlY3VyaXR5MQswCQYDVQQLDAJJ +VDEfMB0GA1UEAwwWazNkLXJlZ2lzdHJ5LmxvY2FsaG9zdDAeFw0yMzEyMDkyMzQw +NTlaFw0zMzEyMDYyMzQwNTlaME4xCzAJBgNVBAYTAkRFMREwDwYDVQQKDAhTZWN1 +cml0eTELMAkGA1UECwwCSVQxHzAdBgNVBAMMFmszZC1yZWdpc3RyeS5sb2NhbGhv +c3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCzrUmtfQEode3Q7TVm +YtCIPr4QQfo2kX3ywT9kRmaoiPuQakHwkM62F5B00fT0aN+De6CjeWzLX8tnEaOG +buFaz+iflZzi3FjmKRZ8CqxuD1gaKtVWtecLdxxnJvKTuly3hOTUZdJvzmkLzy36 +geuiJissfqRpoE61XfJVS7dz/oSPDhIpAgv+i08rbf9b/C53x8d4fmPn26Pb+4n2 +BGMvbV3XXkhwfczjfrHSaue2ZMGpXsdJGuag41G5gwbo5tCps9vQiMYnDZmElt+t +7g3FL3qS9yZ89jtK3cdDn94qghnNdHICpfxGNQrVrU72q3aD24KmIkRHvlb18aM4 +iDQVAgMBAAGjggEEMIIBADAdBgNVHQ4EFgQUSWvj9CcVMWsSOoiG/UylE9z4pfUw +gYsGA1UdIwSBgzCBgIAUSWvj9CcVMWsSOoiG/UylE9z4pfWhUqRQME4xCzAJBgNV +BAYTAkRFMREwDwYDVQQKDAhTZWN1cml0eTELMAkGA1UECwwCSVQxHzAdBgNVBAMM +FmszZC1yZWdpc3RyeS5sb2NhbGhvc3SCFBDfkXnyw8wu7yQOwTWFSYqr9KFXMAwG +A1UdEwQFMAMBAf8wCwYDVR0PBAQDAgL8MBoGA1UdEQQTMBGCDyouazNkLmxvY2Fs +aG9zdDAaBgNVHRIEEzARgg8qLmszZC5sb2NhbGhvc3QwDQYJKoZIhvcNAQELBQAD +ggEBAE+5YIjS9E+UInr6U6XChi9qurxxj1iZdhKFSHWpyUvPUmvkHaFQld8r1PqO +06ixsPluI0HSyfQDw8pbVDCpVXaGQ8u2WzPCBfvGHDT+hAebLsgcsX3S0IBa+YwE +LERC2z+SgldTQvZUVl2a2fRQWj3gaL5ZA944Q4xSrDubjZPkQJiDst3LP9rX7wd3 +wkxljT2oPgLgeOUDK2ovkIYnEZ9Oar5760ffqugWY8loOZ94sPqgUW/Jtj3zBoBx +KS98xjOlNUaEBO+YnAMW8Ga6cn145Oor9jzkmIvsHstXUDLhIRdaZTxDnYcD14a1 +/C+M+ASnnjYTxZYAsTsFZmnDqbs= +-----END CERTIFICATE----- diff --git a/registry/certs/self-signed-ca.key b/registry/certs/self-signed-ca.key new file mode 100644 index 0000000..012f2b8 --- /dev/null +++ b/registry/certs/self-signed-ca.key @@ -0,0 +1,27 @@ +-----BEGIN RSA PRIVATE KEY----- +MIIEpQIBAAKCAQEAs61JrX0BKHXt0O01ZmLQiD6+EEH6NpF98sE/ZEZmqIj7kGpB +8JDOtheQdNH09Gjfg3ugo3lsy1/LZxGjhm7hWs/on5Wc4txY5ikWfAqsbg9YGirV +VrXnC3ccZybyk7pct4Tk1GXSb85pC88t+oHroiYrLH6kaaBOtV3yVUu3c/6Ejw4S +KQIL/otPK23/W/wud8fHeH5j59uj2/uJ9gRjL21d115IcH3M436x0mrntmTBqV7H +SRrmoONRuYMG6ObQqbPb0IjGJw2ZhJbfre4NxS96kvcmfPY7St3HQ5/eKoIZzXRy +AqX8RjUK1a1O9qt2g9uCpiJER75W9fGjOIg0FQIDAQABAoIBAELvBQlqDepitCsm +ksQHf6Ohq/Amg+2cGY07TEAwscGK9hwAWXwilb7j3F6IcHY0Mz+p6Fe9vjetAfrt +aTuFRtIrrX71nzIr37ouCEBBQabmdG2bE0pwDRgKUTxkF+3MeQmsqgotvE0GWb+a +eIhfqcYH7gFvVBQ9MdUCoGoCAN9mirWh9YN5Z4lD7wIce/iaCYjt3oFjJqEiZAaS +9n2+QHK/fCSC8ohROereJfxWW6R/GWjhZXv8SbT40uylwIJ/Hc6fHNEApyotzlC5 +Q2nUGGn3NBCxOYY0Mxsh7xTzwKuOF5h94CGNyRjMmXXeEYew84o8ra1z+igbK9hQ +1FuF8CkCgYEA7DXTKoVT+qZ9Y8UEI6uwxHVJalWD/cLVAGXODoZNJaSKArlQbGVd +OR9jkLtLyxkLfJZRWEEWdHXVGFbqfYVYaPqJfwqwoUjDqYWRhiD9ygujSZht5kn/ +xK6v78sWTMG0uia7UoRajg0LNuhWGNLHmI2wT3krNhcKFLs9CzAUp2sCgYEAwrrz +W9lLbYcpfolAq0u8XZWdjfH9/Ca8IqTkV9cZDMCOGN5Epba9e+udoFyYxKFDOsxu +x6nmOqoIaobut7vEQb/ZZaSBTImt518N3OJRp2xK1/SWcItnG9+CBPLrMdQFm3of +RN7j/e9zlQdJlhZxOTepIpXzzxX+T1IqYleS8n8CgYEAlZyKAbatOGdXqGfTdVx7 +/BNlysobbhGHpFbXKu/50UvkdlE2FVImt9Wf1L4tBZj5Y5I6N7ISLPWtYVs77wEU +vHljuWnl8mIKPEdzowcmeGDirey0P1bDf6o9mZ6Co19Vv0pk/6EBUlQEtGR4lAR9 +k6y67buLmRkCs5rc0UdE7b0CgYEAqEec5MPcbhN+5U3BKAa0Cn+v/ymf3pauwJWU +iuBkUmPYqF70HezUxNExj5JVpp7hlV9zYHJPdmTYDw76HqdSC6u6OB84aacVNLwt +I7Qk8jUgXI2OsEWX3juBhbtTSuU5z0ly2jZv+lNCpAf/LIfKYrLaI/xIDK6p7WxJ +GmEpqQsCgYEA3HUrT+BGVdpuXrwF5lhQWL+rot82gtK+SDG+HqX/mlCLw2qPlrV3 +hKDyUeeWv+FqMwm/jRWqYHaVsd5RGHmcToSkAB6Fd32E1zamRoUNQUg9nYElXfZP +BLjB5P8WEY2BijaT/gKQlwLOEbHl2Isn+mfI1+QZzKtfwL3D2ERTrtA= +-----END RSA PRIVATE KEY----- diff --git a/registry/certs/v3.ext b/registry/certs/v3.ext new file mode 100644 index 0000000..f1ab8fa --- /dev/null +++ b/registry/certs/v3.ext @@ -0,0 +1,6 @@ +subjectKeyIdentifier = hash +authorityKeyIdentifier = keyid:always,issuer:always +basicConstraints = CA:TRUE +keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign +subjectAltName = DNS:*.k3d.localhost, DNS:host.k3d.internal +issuerAltName = issuer:copy \ No newline at end of file diff --git a/registry/files/provider-temporal-v0.0.0-2.g3abd9d5.dirty.xpkg b/registry/files/provider-temporal-v0.0.0-2.g3abd9d5.dirty.xpkg new file mode 100644 index 0000000..a6dbfc4 Binary files /dev/null and b/registry/files/provider-temporal-v0.0.0-2.g3abd9d5.dirty.xpkg differ diff --git a/registry/registries.yaml b/registry/registries.yaml new file mode 100644 index 0000000..7dbabec --- /dev/null +++ b/registry/registries.yaml @@ -0,0 +1,9 @@ +mirrors: + "registry.k3d.localhost:5000": + endpoint: + - https://registry.k3d.localhost:5000 + +configs: + "registry.k3d.localhost:5000": + tls: + ca_file: "/etc/ssl/certs/self-signed-ca.crt" \ No newline at end of file