Crossplane version 2 upgrade

README_Crossplane_V2_Upgrade.md

@@ -0,0 +1,384 @@
+# Crossplane v1.17.1 → v2.0.2 Upgrade Guide
+
+## ⚠️ CRITICAL BREAKING CHANGES
+
+### 🚨 Composition API Completely Changed
+**ALL existing compositions will BREAK and need refactoring!**
+
+- ❌ **`resources:` field REMOVED** - No longer supported
+- ✅ **`pipeline:` mode now MANDATORY** - All compositions must use pipeline mode
+- 🔧 **Composition Functions Required** - Must install functions like `function-go-templating`
+- 📋 **Migration Required** - Every composition needs manual conversion
+
+### 📁 Affected Files in This Project:
+```
+❌ platform/crossplane/compositions/dynamodb/ddb-table.yml
+❌ platform/crossplane/compositions/rds/rds-postgres.yaml  
+❌ platform/crossplane/compositions/rds/postgres-aurora.yaml
+❌ platform/crossplane/compositions/s3/multi-tenant.yaml
+❌ platform/crossplane/compositions/s3/general-purpose.yaml
+```
+
+### 🔄 Required Actions:
+1. **Install composition functions** before upgrading
+2. **Convert all compositions** to pipeline mode
+3. **Test each composition** individually
+4. **Update ArgoCD applications** to deploy functions first
+
+---
+
+## Overview
+This guide covers upgrading Crossplane from v1.17.1 to v2.0.2 across management, dev, and prod clusters managed by ArgoCD.
+
+## Additional Breaking Changes in v2.0.2
+- `--enable-environment-configs` flag **removed** (causes crashes)
+- CRD version alignment required (`storedVersions` vs `spec.versions`)
+- Some CRDs need recreation due to version conflicts
+
+## Pre-Upgrade: File Changes
+
+### 1. Update Version Numbers
+Update `targetRevision` from `1.17.1` to `2.0.2` in:
+- `platform/infra/terraform/mgmt/terraform/templates/argocd-apps/crossplane.yaml` (line 15)
+- `platform/infra/terraform/deploy-apps/manifests/crossplane-dev.yaml` (line 11)
+- `platform/infra/terraform/deploy-apps/manifests/crossplane-prod.yaml` (line 11)
+
+### 2. Remove Deprecated Flag
+Comment out or remove `--enable-environment-configs` from:
+- `packages/crossplane/dev/values.yaml`
+- Change `args: [--enable-environment-configs]` to `args: []` in dev/prod manifests
+
+## Upgrade Process
+
+### Management Cluster (First)
+
+1. **Apply File Changes & ArgoCD Refresh**
+   ```bash
+   kubectl patch application crossplane -n argocd --type='merge' -p='{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'
+   ```
+
+2. **Fix CRD Issues**
+   ```bash
+   # Delete problematic CRDs
+   kubectl delete crd environmentconfigs.apiextensions.crossplane.io usages.apiextensions.crossplane.io
+
+   # Delete crash-looping pod
+   kubectl delete pod -n crossplane-system <crash-looping-pod-name>
+   ```
+
+3. **Verify Success**
+   ```bash
+   kubectl get pods -n crossplane-system
+   # Should show Crossplane v2.0.2 running
+   ```
+
+### Dev & Prod Clusters (Remote)
+
+**Issue**: ArgoCD applications have deprecated flag hardcoded in Helm values, causing continuous crashes.
+
+#### Step 1: Disable ArgoCD Auto-Sync (Management Cluster)
+```bash
+# Disable auto-sync to prevent ArgoCD from reverting manual fixes
+kubectl patch application crossplane-dev -n argocd --type='json' -p='[{"op": "remove", "path": "/spec/syncPolicy/automated"}]'
+kubectl patch application crossplane-prod -n argocd --type='json' -p='[{"op": "remove", "path": "/spec/syncPolicy/automated"}]'
+```
+
+#### Step 2: Fix Each Remote Cluster
+
+**For Dev Cluster:**
+```bash
+# Switch to dev cluster
+# Delete problematic CRDs
+kubectl delete crd environmentconfigs.apiextensions.crossplane.io usages.apiextensions.crossplane.io
+
+# Manual deployment patch to remove deprecated flag
+kubectl patch deployment crossplane -n crossplane-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": ["core", "start"]}]'
+
+# Verify success
+kubectl get pods -n crossplane-system
+```
+
+**For Prod Cluster:**
+```bash
+# Switch to prod cluster
+# Delete problematic CRDs
+kubectl delete crd environmentconfigs.apiextensions.crossplane.io usages.apiextensions.crossplane.io
+
+# Manual deployment patch to remove deprecated flag
+kubectl patch deployment crossplane -n crossplane-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": ["core", "start"]}]'
+
+# Verify success
+kubectl get pods -n crossplane-system
+```
+
+#### Step 3: Fix ArgoCD Application Sources (Management Cluster)
+```bash
+# Update Helm values to remove deprecated flag
+kubectl patch application crossplane-dev -n argocd --type='json' -p='[{"op": "replace", "path": "/spec/source/helm/values", "value": "args: []\n"}]'
+kubectl patch application crossplane-prod -n argocd --type='json' -p='[{"op": "replace", "path": "/spec/source/helm/values", "value": "args: []\n"}]'
+```
+
+#### Step 4: Re-enable ArgoCD Auto-Sync (Management Cluster)
+```bash
+# Re-enable auto-sync with correct configuration
+kubectl patch application crossplane-dev -n argocd --type='json' -p='[{"op": "add", "path": "/spec/syncPolicy/automated", "value": {"selfHeal": true}}]'
+kubectl patch application crossplane-prod -n argocd --type='json' -p='[{"op": "add", "path": "/spec/syncPolicy/automated", "value": {"selfHeal": true}}]'
+```
+
+## Verification
+
+### Check All Clusters
+```bash
+# On each cluster
+kubectl get pods -n crossplane-system
+kubectl get crd | grep -E "(environmentconfigs|usages)"
+
+# Should show:
+# - Crossplane v2.0.2 pods running (1/1 Ready)
+# - All providers updated and running
+# - CRDs recreated with proper versions
+```
+
+### Check ArgoCD Applications (Management Cluster)
+```bash
+kubectl get applications -n argocd | grep crossplane
+# Should show all as "Synced" and "Healthy"
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **CrashLoopBackOff with "unknown flag --enable-environment-configs"**
+   - **Cause**: Deprecated flag still in deployment args
+   - **Fix**: Apply manual deployment patch or fix ArgoCD Helm values
+
+2. **Init container fails with CRD version mismatch**
+   - **Cause**: `storedVersions` doesn't match `spec.versions`
+   - **Fix**: Delete problematic CRDs and let Crossplane recreate them
+
+3. **ArgoCD keeps reverting manual patches**
+   - **Cause**: Auto-sync enabled with wrong configuration
+   - **Fix**: Disable auto-sync, fix source configuration, re-enable
+
+### Emergency Rollback
+If upgrade fails completely:
+```bash
+# Revert version in ArgoCD applications
+kubectl patch application crossplane-dev -n argocd --type='json' -p='[{"op": "replace", "path": "/spec/source/targetRevision", "value": "1.17.1"}]'
+kubectl patch application crossplane-prod -n argocd --type='json' -p='[{"op": "replace", "path": "/spec/source/targetRevision", "value": "1.17.1"}]'
+```
+
+## Success Criteria
+- ✅ All clusters running Crossplane v2.0.2
+- ✅ No crash-looping pods
+- ✅ All providers updated and healthy
+- ✅ ArgoCD applications synced and healthy
+- ✅ CRDs properly aligned with correct versions
+
+## In depth how to upgrade guide
+
+### Prerequisites
+- management cluster alias: engineering
+- development cluster alias: dev
+- production cluster alias: prod
+
+### Step-by-Step Execution Order
+
+#### Phase 1: ArgoCD UI Updates
+1. **Update ArgoCD UI Application Manifests**:
+
+    1. `argocd/crossplane` manifest changes:
+
+        - `targetRevision`: 2.0.2 (formerly 1.17.1)
+        - `targetRevision`: crossplane-version-upgrade (formerly main)
+        - If repoURL for the git repo is missing the acutal link replace with `https://github.com/aws-samples/appmod-blueprints` manually
+    2. argocd/crossplane-dev
+
+        - `targetRevision`: 2.0.2 (formerly 1.17.1)
+    3. argocd/crossplane-prod
+
+        - `targetRevision`: 2.0.2 (formerly 1.17.1)
+
+
+#### Phase 2: Management Cluster Upgrade
+2. **Trigger upgrade**:
+   ```bash
+   kubectl patch application crossplane -n argocd --type='merge' -p='{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'
+   ```
+
+3. **Fix CRD issues** (when you see Init:CrashLoopBackOff):
+   ```bash
+   kubectl delete crd environmentconfigs.apiextensions.crossplane.io usages.apiextensions.crossplane.io
+   kubectl delete pod <crash-looping-pod-name> -n crossplane-system
+   ```
+
+4. **Verify management cluster**:
+   ```bash
+   kubectl get pods -n crossplane-system  # All should be Running
+   ```
+
+#### Phase 3: Remote Clusters (Dev/Prod)
+5. **Fix ArgoCD applications** (from management cluster):
+   ```bash
+   # Disable auto-sync
+   kubectl patch application crossplane-dev -n argocd --type='json' -p='[{"op": "remove", "path": "/spec/syncPolicy/automated"}]'
+   kubectl patch application crossplane-prod -n argocd --type='json' -p='[{"op": "remove", "path": "/spec/syncPolicy/automated"}]'
+
+   # Fix Helm values
+   kubectl patch application crossplane-dev -n argocd --type='json' -p='[{"op": "replace", "path": "/spec/source/helm/values", "value": "args: []\n"}]'
+   kubectl patch application crossplane-prod -n argocd --type='json' -p='[{"op": "replace", "path": "/spec/source/helm/values", "value": "args: []\n"}]'
+
+   # Re-enable auto-sync
+   kubectl patch application crossplane-dev -n argocd --type='json' -p='[{"op": "add", "path": "/spec/syncPolicy/automated", "value": {"selfHeal": true}}]'
+   kubectl patch application crossplane-prod -n argocd --type='json' -p='[{"op": "add", "path": "/spec/syncPolicy/automated", "value": {"selfHeal": true}}]'
+   ```
+
+6. **Switch to dev cluster and fix CRDs**:
+   ```bash
+   # Switch context to dev cluster
+   kubectl delete crd environmentconfigs.apiextensions.crossplane.io usages.apiextensions.crossplane.io
+   kubectl delete pod <crash-looping-pod-name> -n crossplane-system
+   kubectl get pods -n crossplane-system  # Verify all Running
+   ```
+
+7. **Switch to prod cluster and fix CRDs**:
+   ```bash
+   # Switch context to prod cluster
+   kubectl delete crd environmentconfigs.apiextensions.crossplane.io usages.apiextensions.crossplane.io
+   kubectl delete pod <crash-looping-pod-name> -n crossplane-system
+   kubectl get pods -n crossplane-system  # Verify all Running
+   ```
+
+#### Phase 4: Final Verification
+8. **Check all ArgoCD applications** (from management cluster):
+   ```bash
+   kubectl get applications -n argocd | grep crossplane
+   # All should show "Synced" and "Healthy"
+   ```
+
+### Critical Success Indicators
+- ✅ No pods in CrashLoopBackOff state
+- ✅ All Crossplane pods show image version v2.0.2
+- ✅ ArgoCD applications show "Synced" and "Healthy"
+- ✅ Providers are updating to newer versions
+- ✅ All compositions working with new pipeline mode
+
+## Composition Migration Guide
+
+**Status:** All three core compositions successfully migrated to Crossplane v2 + Upbound providers:
+- ✅ **DynamoDB**: Functioning but not throughly tested
+- ✅ **S3**: Functioning but not throughly tested
+- ❌ **RDS**: Still giving issues
+
+### What Changes Were Made
+
+**Crossplane v2 Requirements:**
+- ✅ **Pipeline Mode** - v2 removed support for `spec.resources` format
+- ✅ **String Transform Syntax** - `type: Format` mandatory in v2
+- ✅ **Function Integration** - Must use `function-patch-and-transform`
+
+**Upbound Provider Requirements:**
+- ✅ **API Version Changes** - Old community provider APIs don't exist
+- ✅ **S3 Resource Splitting** - Upbound removed nested fields from Bucket:
+  - `publicAccessBlockConfiguration` → `BucketPublicAccessBlock` CRD
+  - `objectOwnership` → `BucketOwnershipControls` CRD
+  - `serverSideEncryptionConfiguration` → `BucketServerSideEncryptionConfiguration` CRD
+- ✅ **RDS Field Renames** - Old fields don't exist in Upbound:
+  - `masterUsername` → `username`
+  - `masterUserPasswordSecretRef` → `passwordSecretRef`
+- ✅ **Region Requirements** - Upbound enforces region on all resources
+- ✅ **Schema Validation** - Upbound rejects invalid formats
+
+**Error-Driven Fixes:**
+- ✅ **Connection Secret Namespace** - S3 failed validation without it
+- ✅ **Rule Format Fixes** - Array vs object validation errors
+- ✅ **EC2 Provider Installation** - SecurityGroup CRD missing
+
+#### 🔧 Optional Simplifications (Could Be Enhanced Later)
+
+- **Tags Handling** - Removed for simplicity (could implement transforms)
+- **SecurityGroup Features** - Simplified (could restore with field mapping)
+
+**Summary:** ~90% of changes were absolutely required for the upgrade.
+
+### Install Required Functions First
+```bash
+# Install patch-and-transform function (required for pipeline mode)
+kubectl apply -f - <<EOF
+apiVersion: pkg.crossplane.io/v1
+kind: Function
+metadata:
+  name: function-patch-and-transform
+spec:
+  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.2.1
+EOF
+
+# Wait for function to be ready
+kubectl get function.pkg.crossplane.io -w
+```
+
+### Migration Examples
+
+**S3 Composition Changes:**
+```yaml
+# Before (v1 - Single Resource)
+resources:
+  - name: s3-bucket
+    base:
+      apiVersion: s3.aws.crossplane.io/v1beta1
+      kind: Bucket
+      spec:
+        forProvider:
+          publicAccessBlockConfiguration:
+            blockPublicPolicy: true
+
+# After (v2 - Split into 4 Resources)
+resources:
+  - name: bucket
+    base:
+      apiVersion: s3.aws.upbound.io/v1beta2
+      kind: Bucket
+  - name: bucket-public-access-block
+    base:
+      apiVersion: s3.aws.upbound.io/v1beta1
+      kind: BucketPublicAccessBlock
+```
+
+**RDS Composition Changes:**
+```yaml
+# Before (Community Provider)
+base:
+  apiVersion: rds.aws.crossplane.io/v1alpha1
+  kind: DBInstance
+  spec:
+    forProvider:
+      masterUsername: root
+      dbInstanceClass: db.t4g.small
+
+# After (Upbound Provider)
+base:
+  apiVersion: rds.aws.upbound.io/v1beta3
+  kind: Instance
+  spec:
+    forProvider:
+      username: root
+      instanceClass: db.t4g.small
+```
+
+### Migration Results
+
+| Service | Resources Created | Status | Notes |
+|---------|------------------|--------|---------|
+| **DynamoDB** | 1 Table | ✅ Ready | Minimal changes needed |
+| **S3** | 4 Resources | ✅ Ready | Bucket + 3 companion resources |
+| **RDS** | 3 Resources | ✅ Creating | SubnetGroup + SecurityGroup + Instance |
+
+## Key Lessons
+1. **Composition API changed completely** - All compositions need migration
+2. **Install functions first** - Pipeline mode requires composition functions
+3. **ArgoCD Application specs** can override local file changes
+4. **Disable auto-sync** when manual intervention is needed
+5. **CRD cleanup** is often required for major version upgrades
+6. **Test management cluster first** before touching remote clusters
+7. **Always fix ArgoCD applications** to prevent future issues

packages/crossplane/dev/values.yaml

@@ -1,3 +1,3 @@
 args:
   - --debug
-  - --enable-environment-configs
+  # - --enable-environment-configs  # Removed in v2.x