Patches of nested attribute for some parameters are not being generated

[jehadnasser]

What happened?

After migrating RDS dbcluster to Upbound official provider and update the generate.yaml file accordingly, the generated composition had only the object/array name patch for the restoreToPointInTime field and ignored any nested fields underneath restoreToPointInTime. Similar behaviour happened with: s3Import, scalingConfiguration, and serverlessv2ScalingConfiguration. Unlike the normal behaviour with the old provider where there're patches for all nested attribute of similar nested fields.
Except for scalingConfiguration, the other three are new parameters, or let's say they got restructured with the Upbound provider.

How can we reproduce it?

• have the generate.yaml ready for Upbound provider and run make generate:

group: rds.aws.X.cloud
name: DBCluster
version: v1alpha1
provider:
  baseURL: https://raw.githubusercontent.com/upbound/%s/%s/package/crds/%s
  name: provider-aws
  version: v1.6.0
  crd:
    file: rds.aws.upbound.io_clusters.yaml
    version: v1beta1
ignore: false
connectionSecretKeys:
  - username
  - password
  - endpoint
  - port
compositions:
  - name: compositedbcluster.rds.aws.X.cloud
    provider: X
    default: true

• Generate script output

make generate
09:18:27 [ .. ] Generating CRDs
./generator-config.yaml
2024/06/26 09:18:37 Retrieving CRD file from acm.aws.crossplane.io_certificates.yaml
Generator for CertificateAuthority asks to be ignored, skipping...
2024/06/26 09:18:38 Retrieving CRD file from cloudwatchlogs.aws.upbound.io_groups.yaml
2024/06/26 09:18:38 Retrieving CRD file from datasync.aws.upbound.io_locations3s.yaml
2024/06/26 09:18:39 Retrieving CRD file from datasync.aws.upbound.io_tasks.yaml
2024/06/26 09:18:40 Retrieving CRD file from dms.aws.upbound.io_endpoints.yaml
2024/06/26 09:18:40 Retrieving CRD file from dms.aws.upbound.io_replicationinstances.yaml
2024/06/26 09:18:41 Retrieving CRD file from dms.aws.upbound.io_replicationsubnetgroups.yaml
2024/06/26 09:18:42 Retrieving CRD file from dms.aws.upbound.io_replicationtasks.yaml
2024/06/26 09:18:43 Retrieving CRD file from dynamodb.aws.upbound.io_tables.yaml
2024/06/26 09:18:43 Retrieving CRD file from ec2.aws.crossplane.io_instances.yaml
2024/06/26 09:18:44 Retrieving CRD file from ec2.aws.upbound.io_keypairs.yaml
2024/06/26 09:18:45 Retrieving CRD file from ec2.aws.crossplane.io_securitygroups.yaml
2024/06/26 09:18:46 Retrieving CRD file from ec2.aws.crossplane.io_securitygrouprules.yaml
2024/06/26 09:18:46 Retrieving CRD file from ecr.aws.upbound.io_repositories.yaml
2024/06/26 09:18:47 Retrieving CRD file from cache.aws.crossplane.io_cachesubnetgroups.yaml
2024/06/26 09:18:48 Retrieving CRD file from cache.aws.crossplane.io_cacheclusters.yaml
2024/06/26 09:18:49 Retrieving CRD file from cache.aws.crossplane.io_replicationgroups.yaml
2024/06/26 09:18:50 Retrieving CRD file from firehose.aws.upbound.io_deliverystreams.yaml
2024/06/26 09:18:51 Retrieving CRD file from oss.grafana.crossplane.io_dashboards.yaml
2024/06/26 09:18:51 Retrieving CRD file from oss.grafana.crossplane.io_datasources.yaml
2024/06/26 09:18:52 Retrieving CRD file from oss.grafana.crossplane.io_folders.yaml
Generator for AccessKey asks to be ignored, skipping...
2024/06/26 09:18:53 Retrieving CRD file from iam.aws.upbound.io_instanceprofiles.yaml
2024/06/26 09:18:53 Retrieving CRD file from iam.aws.crossplane.io_policies.yaml
2024/06/26 09:18:54 Retrieving CRD file from iam.aws.crossplane.io_roles.yaml
2024/06/26 09:18:55 Retrieving CRD file from iam.aws.crossplane.io_rolepolicyattachments.yaml
2024/06/26 09:18:55 Retrieving CRD file from iam.aws.upbound.io_users.yaml
2024/06/26 09:18:56 Retrieving CRD file from kafka.aws.crossplane.io_clusters.yaml
2024/06/26 09:18:57 Retrieving CRD file from kafka.aws.crossplane.io_configurations.yaml
2024/06/26 09:18:57 Retrieving CRD file from kafka.crossplane.io_providerconfigs.yaml
2024/06/26 09:18:58 Retrieving CRD file from topic.kafka.crossplane.io_topics.yaml
2024/06/26 09:19:00 Retrieving CRD file from acl.kafka.crossplane.io_accesscontrollists.yaml
2024/06/26 09:19:01 Retrieving CRD file from kms.aws.crossplane.io_aliases.yaml
2024/06/26 09:19:01 Retrieving CRD file from kms.aws.crossplane.io_keys.yaml
2024/06/26 09:19:02 Retrieving CRD file from opensearchservice.aws.crossplane.io_domains.yaml
2024/06/26 09:19:03 Retrieving CRD file from user.pagerduty.crossplane.io_contactmethods.yaml
2024/06/26 09:19:04 Retrieving CRD file from escalation.pagerduty.crossplane.io_policies.yaml
2024/06/26 09:19:05 Retrieving CRD file from extensions.pagerduty.crossplane.io_extensions.yaml
2024/06/26 09:19:06 Retrieving CRD file from service.pagerduty.crossplane.io_integrations.yaml
2024/06/26 09:19:07 Retrieving CRD file from team.pagerduty.crossplane.io_memberships.yaml
2024/06/26 09:19:07 Retrieving CRD file from user.pagerduty.crossplane.io_notificationrules.yaml
2024/06/26 09:19:08 Retrieving CRD file from schedule.pagerduty.crossplane.io_schedules.yaml
2024/06/26 09:19:09 Retrieving CRD file from tag.pagerduty.crossplane.io_tags.yaml
2024/06/26 09:19:09 Retrieving CRD file from team.pagerduty.crossplane.io_teams.yaml
2024/06/26 09:19:10 Retrieving CRD file from maintenance.pagerduty.crossplane.io_windows.yaml
2024/06/26 09:19:11 Retrieving CRD file from rds.aws.upbound.io_clusters.yaml
2024/06/26 09:19:12 Retrieving CRD file from rds.aws.crossplane.io_dbclusterparametergroups.yaml
2024/06/26 09:19:13 Retrieving CRD file from rds.aws.crossplane.io_dbinstances.yaml
2024/06/26 09:19:14 Retrieving CRD file from rds.aws.crossplane.io_dbinstanceroleassociations.yaml
2024/06/26 09:19:15 Retrieving CRD file from rds.aws.crossplane.io_dbparametergroups.yaml
2024/06/26 09:19:15 Retrieving CRD file from rds.aws.crossplane.io_optiongroups.yaml
2024/06/26 09:19:16 Retrieving CRD file from route53.aws.crossplane.io_resourcerecordsets.yaml
2024/06/26 09:19:17 Retrieving CRD file from s3.aws.crossplane.io_buckets.yaml
2024/06/26 09:19:18 Retrieving CRD file from s3.aws.crossplane.io_bucketpolicies.yaml
2024/06/26 09:19:18 Retrieving CRD file from ses.aws.upbound.io_activereceiptrulesets.yaml
2024/06/26 09:19:19 Retrieving CRD file from ses.aws.upbound.io_configurationsets.yaml
Generator for DomainDKIM asks to be ignored, skipping...
2024/06/26 09:19:20 Retrieving CRD file from ses.aws.upbound.io_domainidentities.yaml
2024/06/26 09:19:20 Retrieving CRD file from ses.aws.upbound.io_domainmailfroms.yaml
2024/06/26 09:19:21 Retrieving CRD file from ses.aws.upbound.io_emailidentities.yaml
2024/06/26 09:19:22 Retrieving CRD file from ses.aws.upbound.io_eventdestinations.yaml
2024/06/26 09:19:23 Retrieving CRD file from ses.aws.upbound.io_identitynotificationtopics.yaml
2024/06/26 09:19:23 Retrieving CRD file from ses.aws.upbound.io_identitypolicies.yaml
2024/06/26 09:19:24 Retrieving CRD file from ses.aws.upbound.io_receiptfilters.yaml
2024/06/26 09:19:25 Retrieving CRD file from ses.aws.upbound.io_receiptrules.yaml
2024/06/26 09:19:26 Retrieving CRD file from ses.aws.upbound.io_receiptrulesets.yaml
2024/06/26 09:19:27 Retrieving CRD file from ses.aws.upbound.io_templates.yaml
2024/06/26 09:19:27 Retrieving CRD file from sns.aws.upbound.io_topics.yaml
2024/06/26 09:19:28 Retrieving CRD file from sns.aws.upbound.io_topicpolicies.yaml
2024/06/26 09:19:29 Retrieving CRD file from sns.aws.upbound.io_topicsubscriptions.yaml
2024/06/26 09:19:29 Retrieving CRD file from sqs.aws.upbound.io_queues.yaml
2024/06/26 09:19:30 Retrieving CRD file from zpa.crossplane.io_applicationsegments.yaml
2024/06/26 09:19:31 Retrieving CRD file from zpa.crossplane.io_segmentgroups.yaml
09:19:31 [ OK ] Generating CRDs

• Check the generated Composition e.g for restoreToPointInTime field

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  labels:
    X.cloud/provider: X
  name: compositedbcluster.rds.aws.X.cloud
spec:
  compositeTypeRef:
    apiVersion: rds.aws.X.cloud/v1alpha1
    kind: CompositeDBCluster
  mode: Pipeline
  pipeline:
  - functionRef:
      name: function-patch-and-transform
    input:
      apiVersion: pt.fn.crossplane.io/v1beta1
      kind: Resources
      patchSets:
      # ...
      - name: Parameters
         patches:
           # This is the only generated patch for restoring db:
           - fromFieldPath: spec.forProvider.restoreToPointInTime
              policy:
                 fromFieldPath: Optional
              toFieldPath: spec.forProvider.restoreToPointInTime
              type: FromCompositeFieldPath
          # ...

whereas we expects patches for the whole new structure

- restoreToPointInTime, type array
      - restoreToTime, type string
      - restoreType, type string
      - sourceClusterIdentifier, type string
      - sourceClusterIdentifierRef, type object
        - name, type string
        - policy, type object
          - resolution, type string
          - resolve, type string
      - sourceClusterIdentifierSelector, type object
        - matchControllerRef, type boolean
        - matchLabels, type object
        - policy, type object
          - resolution, type string
          - resolve, type string
      - useLatestRestorableTime, type boolean
- s3Import, type array
  - bucketName, type string
  - bucketNameRef, type object
    - name, type string
    - policy, type object
      - resolution, type string
      - resolve, type string
  - bucketNameSelector, type object
    - matchControllerRef, type boolean
    - matchLabels, type object
    - policy, type object
      - resolution, type string
      - resolve, type string
  - bucketPrefix, type string
  - ingestionRole, type string
  - sourceEngine, type string
  - sourceEngineVersion, type string
- scalingConfiguration, type array
      - autoPause, type boolean
      - maxCapacity, type number
      - minCapacity, type number
      - secondsUntilAutoPause, type number
      - timeoutAction, type string
- serverlessv2ScalingConfiguration, type array
      - maxCapacity, type number
      - minCapacity, type number

What environment did it happen in?

• Crossplane version: v1.15.2-up.1
• Cloud provider or hardware configuration: AWS
• Kubernetes version (use kubectl version): v1.27.13-eks-3af4770
• Kubernetes distribution (e.g. Tectonic, GKE, OpenShift): EKS

[haarchri]

the patches are fine for arrays like restoreToPointInTime or scalingConfiguration

we can use the following generator:

group: rds.aws.example.cloud
name: DBCluster
version: v1alpha1
provider:
  baseURL: https://raw.githubusercontent.com/upbound/%s/%s/package/crds/%s
  name: provider-aws
  version: v1.6.0
  crd:
    file: rds.aws.upbound.io_clusters.yaml
    version: v1beta1
ignore: false
usePipeline: true # <-- set pipeline so we can test with crossplane beta render
connectionSecretKeys:
  - username
  - password
  - endpoint
  - port
compositions:
  - name: compositedbcluster.rds.aws.example.cloud
    provider: example
    default: true

lets use the following example:

apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
  namespace: assert-db
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    skipFinalSnapshot: true
    scalingConfiguration:
      - autoPause: true
        maxCapacity: 8
  writeConnectionSecretToRef:
    name: sample-rds-cluster-secret
    namespace: upbound-system

use crossplane beta render:

crossplane beta render examples/rds/db-cluster.yaml package/RDS-Cluster/composition-compositedbcluster.rds.aws.example.cloud.yaml examples/functions.yaml 
---
apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
status:
  conditions:
  - lastTransitionTime: "2024-01-01T00:00:00Z"
    message: 'Unready resources: Cluster'
    reason: Creating
    status: "False"
    type: Ready
---
apiVersion: rds.aws.upbound.io/v1beta1
kind: Cluster
metadata:
  annotations:
    crossplane.io/composition-resource-name: Cluster
  generateName: example-db-cluster-
  labels:
    commonLabelA: commonLabelAValue
    commonLabelB: commonLabelBValue
    crossplane.io/composite: example-db-cluster
  ownerReferences:
  - apiVersion: rds.aws.example.cloud/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: example-db-cluster
    uid: ""
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    scalingConfiguration:
    - autoPause: true
      maxCapacity: 8
    skipFinalSnapshot: true
  providerConfigRef:
    name: default
  writeConnectionSecretToRef:
    namespace: crossplane-system

so you can see scalingConfiguration is patched correctly

[haarchri]

please use provider-aws 1.7.0 and if its possible v1beta2 for DBCluster - check release notes for cross-reference: https://github.com/crossplane-contrib/provider-upjet-aws/releases/tag/v1.7.0

i highly recommend to use a generator config like:

group: rds.aws.example.cloud
name: DBCluster
version: v1alpha1
provider:
  baseURL: https://raw.githubusercontent.com/upbound/%s/%s/package/crds/%s
  name: provider-aws
  version: v1.7.0 # use this version to get rid of the 1 element arrays
  crd:
    file: rds.aws.upbound.io_clusters.yaml
    version: v1beta2 # bump to v1beta2
ignore: false
usePipeline: true
connectionSecretKeys:
  - username
  - password
  - endpoint
  - port
compositions:
  - name: compositedbcluster.rds.aws.example.cloud
    provider: example
    default: true

lets use the following example:

apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
  namespace: assert-db
  annotations:
    render.crossplane.io/composition-path: package/RDS-Cluster/composition-compositedbcluster.rds.aws.example.cloud.yaml
    render.crossplane.io/function-path: examples/functions.yaml
    render.crossplane.io/environment-path: test/environmentconfig.yaml
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    skipFinalSnapshot: true
    scalingConfiguration:
      autoPause: true
      maxCapacity: 8
  writeConnectionSecretToRef:
    name: sample-rds-cluster-secret
    namespace: upbound-system

use crossplane beta render:

crossplane beta render examples/rds/db-cluster.yaml package/RDS-Cluster/composition-compositedbcluster.rds.aws.example.cloud.yaml examples/functions.yaml 
---
apiVersion: rds.aws.example.cloud/v1alpha1
kind: Cluster
metadata:
  name: example-db-cluster
status:
  conditions:
  - lastTransitionTime: "2024-01-01T00:00:00Z"
    message: 'Unready resources: Cluster'
    reason: Creating
    status: "False"
    type: Ready
---
apiVersion: rds.aws.upbound.io/v1beta2
kind: Cluster
metadata:
  annotations:
    crossplane.io/composition-resource-name: Cluster
  generateName: example-db-cluster-
  labels:
    commonLabelA: commonLabelAValue
    commonLabelB: commonLabelBValue
    crossplane.io/composite: example-db-cluster
  ownerReferences:
  - apiVersion: rds.aws.example.cloud/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: example-db-cluster
    uid: ""
spec:
  forProvider:
    engine: aurora-postgresql
    masterPasswordSecretRef:
      key: password
      name: sample-cluster-password
      namespace: upbound-system
    masterUsername: cpadmin
    region: us-west-1
    scalingConfiguration:
      autoPause: true
      maxCapacity: 8
    skipFinalSnapshot: true
  providerConfigRef:
    name: default
  writeConnectionSecretToRef:
    namespace: crossplane-system