KIM: Exploratory testing

[akgalwas]

We need to perform testing of the current implementation to:

• Spot defects in the current implementation
• Verify what is missing in the current implementation
• Walk through the full flow: creating new clusters, migrating clusters
• Assess how far we are from being "production ready"

Proposed tests

• Make e2e tests for AWS (26.09.2024)
  • Functional tests
    • Create clusters with various settings
    • Verify Audit log is configured properly Problems on DEV : according to the status configured correctly but we were not able to find entries for the specific cluster in the Audit log tool
    • Verify administrator list is set correctly
    • Verify OIDC is configured properly (new implementation not merged yet)
    • Apply couple of Runtime CRs a once - its expected to have all shoots created
    • Prepare series of updates (purpose, workers.volume.size, kubernetes.version, machine.type) Problem: Updating machine version to c1.xlarge failed ; setting it back for correct value in Runtime CR didn't cause shoot spec to reconcile to the correct value (shoot was in error state, incorrect machine type was set, runtime was in Pending state)
    • Prepare runtime with incorrect values to see how the error status will be set
  • Lifecycle tests
    • Create Runtime CR, update and delete. Repeat couple of times.
    • Perform two separate updates, one by one in some short time
• Verify migration tool works correctly. Compare shoot files that were created after migrator (existent shoot->migrator->Runtime CR->new shoot) to make sure migrator does the job correctly
  • GCP - issues found, to be retested
  • AWS
  • Azure
  • Openstack
• Provision new clusters for all hyperscalers: @akgalwas
  • Use examples from documentation and make sure they are up to date
  • Read log files to verify they are descriptive enough - we need log levels
• Provision new clusters for all hyperscalers with KEB (Provisioner is the leading system, KIM only stores shoot spec)
  • Verify KIM, and Provisioner create logically equal files on DEV (mind there are some slight differences)
  • Verify KIM, and Provisioner create logically equal files on STAGE (mind there are some slight differences) In progress

[akgalwas]

Findings:

• Migrated shoot differs from the original one #342
• Shoot produced by KIM differs from the one created by Provisioner #343
• Unable to provision cluster #332 (comment)
• We need to have log levels, as for now the log is too noisy
• Currently in case Gardener error occurred the following error will be stored in status: Gardener API create error. We should add more descriptive error here (it shouldn't be detailed go error, rather description of the error category (e.g. validation error)). For now we will provide detailed error message to make things more descriptive, but it should be cleaned up at some point.

PRs:

• Fixes to make migrator up to date #329
• [WiP] Fixes to make KIM produce the same output as Provisioner #341
• Runtime CR examples updated #345

[tobiscr]

Latest findings:

• KIM skips consecutive shoot update  #416
• Auditlogs not visible in Auditlog-Viewer
• Duplicate reconciliations caused by Auditlog-State (might be fine for stage and has to be fixed before going to PROD)

[mvshao]

Audit Logs not visible in Auditlog-Viewer
During my verification, I found that the Audit Service took about 20 minutes to successfully record the action performed on the cluster in the audit log viewer.

Verification was done on DEV, cluster was created by KIM. Action (create namespace) was performed at 11:39:22 and this record was visible in Audit Log Viewer around 12:00

First test

Second test (namespace creation) was executed at 11:45:39, and record was visible in Audit Log Viewer around 12:07