👷 Review and test inconsistencies in Data Catalogue RDS variable assignment

[mitchdawson1982]

User Story

As a developer I want a consistent way of defining access to RDS database variables workflows, so that we remove unnecessary configuration and potential confusion.

Background

Datahub accesses an RDS database instance configured as part of the cloud platform deployment. When the RDS database is configured, secrets such as the username, password, database name, and host url are added to a secret store within the namespace.

We have noticed inconsistencies in that some of the parameters are being pulled from the secret store, stored in github and then sent in via the workflow. Other parameters are simply applied to the pod environment from the secret store as we would expect.

Secrets pulled from K8's

Secrets pulled from github via workflow

Data Catalogue readme

PR

Definition of Done

• Identify each RDS variable and how it is applied (via secret store or github)
• test accessing all variables directly from the RDS secret store and validate results
• where issues are experienced accessing variables from the secret store, document these inconsistencies.

[mitchdawson1982]

https://datahubproject.io/docs/deploy/aws#rds

Keys within the rds store

{
  "database_name": "",
  "database_password": "",
  "database_username": "",
  "rds_instance_address": "",
  "rds_instance_endpoint": "",
  "rds_url": ""
}

Keys applied from rds store

• database_name
• database_password
• database_name

[mitchdawson1982]

Deploying the changes gives this deployment error -

2024/12/20 10:22:09 bad hostname provided: tcp://map[secretKey:rds_instance_address. parse "tcp://map[secretKey:rds_instance_address": invalid port ":rds_instance_address" after host                                                   │
│ Stream closed EOF for data-platform-datahub-catalogue-dev/datahub-postgresql-setup-job-2lpzk (postgresql-setup-job)