Build data-qa EKS cluster and deploy QA version of OpenMetaData into it

[feoh]

Description/Context

Exactly what it says on the tin :)

• Create the data-qa EKS cluster
• Get the Open Metadata application deployed into it

Plan/Design

Just Do It.

[feoh]

sigh

Good progress today, but I didn't get the W I wanted. Not quite an L either. Maybe a "D" :)

Spinning up the openmetadata application in QA yielded top level errors saying that the helm chart failed to initialize, offering bogus advice about consulting the CLI.

That lead me to understand the fact that my kubectl configuration was out of date in that it didn't include the data-qa cluster I spun up for this project.

Mike wrote a script to dynamically generate your kubectl config, so that's fixed.

The app still won't spin up though. I noticed the following error when running kubectl describe pod:

  Warning  Failed     33m (x8 over 35m)      kubelet            Error: secret "pgsql-db-creds" not found

Then a bunch of flailing ensued, until Mike suggested the very helpful invocation:

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/src/mit/ol-infrastructure/src/ol_infrastructure/applications/open_metadata (cpatti_omd_qa) » kubectl describe vaultdynamicsecret -n open-metadata
Name:         openmetadata-db-credentials
Namespace:    open-metadata
Labels:       pulumi_managed=true
              pulumi_stack=applications.open_metadata.QA
Annotations:  <none>
API Version:  secrets.hashicorp.com/v1beta1
Kind:         VaultDynamicSecret
Metadata:
  Creation Timestamp:  2024-10-15T19:12:28Z
  Generation:          1
  Resource Version:    1838238
  UID:                 bdf14922-e47d-443a-8f91-1806cc20cc47
Spec:
  Destination:
    Create:     true
    Name:       pgsql-db-creds
    Overwrite:  true
    Transformation:
      Excludes:
        .*
      Templates:
        DB_USER:
          Text:  {{ get .Secrets "username" }}
        DB_USER_PASSWORD:
          Text:     {{ get .Secrets "password" }}
  Mount:            postgres-open-metadata
  Path:             creds/app
  Renewal Percent:  67
  Rollout Restart Targets:
    Kind:          Deployment
    Name:          openmetadata
  Vault Auth Ref:  open-metadata-auth
Events:
  Type     Reason           Age                     From                Message
  ----     ------           ----                    ----                -------
  Warning  SecretSyncError  4m55s (x118 over 129m)  VaultDynamicSecret  (combined from similar events): Failed to sync the secret, horizon=34.101487334s, err=Error making API request.

URL: GET https://vault-qa.odl.mit.edu/v1/postgres-open-metadata/creds/app
Code: 500. Errors:

* 1 error occurred:
  * failed to execute query: ERROR: role "open_metadata" does not exist (SQLSTATE 42704)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------

Mike says we need a new Vault approle. I can't figure out how to create that. It seems like it might require a Vault CLI invocation, and we have no Vault CLI grimoire for such things. The Vault documentation offers some proposed incantations but I couldn't get any of them to work.

Tobias says he thought we weren't using approles in favor of service accounts.

I say I'm confused and look forward to being less so tomorrow :)

[feoh]

https://open-metadata-qa.ol.mit.edu is live! \o/