Deploying determined ai on premise kubernetes cluster with custom registry for the postgres db pod results in forbidden error while pulling image🐛[bug]

[humbleearth]

Describe the bug

The postgres db pod image does not download with custom registry.

Reproduction Steps

1. Download the latest helm code
2. Update values.yaml file with imagePullSecretName value and defaultImages:
   imageRegistry: sample.registry.custom.com:5000/determinedai

PostgreSQL image

postgreSQL: "sample.registry.custom.com:5000/postgres:10.14"
4. helm install

The master image downloads but while downloading the postgresql image results in below error:

Error inside db pod:
Warning Failed 6s (x3 over 46s) kubelet Failed to pull image
"sample.registry.custom.com:5000/postgres:10.14": rpc error: code = Unknown desc = failed to pull and unpack image "sample.registry.custom.com:5000/postgres:10.14": failed to resolve reference "sample.registry.custom.com:5000/postgres:10.14": pulling from hostsample.registry.custom.com:5000 failed with status code [manifests
10.14]: 403 Forbidden

Expected Behavior

It should pull both the master and postgres images

Screenshot

Environment

• Device or hardware: k8s
• OS: linux
• Browser n/a
• Version 0.24.0

Additional Context

No response

[ioga]

Hello,

Sorry about that. postgres is a public image, it's uncommon to use a private registry for it and we've never hit this before.

As a workaround, you can change the db deployment to set imagePullSecrets the same way master deployment does it.

[tpanza]

hi @ioga . In a corporate environment, we have to route all container image requests through an on-prem Artifactory server that sits between the corporate network and the outside world. This includes even public images like postgres

Could you please consider updating the helm chart to provide a "global" imagePullSecret that is used for all image pulls?

Further, for all of the images (master, postgreSQL, kubeScheduler, kubeSchedulerPreemption, cpuImage, etc), please consider having the Helm chart break that up into 3 separate editable fields: registry, repository, tag.

Then in this scenario, we would only override the registry for each image to point to the Artifactory proxy server, and leave the repository and tag as defaults

[ioga]

Your request makes sense to me. I've added an internal tickets to address these two points.

if you end up forking the helm chart and implementing this, while keeping the backwards compatibility with the existing defaultImages & imageRegistry options, we'd be happy to take a PR.