Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: migration fails when .Values.db.useExisting is true #6791

Closed
stevencrake-nscale opened this issue Nov 18, 2024 · 0 comments · May be fixed by #6792
Closed

[Bug]: migration fails when .Values.db.useExisting is true #6791

stevencrake-nscale opened this issue Nov 18, 2024 · 0 comments · May be fixed by #6792
Labels
bug Something isn't working

Comments

@stevencrake-nscale
Copy link

stevencrake-nscale commented Nov 18, 2024 •
edited
Loading

What happened?

When .Values.db.useExisting is true, the migration job is unable to connect to the DB, because the .Values.db.url references env vars that the migration job does not have access to. As a result, the env vars are not replaced, and the url looks like postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME) verbatim.

This is because the migration template does not include them, whereas the deployment template does.

I propose that we simply add the missing env vars to the migration template.

Relevant log output

kubectl logs litellm-migrations-xw76p -n litellm --previous
Attempt 1...
Retrying in 10 seconds...
Attempt 2...
Retrying in 10 seconds...
Attempt 3...
Unable to push database changes after 3 retries.


Shelling into the migration job highlights the missing env vars:

root@litellm-migrations-xw76p:/app# env | sort
DATABASE_URL=postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)
DISABLE_SCHEMA_UPDATE=false
...


After this change applied:

kubectl logs litellm-migrations-nw7gh -n litellm
Attempt 1...
Database push successful!

Twitter / LinkedIn details

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Add credential templates in migration job when using existing DB stevencrake-nscale/litellm
1 participant
@stevencrake-nscale