Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Entire database wipe when deploying proxy #9558

Closed
aguadoenzo opened this issue Mar 26, 2025 · 2 comments · Fixed by #9637
Closed

[Bug]: Entire database wipe when deploying proxy #9558

aguadoenzo opened this issue Mar 26, 2025 · 2 comments · Fixed by #9637
Assignees
Labels
bug Something isn't working priority

Comments

@aguadoenzo
Copy link
Contributor

What happened?

This is a duplicate of #3035 but it's still happening and too big of an issue to just leave a comment on a closed issue.

Pointing LiteLLM at your database will result in data loss when you re-deploy the LiteLLM proxy.

Here are some logs I found:

⚠️ There might be data loss when applying the changes: 
• You are about to drop the `migratehistory` table, which is not empty (4 rows).
• You are about to drop the `tenant` table, which is not empty (1 rows).
• You are about to drop the `user` table, which is not empty (1 rows).

This is due to LiteLLM blindly updating the schema without taking prisma's behavior and warning into consideration.
See migration script

This is extremely dangerous and will (but more likely already has) result in production data being lost.

The other issue suggests pointing LiteLLM at a different database. While the advice is sound and valid, this is not enough, especially considering none of this is mentioned in the docs

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

latest

Twitter / LinkedIn details

No response

@aguadoenzo aguadoenzo added the bug Something isn't working label Mar 26, 2025
@ishaan-jaff
Copy link
Contributor

cc @krrishdholakia I believe your prisma migration work will resolve this issue

@krrishdholakia
Copy link
Contributor

krrishdholakia commented Mar 27, 2025

Hi @aguadoenzo thank you for raising this issue. We've added support for using prisma migration here

litellm --use_prisma_migrate

# runs `prisma migrate deploy` 

We've also added the initial migration baseline + new updates here - https://github.com/BerriAI/litellm/tree/3ca34a181cae83b8b3c73e5937f07e36509a7f35/deploy/migrations

By end of week, we will have these 2 components connected, so you can just use the safer prisma migrate flow.

I will update this ticket once it's done e2e.

@krrishdholakia krrishdholakia self-assigned this Mar 27, 2025
krrishdholakia added a commit that referenced this issue Mar 29, 2025
…prisma migration files (#9637)

* build(README.md): initial commit adding a separate folder for additional proxy files. Meant to reduce size of core package

* build(litellm-proxy-extras/): new pip package for storing migration files

allows litellm proxy to use migration files, without adding them to core repo

* build(litellm-proxy-extras/): cleanup pyproject.toml

* build: move prisma migration files inside new proxy extras package

* build(run_migration.py): update script to write to correct folder

* build(proxy_cli.py): load in migration files from litellm-proxy-extras

Closes #9558

* build: add MIT license to litellm-proxy-extras

* test: update test

* fix: fix schema

* bump: version 0.1.0 → 0.1.1

* build(publish-proxy-extras.sh): add script for publishing new proxy-extras version

* build(liccheck.ini): add litellm-proxy-extras to authorized packages

* fix(litellm-proxy-extras/utils.py): move prisma migrate logic inside extra proxy pkg

easier since migrations folder already there

* build(pre-commit-config.yaml): add litellm_proxy_extras to ci tests

* docs(config_settings.md): document new env var

* build(pyproject.toml): bump relevant files when litellm-proxy-extras version changed

* build(pre-commit-config.yaml): run poetry check on litellm-proxy-extras as well
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working priority
Projects
None yet
3 participants