Add `models import` command #10581

msabramo · 2025-05-06T01:47:45Z

Add a models import command that can import models from a LiteLLM proxy YAML config file.

This doesn't handle shared LLM credentials yet, which I'd like to eventually support.

Examples of usage

$ litellm-proxy models import \
  --only-access-groups-matching-regex='default-models' \
  --dry-run \
  ~/work/dmg/litellm_proxy_config.yaml
                                       Models that would be imported if --dry-run was not provided
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Model Name                                ┃ Upstream Model                                                           ┃ Access Groups  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ gemini-2.5-flash-preview-04-17            │ vertex_ai/gemini-2.5-flash-preview-04-17                                 │ default-models │
│ gpt-4o-2024-05-13-managed                 │ azure/gpt-4o-2024-05-13-prod-filter                                      │ default-models │
│ gpt-4o-2024-05-13-no-filter               │ azure/gpt-4o-2024-05-13-no-filter                                        │ default-models │
...
└───────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────┴────────────────┘
Model Import Summary
┏━━━━━━━━━━━┳━━━━━━━┓
┃ Provider  ┃ Count ┃
┡━━━━━━━━━━━╇━━━━━━━┩
│ vertex_ai │ 11    │
│ azure     │ 14    │
│ bedrock   │ 26    │
│ azure_ai  │ 5     │
│ gemini    │ 1     │
│ sagemaker │ 2     │
│ Total     │ 59    │
└───────────┴───────┘

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
I have added a screenshot of my new test passing locally
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature

Changes

vercel · 2025-05-06T01:47:49Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 9, 2025 8:04pm

msabramo · 2025-05-09T14:53:10Z

Hmmm did unit tests not run? Because I just noticed they're failing for me locally 😄

msabramo · 2025-05-09T14:57:46Z

Hmmm did unit tests not run? Because I just noticed they're failing for me locally 😄

Oh maybe because main is not the target branch?

CLAassistant · 2025-05-09T15:09:29Z

All committers have signed the CLA.

msabramo · 2025-05-09T15:10:46Z

Hmmm did unit tests not run? Because I just noticed they're failing for me locally 😄

Oh maybe because main is not the target branch?

OK I merged from main into my branch (and fixed conflicts) and then changed the target branch to main, but I also fixed the test failure, so the unit tests should hopefully run and pass...

msabramo · 2025-05-09T17:00:11Z

Hmmm, something seems stuck?

msabramo · 2025-05-09T17:19:53Z

Just pushed an empty commit to see if the CI reruns and is happier...

to `models import` which only processes models where `litelllm_params.model` matches the regex

to `models import` which only processes models where at least one item in `model_info.access_groups` matches the regex. Add a unit test.

…aml_file`

msabramo · 2025-05-09T18:04:22Z

Did some refactoring and rebased and removed the empty commit

for better understandability and Intellisense

msabramo · 2025-05-09T20:11:30Z

Cc: @ishaan-jaff

krrishdholakia · 2025-05-10T15:48:32Z

litellm/proxy/client/cli/README.md

@@ -109,6 +109,52 @@ Options:
 - `--param`, `-p`: Model parameters in key=value format (can be specified multiple times)
 - `--info`, `-i`: Model info in key=value format (can be specified multiple times)

+#### Import Models
+
+Import models from a YAML file:


Hey when would you want to import from a yaml?

I don't understand what the user is trying to achieve with this

The use case is one I think I mentioned to you. We have a bunch of models in the YAML config for our self-hosted instance and this would help us bring them over to the cloud instance. The filtering options let us do it gradually.

Hey @msabramo where does this porting from yaml to db happen in the code? from what i can see it gets appended to a model list, but i didn't see a /model/new call in this PR

client.models.new call

litellm/litellm/proxy/client/cli/commands/models.py

Lines 424 to 428 in ae367da

client.models.new(

model_name=model_info_obj.model_name,

model_params=model_info_obj.model_params,

model_info=model_info_obj.model_info,

)

vercel bot deployed to Preview May 6, 2025 01:48 View deployment

vercel bot deployed to Preview May 6, 2025 01:52 View deployment

msabramo force-pushed the proxy-client-cli-models-import branch from 757cc6b to 1dc0ceb Compare May 6, 2025 01:53

vercel bot deployed to Preview May 6, 2025 01:54 View deployment

msabramo force-pushed the proxy-client-cli-models-import branch from 1dc0ceb to dd67030 Compare May 6, 2025 01:56

vercel bot deployed to Preview May 6, 2025 01:57 View deployment

vercel bot deployed to Preview May 7, 2025 02:52 View deployment

vercel bot deployed to Preview May 7, 2025 04:05 View deployment

vercel bot deployed to Preview May 7, 2025 05:47 View deployment

vercel bot deployed to Preview May 7, 2025 05:51 View deployment

vercel bot deployed to Preview May 7, 2025 15:29 View deployment

vercel bot deployed to Preview May 9, 2025 14:57 View deployment

msabramo changed the base branch from litellm_proxy_cli_staging to main May 9, 2025 14:58

msabramo changed the base branch from main to litellm_proxy_cli_staging May 9, 2025 14:58

vercel bot deployed to Preview May 9, 2025 15:01 View deployment

msabramo force-pushed the proxy-client-cli-models-import branch from 3e74b82 to 6a0ee95 Compare May 9, 2025 15:09

msabramo changed the base branch from litellm_proxy_cli_staging to main May 9, 2025 15:09

vercel bot deployed to Preview May 9, 2025 15:10 View deployment

vercel bot deployed to Preview May 9, 2025 17:20 View deployment

msabramo added 4 commits May 9, 2025 11:03

Add --only-models-matching-regex option

4755e84

to `models import` which only processes models where `litelllm_params.model` matches the regex

Add test_models_import_only_models_matching_regex

2b1e335

Print each model we're importing

a3ba698

Add --only-access-groups-matching-regex option

5065e1d

to `models import` which only processes models where at least one item in `model_info.access_groups` matches the regex. Add a unit test.

msabramo added 6 commits May 9, 2025 11:03

ruff format litellm/proxy/client/cli/commands/models.py

c6243e4

Make models import display tabular output

c3bd23b

models import refactoring

63820f4

Fix failing tests in test_models_commands.py

a8ce0e1

Refactor import_models to make it shorter and more readable

e8ad04d

Extract from import_models a function called `get_model_list_from_y…

2c85fef

…aml_file`

msabramo force-pushed the proxy-client-cli-models-import branch from 604dba0 to 2c85fef Compare May 9, 2025 18:03

vercel bot deployed to Preview May 9, 2025 18:05 View deployment

Fix mypy error

80dcfca

vercel bot deployed to Preview May 9, 2025 18:16 View deployment

Add more specific typing

1df0e09

for better understandability and Intellisense

vercel bot deployed to Preview May 9, 2025 18:36 View deployment

More import_models refactoring

bfc0254

vercel bot deployed to Preview May 9, 2025 18:42 View deployment

More refactoring

4a073ed

vercel bot deployed to Preview May 9, 2025 19:05 View deployment

More refactoring

45b2d41

vercel bot deployed to Preview May 9, 2025 19:10 View deployment

Write unit tests for format_iso_datetime_str

6779144

vercel bot deployed to Preview May 9, 2025 19:25 View deployment

msabramo added 3 commits May 9, 2025 12:30

Add more unit tests

68ec88a

ruff format tests/litellm/proxy/client/cli/test_models_commands.py

a06f3fd

ruff format litellm/proxy/client/cli/commands/models.py

349a4bd

vercel bot deployed to Preview May 9, 2025 19:35 View deployment

Make test_format_timestamp use UTC time

ae367da

vercel bot deployed to Preview May 9, 2025 20:04 View deployment

krrishdholakia reviewed May 10, 2025

View reviewed changes

krrishdholakia merged commit 322b678 into BerriAI:main May 15, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add `models import` command #10581

Add `models import` command #10581

Uh oh!

msabramo commented May 6, 2025 •

edited

Loading

Uh oh!

vercel bot commented May 6, 2025 •

edited

Loading

Uh oh!

msabramo commented May 9, 2025 •

edited

Loading

Uh oh!

msabramo commented May 9, 2025

Uh oh!

CLAassistant commented May 9, 2025 •

edited

Loading

Uh oh!

msabramo commented May 9, 2025 •

edited

Loading

Uh oh!

msabramo commented May 9, 2025

Uh oh!

msabramo commented May 9, 2025

Uh oh!

msabramo commented May 9, 2025

Uh oh!

msabramo commented May 9, 2025

Uh oh!

krrishdholakia May 10, 2025

Uh oh!

msabramo May 10, 2025

Uh oh!

krrishdholakia May 10, 2025 •

edited

Loading

Uh oh!

msabramo May 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

	client.models.new(
	model_name=model_info_obj.model_name,
	model_params=model_info_obj.model_params,
	model_info=model_info_obj.model_info,
	)

Uh oh!

Add models import command #10581

Add models import command #10581

Uh oh!

Conversation

msabramo commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Examples of usage

Relevant issues

Pre-Submission checklist

Type

Changes

Uh oh!

vercel bot commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msabramo commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msabramo commented May 9, 2025

Uh oh!

CLAassistant commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msabramo commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msabramo commented May 9, 2025

Uh oh!

msabramo commented May 9, 2025

Uh oh!

msabramo commented May 9, 2025

Uh oh!

msabramo commented May 9, 2025

Uh oh!

krrishdholakia May 10, 2025

Choose a reason for hiding this comment

Uh oh!

msabramo May 10, 2025

Choose a reason for hiding this comment

Uh oh!

krrishdholakia May 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msabramo May 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Add `models import` command #10581

Add `models import` command #10581

msabramo commented May 6, 2025 •

edited

Loading

vercel bot commented May 6, 2025 •

edited

Loading

msabramo commented May 9, 2025 •

edited

Loading

CLAassistant commented May 9, 2025 •

edited

Loading

msabramo commented May 9, 2025 •

edited

Loading

krrishdholakia May 10, 2025 •

edited

Loading

msabramo May 11, 2025 •

edited

Loading