Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pydantic config #189

Merged
merged 53 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
362b8d5
config is now backed by pydantic (WIP)
SecretiveShell Sep 5, 2024
36e991c
automate arg parse
SecretiveShell Sep 5, 2024
8e93446
patch pydantic config into old config
SecretiveShell Sep 6, 2024
420fd84
add env var loading automation
SecretiveShell Sep 6, 2024
e8fcecd
Merge remote-tracking branch 'upstream/main' into HEAD
SecretiveShell Sep 11, 2024
0d74591
fix arg parser for dict types
SecretiveShell Sep 11, 2024
c6f9806
remove unused imports
SecretiveShell Sep 11, 2024
05f1c3e
fix line lengths
SecretiveShell Sep 11, 2024
8b48f00
fix model names
SecretiveShell Sep 12, 2024
e11d80b
fix missing rename
SecretiveShell Sep 12, 2024
eb5f42c
add error message for invalid use_as_default
SecretiveShell Sep 12, 2024
6e935c5
remove private attributes in args
SecretiveShell Sep 12, 2024
21747bf
Args: Switch to use model_field for everything
bdashore3 Sep 13, 2024
d5b3fde
Config: Fix descriptions
bdashore3 Sep 13, 2024
dc4946b
make pydantic do all the validation
SecretiveShell Sep 13, 2024
533e7c9
remove unnecessary code
SecretiveShell Sep 14, 2024
0903f85
add export openAPI to config
SecretiveShell Sep 14, 2024
a09dd80
Config: Cleanup and organize functions
bdashore3 Sep 15, 2024
6f28cfe
Logging: Remove preferences global
bdashore3 Sep 15, 2024
d013729
Config: Add aliases for logging config
bdashore3 Sep 15, 2024
5bfa952
Actions: Format
bdashore3 Sep 15, 2024
f05229b
Merge branch 'main' into pydantic-config
bdashore3 Sep 15, 2024
92af656
improve config generation action
SecretiveShell Sep 15, 2024
250d76f
Config: Alter YAML generator function
bdashore3 Sep 16, 2024
8ff9f2c
Config: Rewrite docstrings for models
bdashore3 Sep 16, 2024
4c8bb42
Config: Reorder models
bdashore3 Sep 16, 2024
3340c3b
Config: Rewrite descriptions
bdashore3 Sep 16, 2024
b6dd21f
Config: Handle default factories in config generation
bdashore3 Sep 16, 2024
564bdcf
add legacy config converter
SecretiveShell Sep 16, 2024
7f03003
rephrase info message
SecretiveShell Sep 16, 2024
81ae461
Config: Allow existing values to get included in generated file
bdashore3 Sep 16, 2024
c715094
Config: Add logging config to migration checks
bdashore3 Sep 16, 2024
e60c4ba
Config: Fix existing value check
bdashore3 Sep 16, 2024
ebe7f35
Config: Alter migration error handling and cleanup
bdashore3 Sep 16, 2024
d2d07ed
Config: Update auto-migration flow
bdashore3 Sep 16, 2024
46f9fff
Config: Move config file generation to tabby_config
bdashore3 Sep 17, 2024
06a798d
Main: Remove debug print statement for config object
bdashore3 Sep 17, 2024
26ad0ef
API: Fix model info reporting
bdashore3 Sep 17, 2024
8e6b8bd
Update .gitignore
bdashore3 Sep 17, 2024
f6fb60a
Config: Inline model loading is False
bdashore3 Sep 17, 2024
ececce1
Config: Fix addition of preamble
bdashore3 Sep 17, 2024
852ea8f
Config: Don't load from file if actions present
bdashore3 Sep 17, 2024
63f8c46
Config: Make a better description for lora config
bdashore3 Sep 17, 2024
7fe0dbd
Tree: Update config_sample
bdashore3 Sep 17, 2024
daa57ce
API: Upgrade config declarations
bdashore3 Sep 17, 2024
bb4dd72
fix defaults for api_servers
SecretiveShell Sep 17, 2024
948fcb7
migrate to ruamel.yaml
SecretiveShell Sep 18, 2024
a34bd9a
Config: Alter YAML generation script for formatting adherence
bdashore3 Sep 18, 2024
754fb15
Config: Fix draft model migration and loading
bdashore3 Sep 18, 2024
63634be
Config: Clarify Rope alpha options
bdashore3 Sep 18, 2024
6c7542d
migrate all yaml loaders to ruamel.yaml
SecretiveShell Sep 18, 2024
24ea85b
Tree: Use safe loader for YAML
bdashore3 Sep 18, 2024
4cf8551
Tree: Format
bdashore3 Sep 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Config: Allow existing values to get included in generated file
Allows for generation from an existing config file. Primarily used
for migration purposes.

Signed-off-by: kingbri <[email protected]>
  • Loading branch information
bdashore3 committed Sep 16, 2024
commit 81ae461eb84d1f1128580647aad1a73d1f4d68fd
4 changes: 2 additions & 2 deletions common/args.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from pydantic import BaseModel

from common.config_models import TabbyConfigModel
from common.utils import is_list_type, unwrap_optional
from common.utils import is_list_type, unwrap_optional_type


def add_field_to_group(group, field_name, field_type, field) -> None:
Expand Down Expand Up @@ -32,7 +32,7 @@ def init_argparser() -> argparse.ArgumentParser:

# Loop through each top-level field in the config
for field_name, field_info in TabbyConfigModel.model_fields.items():
field_type = unwrap_optional(field_info.annotation)
field_type = unwrap_optional_type(field_info.annotation)
group = parser.add_argument_group(
field_name, description=f"Arguments for {field_name}"
)
Expand Down
17 changes: 13 additions & 4 deletions common/config_models.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
from inspect import getdoc
from pathlib import Path
from pydantic import BaseModel, ConfigDict, Field, PrivateAttr
from pydantic_core import PydanticUndefined
from textwrap import dedent
from typing import List, Literal, Optional, Union

from pydantic_core import PydanticUndefined
from common.utils import unwrap

CACHE_SIZES = Literal["FP16", "Q8", "Q6", "Q4"]

Expand Down Expand Up @@ -488,12 +489,17 @@ def generate_config_file(
# You can use https://www.yamllint.com/ if you want to check your YAML formatting.\n
""")

schema = model if model else TabbyConfigModel()
schema = unwrap(model, TabbyConfigModel())

# TODO: Make the disordered iteration look cleaner
iter_once = False
for field, field_data in schema.model_fields.items():
subfield_model = field_data.default_factory()
# Fetch from the existing model class if it's passed
# Probably can use this on schema too, but play it safe
if model:
subfield_model = getattr(model, field, None)
else:
subfield_model = field_data.default_factory()

if not subfield_model._metadata.include_in_config:
continue
Expand All @@ -519,7 +525,10 @@ def generate_config_file(
else:
sub_iter_once = True

if subfield_data.default_factory:
# If a value already exists, use it
if hasattr(subfield_model, subfield):
value = getattr(subfield_model, subfield)
elif subfield_data.default_factory:
value = subfield_data.default_factory()
else:
value = subfield_data.default
Expand Down
7 changes: 5 additions & 2 deletions common/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,11 @@ def is_list_type(type_hint) -> bool:
return False


def unwrap_optional(type_hint) -> Type:
"""unwrap Optional[type] annotations"""
def unwrap_optional_type(type_hint) -> Type:
"""
Unwrap Optional[type] annotations.
This is not the same as unwrap.
"""

if get_origin(type_hint) is Union:
args = get_args(type_hint)
Expand Down
Loading