Make Settings Pydantic and use the power of BaseSettings to simplify CLI

[stan-dot]

Fixes #495

Using Pydantic for logic to do with file loading. It has a bit of magic but will lower our lines of code. ApplicationConfig will extend BaseSettings, while the nested configs like ScratchConfig are still BlueapiBaseModels.

[stan-dot]

File "/workspaces/blueapi/src/blueapi/service/interface.py", line 21, in <module>
INTERNALERROR>     _CONFIG: ApplicationConfig = ApplicationConfig()

now it's the time when the use of global constants bites us

[stan-dot]

@callumforrester why do we get another instance of ApplicationConfig inside the interface.py?

"""This module provides interface between web application and underlying Bluesky
context and worker"""


_CONFIG: ApplicationConfig = ApplicationConfig()


def config() -> ApplicationConfig:
    return _CONFIG


def set_config(new_config: ApplicationConfig):
    global _CONFIG

    _CONFIG = new_config

[callumforrester]

@stan-dot that's for the subprocess, you need an instance in the main process, which is passed down to the subprocess via set_config. I'm not sure why it defaults to the default config rather than None, though.

[stan-dot]

I think that the global keyword made it more confusing to me. Would SUBPROCESS_CONFIG be a correct name?

why would we want it to be None? Pydantic expects it to not be None.

I haven't worked with python subprocesses before and not sure how to debug successfully

[DiamondJoseph]

Does this handle the case of tuple[Path, ...]?

[stan-dot]

not sure, not sure if we have a test for it at the moment.

might need to add some logic to handle that

[stan-dot]

pydantic_settings defines PathType as: PathType = Union[Path, str, Sequence[Union[Path, str]]]

[DiamondJoseph]

Should: rather than setting this class variable then calling the constructor a constructor that takes a yaml file, or a class method?

e.g.

class ApplicationConfig(...):
    def __init__(self, yaml_file: Path | tuple[Path, ...] | None = None):
        self.model_config = SettingsConfigDict(env_nested_delimiter="__", yaml_file=DEFAULT_PATH or yaml_file, yaml_file_encoding="utf-8")
        super().__init__()

    #  or 
    @classmethod
    def from_yaml(cls, yaml_file: Path | tuple[Path, ...]):
          cls.model_config = yaml_field
          return cls()

[DiamondJoseph]

I'd prefer if instantiating the Config a second time isn't modified by instantiating it the first time.

[stan-dot]

there is some Pydantic magic behind the scenes. I agree that this setup is odd but that's what the docs use

https://docs.pydantic.dev/latest/concepts/pydantic_settings/#changing-priority

[stan-dot]

I'm hesitant to change that as now we're using the settings_customize_sources

[stan-dot]

in pydantic_settings model_config is specified as a ClassVariable

    model_config: ClassVar[SettingsConfigDict] = SettingsConfigDict(
        extra='forbid',
        arbitrary_types_allowed=True,
        validate_default=True,
        case_sensitive=False,
        env_prefix='',
        nested_model_default_partial_update=False,
        env_file=None,
        env_file_encoding=None,
        env_ignore_empty=False,
        env_nested_delimiter=None,
        env_parse_none_str=None,
        env_parse_enums=None,
        cli_prog_name=None,
        cli_parse_args=None,
        cli_parse_none_str=None,
        cli_hide_none_type=False,
        cli_avoid_json=False,
        cli_enforce_required=False,
        cli_use_class_docs_for_groups=False,
        cli_exit_on_error=True,
        cli_prefix='',
        cli_flag_prefix_char='-',
        cli_implicit_flags=False,
        cli_ignore_unknown_args=False,
        json_file=None,
        json_file_encoding=None,
        yaml_file=None,
        yaml_file_encoding=None,
        toml_file=None,
        secrets_dir=None,
        protected_namespaces=('model_', 'settings_'),
    )

[DiamondJoseph]

Looks like settings_customise_sources in the docs?

[DiamondJoseph]

class Settings(BaseSettings):
    foobar: str
    nested: Nested
    model_config = SettingsConfigDict(toml_file='config.toml')

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: Type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> Tuple[PydanticBaseSettingsSource, ...]:
        return (TomlConfigSettingsSource(settings_cls),)

Looking at how a toml file is used by an equivalent

And

settings_customise_sources takes four callables as arguments and returns any number of callables as a tuple. In turn these callables are called to build the inputs to the fields of the settings class.

I think you need the signature to exactly match what is shown?

[stan-dot]

yeah, it needs to be an override. Not sure why did this got changes

[callumforrester]

why would we want it to be None? Pydantic expects it to not be None.

So currently the following sequence of events happens:

1. Subprocess spawned by main process
2. config set to default config, the result of calling ApplicationConfig()
3. Main process calls set_config(), overriding the default config with whatever was in the config file

We could instead make it None at step 2 (and yes, change the type annotation), since whatever it is gets immediately erased anyway.

[stan-dot]

@callumforrester do main and subprocess share a global variable? you didn't mention where does step 2 happen - whether in main process or the subprocess

[callumforrester]

@stan-dot No they can't share a global variable. The main process passes the idea into the subprocess via set_config

[stan-dot]

The main process passes the idea into the subprocess via set_config

I thought usually it's values or references passed, not ideas...

[callumforrester]

Can you tell I haven't finished my morning coffee yet? Honestly no idea what I intended to type there, "config", maybe?

[stan-dot]

no, I cannot tell that. I'd appreciate concise technical terms used

[stan-dot]

very odd errors, trying to fix now

[stan-dot]

trying to refactor the interface to make it testable more easily.

@callumforrester it looks like unlike the patterns I can see in the codebase for worker and rest api the interface was designed to work as a callable module holding its own state, not as a class in its own.

It makes me feel confused, where can I read the ADR for this?

[stan-dot]

huh, I'd need to refactor main as well