Add command anemoi-training train ...

[b8raoult]

Implement loading of hydra config in the commands/train.py and add documentation.

[gmertes]

I know this is just an early example, but fyi the token will not be stored in the user config (it will be in its own config file and generated by code with the anemoi-training mlflow login cmd, not input by the user)

[JesperDramsch]

This seems like a good step toward enabling additional functionality through user-defined configs.

I do wonder how this will work with the work @theissenhelen has started on config validation / ConfigStores, but it's probably a good prototype that we can base that work off of.

[JesperDramsch]

Is this "just an example"?

[JesperDramsch]

This should use Pathlib

[JesperDramsch]

This should probably use the logger if we want this as an output.

[JesperDramsch]

I did a deep dive into hydra and its search paths. I have a feeling we may be able to avoid some complexity (and manual merging) by implementing SearchPathPlugins from ´hydra´. They're criminally under-documented, so I dove a bit deep into what they do.

They have an example here:
https://github.com/facebookresearch/hydra/tree/1.3_branch/examples/plugins/example_searchpath_plugin

hydra searches through different config directories anyways (we just have one right now, which is the packaged one).

We can basically configure a search path plugin to search ~/.anemoi/training (or something) and the current cwd by default in addition to the package defaults. Then they still have the standard hydra option --config-dir, -cd to add a singular other dir to the search path.

Additionally, we would be able to expose a searchpath plugin in anemoi inference to also do some inferency bits if we wanted to.

Then we don't have to do any merging by hand and maintain the full functionality from before. (And we can extend the configs with struct configs and pydantic a bit later this year.)

I think the code would look something like this:

from pathlib import Path

from hydra.core.config_search_path import ConfigSearchPath
from hydra.plugins.search_path_plugin import SearchPathPlugin

class AnemoiHomeSearchPathPlugin(SearchPathPlugin):
    def manipulate_search_path(self, search_path: ConfigSearchPath) -> None:
        anemoi_home_path = Path(Path.home(), ".anemoi", "training", "config")
        if anemoi_home_path.exists():
            search_path.append(
                provider="anemoi-home-searchpath-plugin",
                path=anemoi_home_path,
            )


class UserCWDSearchPathPlugin(SearchPathPlugin):
    def manipulate_search_path(self, search_path: ConfigSearchPath) -> None:
        cwd_path = Path(Path.cwd(), "config")
        if cwd_path.exists():
            search_path.append(
                provider="anemoi-env-searchpath-plugin",
                path=cwd_path,
            )

We may also need the anemoi-training default (maybe?):

class AnemoiTrainingSearchPathPlugin(SearchPathPlugin):
    def manipulate_search_path(self, search_path: ConfigSearchPath) -> None:
        search_path.append(
            provider="anemoi-training-searchpath-plugin",
            path="pkg://anemoi-training/config",
        )

By using the hydra default we also ensure that later on we can expand the code to structured configs easily and the neat part is that technically other packages can implement additional SearchPathPlugins, such as anemoi-inference providing an inference config (or something similar).

Could be something we implement in a second iteration for this?

[FussyDuck]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mchantry]

Implement in version 0.1