Can functional expressions replace the defaults list?

[greaber]

This started as a comment on #2227 that @Jasha10 asked me to open a discussion on.

Hydra allows a kind of functional expression in config in the form of OmegaConf interpolations, but these cannot evaluate to further nested config but just to a value for a config key. It seemed to me that allowing more powerful functional expressions could obviate the need for the defaults list and solve the issue that currently you are forced to choose between using the defaults list and normal config depending on whether you want overriding to work by replacement or merging.

To repeat my comment, what if (instead of using the defaults list) you could do something like

name: 'root_finder'
algorithm:
  _target_: alg.GradientBased
  optimizer: @{optimizers.adam}

optimizers:
  adam:
    _target_: optim.Adam
    lr: 0.3
    eps: 1e-5
  newton:
    _target_: optim.Newton
    lr: 0.4
    eps: 0.0001
    num_iterations: 1000

Or, if you wanted to reuse these optimizer configs in different places, you might have an optimizers.yaml file, and then you would put the contents of the optimizers key there and then do:

name: 'root_finder'
algorithm:
  _target_: alg.GradientBased
  optimizer: @{optimizers/adam}

...

Or, if you needed to override the learning rate,

name: 'root_finder'
algorithm:
  _target_: alg.GradientBased
  optimizer: @{merge(_optimizers/adam, optimizer_overrides)}

_optimizer_overrides:
  lr: 0.1

...

Due to the limitations of yaml syntax, it probably makes sense to not allow nesting dict expressions inside "functional" expressions but instead require them to be assigned to a variable, like _optimizer_overrides. Also, I'm using the convention here that underscored config is for internal use, but there could be a way to make _optimizer_overrides actually not be in the output config (if that is not already possible).

Note also that here I use @{optimizers/adam} to mean the adam key in the optimizers.yaml file in the current directory. If optimizers.yaml were in the training subdirectory of the current directory, you could use @{training/optimizers/adam}. And you could use .. and absolute paths as well. To reduce ambiguity, it might be better to use : instead of / to separate the path to the yaml file from key inside the file, i.e., @{optimizers:adam} instead of @{optimizers/adam}.

[Jasha10]

"Can functional expressions replace the defaults list" is a good question to ask. The short answer is "yes, partially".

Hydra's defaults lists integrate two core functionalities:

• config merging and organization
• config discovery and loading

The "config merging and organization" part can certainly be handled by functional expressions. The functional operators can be implemented using OmegaConf's variable interpolations and custom resolvers.

The "config discovery and loading" part would be trickier to implement using pure functional expressions. The reason is that Hydra's config discovery/loading depends on program state (see Config Search Path docs). For example, a defaults list defaults: [foo: bar] will trigger Hydra's config loader to search for a config named "bar" in a group named "foo"; the locations searched will differ depending on program state.

If you are only interested in loading configs from local yaml files or perhaps from interpolations to other config keys, then an expression-based functional replacement for the defaults list is possible.

Below I'll illustrate the use of OmegaConf's custom resolvers to implement the workflow you proposed.

We'll define a few yaml files config.yaml and optimizers.yaml, as well as a python file app.py:

# optimizers.yaml
adam:
  _target_: optim.Adam
  lr: 0.3
  eps: 1e-5

# config.yaml
name: 'root_finder'
algorithm:
  _target_: alg.GradientBased
  # Below we use custom resolvers `load` and `merge`, implemented in app.py
  optimizer: ${merge:${load:optimizers,adam},${_optimizer_overrides}}

_optimizer_overrides:
  lr: 0.1

# app.py
from omegaconf import OmegaConf

# Here we implement the custom resolvers `load` and `merge`:

def _load_impl(file_name: str, key=None):
    """
    Load yaml file `file_name` from disk. Append suffix ".yaml" to `file_name` if necessary.
    If `key` is given, select `key` from the loaded config.
    """

    # load the yaml file from disk
    if not file_name.endswith(".yaml"):
        file_name = f"{file_name}.yaml"
    cfg = OmegaConf.load(file_name)

    # select the given key from the loaded config
    if key is not None:
        return OmegaConf.select(cfg, key, throw_on_missing=True)
    else:
        return cfg


def _merge_impl(*args):
    return OmegaConf.merge(*args)


OmegaConf.register_new_resolver("load", _load_impl)
OmegaConf.register_new_resolver("merge", _merge_impl)

# Now we load `config.yaml` and print the result.
conf = OmegaConf.load("config.yaml")  # OR @hydra.main(config_path=".", config_name="config")
OmegaConf.resolve(conf)
print(OmegaConf.to_yaml(conf))

Running the app yields the following result:

$ python app.py
name: root_finder
algorithm:
  _target_: alg.GradientBased
  optimizer:
    _target_: optim.Adam
    lr: 0.1
    eps: 1.0e-05
_optimizer_overrides:
  lr: 0.1

[jzazo]

@Jasha10, when you say "The reason is that Hydra's config discovery/loading depends on program state", and refer to search spaces, you mean that the expression need to exist in any search space, whereas functional resolution limits search spaces to a single one? I am not sure I understand.

Also, would this functional resolution work with structured schemas and command line overrides?

What functionality would exactly be limited by functional resolution?

[Jasha10]

Also, would this functional resolution work with structured schemas and command line overrides?

Merging structured schemas should work: ${merge:${my_schema},${some_data}}.

Command-line overrides will be trickier. This comes down to the classic distinction between "pure functions" vs "functions with side effects". If we want to use command-line overrides, we will need the output of our custom resolvers to depend on the command line. This means implementing a resolver that inspects sys.argv or that uses argparse. Hydra's CLI functionality is advanced in this regard; it is theoretically possible to re-implement Hydra's command-line overrides using OmegaConf resolvers, but the resolver implementation would become quite complicated :P

when you say "The reason is that Hydra's config discovery/loading depends on program state", and refer to search spaces, you mean that the expression need to exist in any search space, whereas functional resolution limits search spaces to a single one?

Yes... and the set of places where Hydra looks for configs can change dynamically.

Hydra looks for configs:

• on disk
• in the ConfigStore (which can change if you call cs.store in your code)
• possibly in other places if there is a SearchPathPlugin

My _load_impl function above only looks for configs in one place: on disk. In theory the _load_impl function could be extended to also look in a global ConfigStore object (though ConfigStore does not currently have a public API for inspecting it's state, so this would require some hacking). The result of the ${load:...} resolver would depend on the state of the global config store at the time when the ${load:...} resolver is invoked.

What functionality would exactly be limited by functional resolution?

Some simple functionality can be easily implemented using OmegaConf resolvers (e.g. merging configs).
More advanced functionality (e.g. CLI overrides) could theoretically be implemented, but the resolver implementations would probably be just as complicated as the Hydra codebase.