How to access all config combinations from multi-run sweep in one "run"?

[ethnhll]

I want to use Hydra to gather the search space/set of hyper-parameters that would result from a multi-run, and access the whole sweep in one "run" so that I can pass off to a function in PySpark (ParamGridBuilder).
From my limited look at this library so far, it seems like I need to write my own sweeper logic (and possibly a launcher as well), but I am not sure if that is the right approach. Any advice, or examples of a canonical way to do this would be appreciated!

[Jasha10]

Hi @ethnhll,
It is the given sweeper plugin that "decides" which hyperparamenters should be included in the sweep.

This means different sweeper plugins may make different decisions / employ different logic to decide what jobs will be included in the sweep. For example, the BasicSweeper plugin (which is Hydra's default sweeper) sweeps over the product space of sweep parameters (doing a regular grid search), whereas the OptunaSweeper uses optimization techniques and feedback from earlier jobs to dynamically decide what jobs to run.

The launcher plugin is downstream of the sweeper: while the sweeper decides what hyperparameter settings should be used, the launcher is responsible for turning each set of chosen hyperparams into an actual, running job.

If you're interested in using existing logic from one of Hydra's sweepers, you could write a launcher plugin that, instead of creating/running a job per realized hyperparam setting, will instead simply log those hyperparam settings (giving a record of what jobs would have been launched had a standard launcher been used).

[ethnhll]

Is there a method with using the BasicSweeper where I can see the entire product space of params from my app's code? If I have the entire generated config space from the sweep, I can pass that "grid" off to ParamGridBuilder with the basic launcher.

[Jasha10]

One idea is to create a minimal @hydra.main-decorated function whose sole purpose is to access the output from the sweeper. You can then persist the configs that are passed to that function.
Here's a concrete example:

$ cat app.py 
import logging

import hydra

log = logging.getLogger(__name__)


@hydra.main(version_base=None)
def app(cfg):
    log.info(f"Got config: {cfg}")  # you can persist the config to a database or file on disk

if __name__ == "__main__":
    app()

$ python app.py -m +foo=bar,baz,qux
[2022-12-27 11:14:05,573][HYDRA] Launching 3 jobs locally
[2022-12-27 11:14:05,573][HYDRA] 	#0 : +foo=bar
[2022-12-27 11:14:05,668][__main__][INFO] - Got config: {'foo': 'bar'}
[2022-12-27 11:14:05,669][HYDRA] 	#1 : +foo=baz
[2022-12-27 11:14:05,771][__main__][INFO] - Got config: {'foo': 'baz'}
[2022-12-27 11:14:05,771][HYDRA] 	#2 : +foo=qux
[2022-12-27 11:14:05,860][__main__][INFO] - Got config: {'foo': 'qux'}

All of the configs from the sweep are passed to the app function in succession.

[ethnhll]

Is it possible to create a custom sweeper that passes all of the configs to the app function together in one "run"?

[Jasha10]

In theory that should be possible. (Such a sweeper would not be consistent with Hydra's existing conventions around the division of labor between sweepers and launchers, but that might not be of concern given your use-case.)

The sweeper plugin API is open-ended: Hydra passes some data to the sweeper (via the sweep method, defined by this API) and it is up to the sweeper to decide how that data would be processed. The convention in Hydra's existing plugins is that the sweeper is responsible for calling the launcher and passing batched data to the launcher. If you create a custom sweeper plugin, you'd be free to processes the data passed from Hydra to compute the set of all configs from the sweep and then pass those computed configs directly to the @hydra.main-decorated function.

If you want to create such a plugin, I'd recommend familiarizing yourself with how the BasicSweeper and the BasicLauncher work, and using those as a starting point.

[Jasha10]

A simpler way to collect multirun configs might be to use the launch function from the hydra-zen library:

from hydra_zen import launch
from omegaconf import DictConfig

configs: list[DictConfig] = []


def append_cfg_to_list(cfg: DictConfig) -> None:
    configs.append(cfg)


launch(
    config={"foo": "???"},
    multirun=True,
    task_function=append_cfg_to_list,
    overrides=["foo=bar,baz,qux"],
)

print(configs)  # prints [{'foo': 'bar'}, {'foo': 'baz'}, {'foo': 'qux'}]