Grid search through environment variables

[sAviOr287]

Hi all,

I am rather new to hydra but think the framework is awesome.

I have a system where I can spawn many workers with each CPU and GPU specification. Each worker can then be spawned separately and each has an environment variable that can be accessed through $ENV_VARIABLE.

The way I do grid search is usually inside the python script:

worker_parameter = int(os.environ["WORKER_ID"])
seed_vec = [9001]
lr_vec = [1e-2, 1e-3, 1e-4]

# Create a grid search vector using the itertools package
vector_grid = list(
    itertools.product(
        seed_vec,
        lr_vec
    )
)

parameters = vector_grid[int(worker_parameter)]
logging.info("Parameters:")
logging.info(parameters)
logging.info(args)
invoke_main(parameters, args)

This will then spawn for example 3 different workers where each worker will look at the different hyper parameter config. This is currently using parsers etc which i know is against what Hydra wants to do.
Hence I was wondering if there is a way to do this grid search without creating many many config files :)

Thanks a lot in advance for your help. The framework and saving files is great !

[Jasha10]

Yes, this is possible using Hydra's sweep functionality! See the docs on Multi-run.

[Jasha10]

You can use the oc.env OmegaConf resolver to lazily access environment variables from the composed DictConfig object:

# foo.yaml
worker_parameter_str: ${oc.env:WORKER_ID}  # this resolves to the string '123'
worker_parameter_int: ${oc.decode:${oc.env:WORKER_ID}}  # this resolves to the int 123

I have a system where I can spawn many workers with each CPU and GPU specification. Each worker can then be spawned separately and each has an environment variable that can be accessed through $ENV_VARIABLE.

Could you say a bit more about how your system works?

Will you be invoking the python app once on your local machine and want it to spawn a job on each of the many workers? Or were you thinking of invoking the app on each of the machines separately? Hydra's sweeper/multirun accommodates the former use-case via remote launchers (such as e.g. the submitit launcher).

[sAviOr287]

yeah the idea would be to spawn the job on each of the worker and invoking the app on each machine separately.
The thing with my setup is that I can spawn many workers that run the same code at once. But the environment variable changes for each worker so i can exploit that to do grid search.
I have tried to use the slumitit launcher but unfortunately it seems that my system is not slurm

RuntimeError: Could not detect "srun", are you indeed on a slurm cluster?

[Jasha10]

I see.
Give the oc.env resolver (mentioned in my previous comment) a try :)

[sAviOr287]

sorry for the really stupid question but once i added that in the YAML file
how would that help me to iterate over the configs in the python script?
I would still have to create the itertools grid as before i assume?
In that case wouldnt the saving structure of hydra get messed up i.e. in the different runs would have the same config saved no
Thanks again for the quick reply

[Jasha10]

Yes, you're right. Hmm...
Would you mind sharing a more complete example of how you are spawning the jobs? How does Hydra fit into your system? I'm wondering if it will be possible for you to leverage Hydra's sweeping capability without having to write a custom launcher plugin that "wraps" the worker-spawning system.