Environment-dir issue on its stack-eg code

[fanjiaxin23]

I am having issue to create the yaml and db file like this: https://hydra.cc/docs/tutorials/basic/your_first_app/simple_cli/
https://hydra.cc/docs/intro/

• Created conf dir
• run my_app.py code followed by the tutorial but its not creating yaml file rather throwing error as below:

$ python my_app.py +db.driver=mysql +db.user=omry +db.password=secret
Traceback (most recent call last):
  File "my_app.py", line 13, in <module>
    my_app()
......
hydra.errors.MissingConfigException: Cannot find primary config 'config'. Check that it's in your config search path.

Config search path:
	provider=hydra, path=pkg://hydra.conf
	provider=main, path=file:///hydra/examples/tutorials/basic/your_first_hydra_app/1_simple_cli/conf
	provider=schema, path=structured://

[Jasha10]

See the error message "hydra.errors.MissingConfigException: Cannot find primary config 'config'. Check that it's in your config search path" above.
Do you have a file called conf/config.yaml next to your my_app.py file?

[fanjiaxin23]

Thanks for the response. I actually created the conf dir and did not created config.yaml manually? Does it need to be created manually? Also do I need to add some yaml contents inside to make it works?

i thought the code will allow it to create that yaml file and use the following command$ python my_app.py +db.driver=mysql +db.user=omry +db.password=secret to add those on that config file, seems I am doing it wrongly?

My Directory looks like this:

- conf
   |-- (empty dir)
- my_app.py
- ouput

[fanjiaxin23]

Update:
I followed this link: https://hydra.cc/docs/tutorials/basic/your_first_app/simple_cli/
Keep the same code as the link, created a conf dir and inside created a config.yaml file; then run the my_app.py script accordingly, it worked but it did not saved the output that generated from $ python my_app.py +db.driver=mysql +db.user=omry +db.password=secret line.

I alse typed python my_app.py it shows something like this on output: {} Any suggestion how to make it work?

[Jasha10]

Check to see if a folder called outputs has been created next to your my_app.py file. Some log files and other records should be saved there automatically (inside of a dated subdirectory).

If you want to skip using the conf/config.yaml file, you can write @hydra.main(version_base=None, config_path="conf", config_name=None) instead of @hydra.main(version_base=None, config_path="conf", config_name="config").

[fanjiaxin23]

Check to see if a folder called outputs has been created next to your my_app.py file. Some log files and other records should be saved there automatically (inside of a dated subdirectory).

The output dir created some file and log file, but when I cat to open the file, but it shows empty.

If you want to skip using the conf/config.yaml file, you can write @hydra.main(version_base=None, config_path="conf", config_name=None) instead of @hydra.main(version_base=None, config_path="conf", config_name="config").

• I tried this, and used $ python my_app.py +db.driver=mysql +db.user=omry +db.password=secret and shows the output but it does not save this on conf/config.yaml file, I even removed config.yaml file under conf dir to see, nothing created automatically:) Its not working for me well:)

$ python my_app.py +db.driver=mysql +db.user=omry +db.password=secret
db:
  driver: mysql
  user: omry
  password: secret

This above output not saving anywhere in conf/config.yaml neither on output dir log file, all empty:)

Some more example:

$ ls
conf  my_app.py
hydra/examples/tutorials/basic/your_first_hydra_app/4_config_groups$ cat my_app.py 
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
from omegaconf import DictConfig, OmegaConf

import hydra

@hydra.main(version_base=None, config_path="conf")
def my_app(cfg: DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()

~/hydra/examples/tutorials/basic/your_first_hydra_app/4_config_groups$ python my_app.py 
{}

[jieru-hu]

if you are running the example application, it makes sense the log file is empty - there is no logging in the example app.
see this for a logging example

the conf yaml files are by default saved in .hydra folder,

 % tree -a .
.
└── 2022-07-19
    └── 12-52-57
        ├── .hydra
        │   ├── config.yaml
        │   ├── hydra.yaml
        │   └── overrides.yaml
        └── my_app.log

3 directories, 4 files

[fanjiaxin23]

Lets say, I have my_app.py code as the docs in a dir, inside I create a conf dir and a file inside of it renamed config.yaml. Now If I run the my_app.py code, isn't it going to save the output of python code argument automatically inside the conf/config.yaml file?

[jieru-hu]

Lets say, I have my_app.py code as the docs in a dir, inside I create a conf dir and a file inside of it renamed config.yaml. Now If I run the my_app.py code, isn't it going to save the output of python code argument automatically inside the conf/config.yaml file?

you mean the composed config that Hydra passes in to your python application? No, Hydra will not save that. all the configs will be saved in the .hydra folder in the output dir.

[fanjiaxin23]

Lets say, I have my_app.py code as the docs in a dir, inside I create a conf dir and a file inside of it renamed config.yaml. Now If I run the my_app.py code, isn't it going to save the output of python code argument automatically inside the conf/config.yaml file?

you mean the composed config that Hydra passes in to your python application? No, Hydra will not save that. all the configs will be saved in the .hydra folder in the output dir.

I do have the output dir but I do not see any .hydra dir which contain .yaml file, when i run the script it should save its log to their but nothing there, also └── my_app.log showing nothing there:)

[fanjiaxin23]

if you are running the example application, it makes sense the log file is empty - there is no logging in the example app. see this for a logging example

the conf yaml files are by default saved in .hydra folder,

 % tree -a .
.
└── 2022-07-19
    └── 12-52-57
        ├── .hydra
        │   ├── config.yaml
        │   ├── hydra.yaml
        │   └── overrides.yaml
        └── my_app.log

3 directories, 4 files

I followed the same code, and i typed python my_app.py and it did created output dir but its not showing any .hydra dir but seems like its created some hidden yaml files inside of it. when i typed tree -a .. but there's nothing inside those yaml files? How can one add some contents/config inside of it?

[fanjiaxin23]

@jieru-hu May I know how I can check the correct config_path= to setup on the code, for example the config.yaml are located at this tree structured:

 % tree -a .
. output
└── 2022-07-19
    └── 12-52-57
        ├── .hydra
        │   ├── config.yaml
        │   ├── hydra.yaml
        │   └── overrides.yaml
        └── my_app.log
my_app.py
4 directories, 5 files

@hydra.main(version_base=None, config_path=".", config_name="config")
In this line of code, I can set config_path="/.hydra" config_name="config.yaml right. Not sure how to set the path,as because of this I am getting an error message when typed python my_app.py

Error like:

hydra.errors.MissingConfigException: Cannot find primary config 'config'. Check that it's in your config search path.

Config search path:
	provider=hydra, path=pkg://hydra.conf
	provider=main, path=file:///hydra/examples/tutorials/basic/your_first_hydra_app/1_simple_cli/conf
	provider=schema, path=structured://

[jieru-hu]

I do have the output dir but I do not see any .hydra dir which contain .yaml file, when i run the script it should save its log to their but nothing there, also └── my_app.log showing nothing there:)

as i mentioned, the particular example you are running does not actually log anything, so there's nothing in my_app.log. if you want to try out the logging, pls run the other examples i've linked above.

@jieru-hu May I know how I can check the correct config_path= to setup on the code, for example the config.yaml are located at this tree structured:

config_path takes in a relative path to your application code, config_path="/.hydra" is not a correct relative path.
in any case, i think im missing some context here, why are you trying to use the config.yaml file in the output dir? users generally create their own config.yaml and pass that to Hydra. pls see this example https://github.com/facebookresearch/hydra/blob/main/examples/tutorials/basic/your_first_hydra_app/5_defaults/conf/config.yaml

[fanjiaxin23]

Thank you for the reply. @jieru-hu Does this conf directory created by you or the script created it automatically?
Also, If someone wanna load a .yaml file, what is the way to do that on config_path, how specifically they can load that file?

[Jasha10]

Does this conf directory created by you or the script created it automatically?

Usually you will create the conf.yaml file yourself.

@junchoi999 have you seen the examples directory on GitHub? It has working hydra apps that you can use when following the tutorials.
This will be a good place to start:
https://github.com/facebookresearch/hydra/tree/main/examples/tutorials/basic/your_first_hydra_app

[fanjiaxin23]

Does this conf directory created by you or the script created it automatically?

Usually you will create the conf.yaml file yourself.

@junchoi999 have you seen the examples directory on GitHub? It has working hydra apps that you can use when following the tutorials. This will be a good place to start: https://github.com/facebookresearch/hydra/tree/main/examples/tutorials/basic/your_first_hydra_app

Thank you for the reply. Yes, I did working on the examples tutorial. It just I got some error like the very first question I posted, not sure how to fix it. Mostly I used tranditional python my_app.py to run it without adding a flag, and some point did it with some flags based on the documentation.

Also, another thing I noticed, when I run my_app.py using db flags etc, it does create output dir and inside config.yaml file automatically.
So in what case, you are creating conf/config.yaml file? I even created but when I run code each time, its generating all output into the output dir not in conf dir even I specify the config_path="conf", it seems not work properly? @Jasha10

[Jasha10]

So in what case, you are creating conf/config.yaml file? I even created but when I run code each time, its generating all output into the output dir not in conf dir even I specify the config_path="conf", it seems not work properly?

This is the expected behavior. The normal workflow is like this:

1. The user creates conf/config.yaml and my_app.py
2. The user runs python my_app.py +....
3. The app reads the conf/config.yaml file and does some computation.
4. The app creates output files under outputs/....

The files in the outputs/ directory are for logging purposes. They are not used as input for the app.

[fanjiaxin23]

Thanks for the reply. Does that means, the python code reads all config stats or the contents from .yaml file and do the computation.
Another things, if we want to update anything or add some contents inside the .yaml file, how do I change the python script? Can you provide few example of it, I tried to update the contents from it but its always changes in the output directory files not from conf/config.yaml file. Any help on it