Question: How to modify the config search path for a specific config group?

[cbr9]

I have a configuration structure like this:

conf/
  dataset/
    ...
  evaluation/
    ...
  model/
    lscd_graded_apd.yaml
    lscd_binary_apd.yaml

But I would like to have this:

conf/
  config.yaml
  dataset/
    ...
  evaluation/
    ...
  model/
    lscd/
      graded/
         apd.yaml
      binary/
         apd.yaml

I know that from the shell, or even from config.yaml, I can set model=lscd/graded/apd.yaml. However, if I don't type anything into the shell (model is marked with ??? in config.yaml, Hydra will complain that no value was provided, it will try to output a list of possible configurations, but the list will be empty.

Is there some way to modify the config search path to allow Hydra to find config "subgroups"?

[Jasha10]

I have not seen the pattern model=lscd/graded/apd.yaml before. I was not able to make it work when I tried it.

The usual pattern is python my_app.py model/lscd/graded@model=apd. This will place the contents of the model/lscd/graded/apd.yaml config into the model package (see the docs on Defaults List Package Keywords).
If you type model/lscd/graded@model=foo, and if foo.yaml does not exist in the model/lscd/graded directory, then a list of available options for the model/scd/graded group should be printed with the error message.

[cbr9]

Thank you for the help. I managed to succesfully structure my config like that.

However, I am wondering if there is a way to omit the package path (everything before the @ symbol). I have plenty of nested structure and it makes the shell commands incredibly verbose. Correct me if I am wrong, but it seems like hydra should be able to find those packages even without that explict string, by using what is written in the config.

[Jasha10]

I am wondering if there is a way to omit the package path (everything before the @ symbol)

So you want to write python my_app.py model=apd instead of python my_app.py model/lscd/graded@model=apd? Hydra would have no way to know whether you're referring to model/lscd/graded/apd.yaml or to model/lscd/binary/apd.yaml.

Hydra uses the filesystem tree to organize yaml files. If a yaml file is deeply nested on your filesystem, then you'll need to use a deeply-nested name to look it up.

[Jasha10]

You can get some traction using Hydra's support for Interpolation in the Defaults List.
Consider this example:

$ tree
.
├── conf
│   ├── config.yaml
│   ├── dataset
│   ├── evaluation
│   └── model
│       ├── binary.yaml
│       ├── graded.yaml
│       └── lscd
│           ├── binary_apd.yaml
│           └── graded_apd.yaml
└── my_app.py

5 directories, 6 files

$ cat my_app.py
import hydra
from omegaconf import OmegaConf


@hydra.main(config_path="conf", config_name="config", version_base=None)
def app(cfg):
    print(OmegaConf.to_yaml(cfg))


app()

$ cat conf/config.yaml
defaults:
  - model: ???
  - model/lscd@model: ${model}_apd

$ cat conf/model/lscd/binary_apd.yaml
model_spec: binary apd

$ cat conf/model/lscd/graded_apd.yaml
model_spec: graded apd

With the yaml files set up like this, the defaults list interpolation model/lscd@model: ${model}_apd means that setting model=binary at the command line will cause model/lscd@model: binary_apd to be added to the defaults list:

$ python my_app.py model=binary
model:
  model_spec: binary apd

$ python my_app.py model=graded
model:
  model_spec: graded apd