How can I override an element of ListConfig using structured configs?

[dnnspark]

Hello! I would like to override an element of a ListConfig using structured config. I tried to make a toy example by adapting the example in the doc:

# System imports
from dataclasses import dataclass, field
from typing import Any, List

# Third-party imports
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import MISSING, ListConfig, OmegaConf


@dataclass
class MySQLConfig:
    host: str = "mysql"
    # port: int = 3306  <-- original example
    ports: ListConfig = field(default_factory=lambda: ListConfig([3306, 3307, 3308]))


@dataclass
class YourSQLConfig(MySQLConfig):
    defaults: List[Any] = field(default_factory=lambda: ["mysql"])

    host: str = "your_sql"
    ########################################################################
    # Question: what code should go here to override an element of ports here, say, ports[2]=4000?
    # From command line, I can do `db.ports.2=4000`.
    ########################################################################

defaults = [
    # Load the config "mysql" from the config group "db"
    {"db": "yoursql"}
]


@dataclass
class Config:
    # this is unfortunately verbose due to @dataclass limitations
    defaults: List[Any] = field(default_factory=lambda: defaults)

    # Hydra will populate this field based on the defaults list
    db: Any = MISSING


cs = ConfigStore.instance()
cs.store(group="db", name="mysql", node=MySQLConfig)
cs.store(group="db", name="yoursql", node=YourSQLConfig)
cs.store(name="config", node=Config)


@hydra.main(version_base=None, config_name="config")
def my_app(cfg: Config) -> None:
    print(OmegaConf.to_yaml(cfg))


if __name__ == "__main__":
    my_app()

[Jasha10]

Hi @dnnspark,
Here are a few options:

Option 1: create a new field object:

...
@dataclass
class YourSQLConfig(MySQLConfig):
    defaults: List[Any] = field(default_factory=lambda: ["mysql"])
    host: str = "your_sql"
    ports: ListConfig = field(default_factory=lambda: ListConfig([3306, 3307, 4000]))
...

Option 2: pass an override at the command line:

python app.py db.ports.2=4000

Option 3: Use the oc.dict.values trick (detailed here).

from dataclasses import dataclass, field
from typing import Any, List, Dict
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import MISSING, OmegaConf, SI

@dataclass
class MySQLConfig:
    host: str = "mysql"
    ports: List[int] = SI("${oc.dict.values:._ports}")
    _ports: Dict[str, int] = field(
        default_factory=lambda: OmegaConf.create({"0": 3306, "1": 3307, "2": 3308})
    )

@dataclass
class YourSQLConfig(MySQLConfig):
    defaults: List[Any] = field(default_factory=lambda: ["mysql"])
    host: str = "your_sql"
    _ports: Dict[str, int] = field(
        default_factory=lambda: OmegaConf.create({"2": 4000})
    )

defaults = [
    {"db": "yoursql"}
]

@dataclass
class Config:
    defaults: List[Any] = field(default_factory=lambda: defaults)
    db: Any = MISSING

cs = ConfigStore.instance()
cs.store(group="db", name="mysql", node=MySQLConfig)
cs.store(group="db", name="yoursql", node=YourSQLConfig)
cs.store(name="config", node=Config)

@hydra.main(version_base=None, config_name="config")
def my_app(cfg: Config) -> None:
    OmegaConf.resolve(cfg)
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    my_app()

$ python app.py db.ports.2=4000
db:
  host: your_sql
  ports:
  - 3306
  - 3307
  - 4000
  _ports:
    '0': 3306
    '1': 3307
    '2': 4000

The key idea here is to introduce a _ports dict in MySQLConfig and YourSQLConfig. The values of this _ports dict become the elements of the ports list via a call to oc.dict.values. The values of _ports from these two dataclasses are merged together to create a composed config with 4000 the port at index 2.

I've called OmegaConf.resolve(cfg) in the my_app function to make the output easier to read; without calling resolve, you'd see ports: ${oc.dict.values:._ports} in the output.

[dnnspark]

Thanks a lot @Jasha10! 🙏 While they're a bit more involved than I thought, these are all valid options to me.