Dimensions other than region and variable are not read from repositories listed in nomenclature.yaml

[korsbakken]

When creating a new DataStructureDefinition object from a path that includes external repositories in a nomenclature.yaml file, only the region and variable dimensions get read from the repository. If I list, e.g., model and scenario in the dimensions list in nomenclature.yaml, I get a ValueError with message "Empty codelist: model, scenario". Those dimensions are present with valid yaml files in the repository, and are read without any problems if I clone the repository to my computer and read its definitions folder locally.

I have pasted the nomenclature.yaml file at the bottom. It will reproduce the problem if put in a folder with empty definitions and mappings subdirectories, and read with dsd = nomenclature.DataStructureDefinition('./definitions').

I'm not fluent enough in pydantic to diagnose the problem 100% certainly or know for sure how to fix it, but I think the problem lies in the definition of config.DataStructureConfig, which defines the definitions field in config.NomenclatureConfig. The definition looks suspiciously like it only defines region and variable as valid fields, which I suspect means that the other dimensions under definitions in nomenclature.yaml get discarded. Relevant part of the code here:

nomenclature/nomenclature/config.py

Lines 115 to 139 in 829844a

	class DataStructureConfig(BaseModel):
	"""A class for configuration of a DataStructureDefinition
	Attributes
	----------
	region : RegionCodeListConfig
	Attributes for configuring the RegionCodeList
	"""
	region: Optional[RegionCodeListConfig] = Field(default_factory=RegionCodeListConfig)
	variable: Optional[CodeListConfig] = Field(default_factory=CodeListConfig)
	@field_validator("region", "variable", mode="before")
	@classmethod
	def add_dimension(cls, v, info: ValidationInfo):
	return {"dimension": info.field_name, **v}
	@property
	def repos(self) -> dict[str, str]:
	return {
	dimension: getattr(self, dimension).repositories
	for dimension in ("region", "variable")
	if getattr(self, dimension).repositories
	}

I'd greatly appreciate if someone who is familiar with the pydantic definitions and how they are processed can take a look at this relatively soon. At the moment it's a roadblock for deploying a validation tool in an ongoing project.

Here is the content of the nomenclature.yaml file I used:

repositories:
  iamcompact-nomenclature-definitions:
    url: https://github.com/ciceroOslo/iamcompact-nomenclature-definitions.git
    release: validation-ui
dimensions:
  - variable
  - region
  - model
  - scenario
definitions:
  variable:
    repository: iamcompact-nomenclature-definitions
  region:
    repository: iamcompact-nomenclature-definitions
  model:
    repository: iamcompact-nomenclature-definitions
  scenario:
    repository: iamcompact-nomenclature-definitions
mappings:
  repository: iamcompact-nomenclature-definitions

The error traceI get is the following:

File ~/src/repos/iamcompact/iamcompact-nomenclature/.venv/lib/python3.12/site-packages/nomenclature/definition.py:81, in DataStructureDefinition.__init__(self, path, dimensions)
     76     self.__setattr__(
     77         dim, codelist_cls.from_directory(dim, path / dim, self.config)
     78     )
     80 if empty := [d for d in self.dimensions if not getattr(self, d)]:
---> 81     raise ValueError(f"Empty codelist: {', '.join(empty)}")

ValueError: Empty codelist: model, scenario

If I enter debug mode, I get the following content in self.config.dict():

{'dimensions': ['variable', 'region', 'model', 'scenario'],
 'repositories': {'iamcompact-nomenclature-definitions': {'url': 'https://github.com/ciceroOslo/iamcompact-nomenclature-definitions.git',
   'hash': None,
   'release': 'validation-ui',
   'local_path': PosixPath('iamcompact-nomenclature-definitions')}},
 'definitions': {'region': {'dimension': 'region',
   'repositories': {'iamcompact-nomenclature-definitions'},
   'country': False,
   'nuts': None},
  'variable': {'dimension': 'variable',
   'repositories': {'iamcompact-nomenclature-definitions'}}},
 'mappings': {'repositories': {'iamcompact-nomenclature-definitions'}}}

In other words, only variable: and region: have been read from the definitions section of the config file.

[danielhuppmann]

I think you are right, the current implementation is hard-coded to only import definitions for the region and variable dimension. We haven't had this use case yet.

Not sure how quickly @dc-almeida or @phackstock can take a look to extend the feature.

[korsbakken]

Actually, it was simple enough to just add the fields, copying the definition used for the variable field (which doesn't use a custom definition, unlike region). I submitted it as a PR, #415 . Great if someone could take a look to make sure there's isn't some issue here that I'm not aware of. But it seems to fully resolve the issue on my end.