refactor variable scaling, pressure level scalings only applied in specific circumstances #52
Conversation
|
Please consider using the knowledge about variables that come from the dataset metadata. See https://github.com/ecmwf/anemoi-transform/blob/7cbf5f3d4baa37453022a5a97e17cc71a5b8ceeb/src/anemoi/transform/variables/__init__.py#L47 |
We have given this some thought, and after wanting to use the information from the dataset in the beginning, I have opted for allowing the definition of our own groups here to use different scaling for self-defined groups. |
| data_indices, | ||
| ).get_variable_scaling() | ||
|
|
||
| # Instantiate the pressure level scaling class with the training configuration |
There was a problem hiding this comment.
I don't know if this is possible but I wonder if here we could instantiate from a list of scalars rather than specific ones? I think this is how it is done for the validation metrics
There was a problem hiding this comment.
Yes, this would be useful, as we want to allow more scalar methods for model levels/tendency/etc.
I will have a look.
|
|
||
| # Instantiate the pressure level scaling class with the training configuration | ||
| pressurelevelscaler = instantiate( | ||
| config.training.pressure_level_scaler, |
There was a problem hiding this comment.
I think this config location is wrong? It should be training.variable_loss_scaling.pressure_level_scalar
There was a problem hiding this comment.
Indeed, in the config file, the pressure_level_scalar should be defined directly in training as before.
…umstances' of https://github.com/ecmwf/anemoi-core into 7-pressure-level-scalings-only-applied-in-specific-circumstances
sahahner
changed the title
|
|
| @abstractmethod | ||
| def get_variable_scaling(self) -> np.ndarray: ... | ||
|
|
||
| def get_variable_group(self, variable_name: str) -> tuple[str, str, int]: |
There was a problem hiding this comment.
I think maybe we should put this function in an utils. I only say this because I want to use it in another application outside of the loss scalings
There was a problem hiding this comment.
Yes, it is very similar to the function from the plotting function. Will make sure to have only one function that can be used in both cases.
|
Hi, I would like to know what you think about making all scalers explicit in the config file. Something similar to the |
Solve the problem explained in issue #7 by refactoring the variable scalings into a general variable scaling and a pressure level scaling.
@mc4117 , @pinnstorm and me came up with a new structure. This PR implements this.
This is first draft. Feedback very welcome!