Enhance type checking #382

ggalloni · 2024-09-04T12:59:00Z

The method validate_info of CobayaComponent, checks only bool.

This PR is enhancing that to check every relevant type, including generic types (List[], Dict[], Tuple[], etc).

ggalloni · 2024-09-04T16:06:12Z

The new code is raising a TypeError when max_samples="bad_value", however, the test_mcmc.py (MPI case) is still breaking as if it is not catching that.

Do you have an idea why this could be happening?

codecov-commenter · 2024-09-04T16:29:48Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 26.31579% with 42 lines in your changes missing coverage. Please review.

Project coverage is 74.25%. Comparing base (735f7a8) to head (19e4143).

Files with missing lines	Patch %	Lines
cobaya/component.py	22.64%	41 Missing ⚠️
cobaya/samplers/polychord/polychord.py	0.00%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #382      +/-   ##
==========================================
- Coverage   74.57%   74.25%   -0.33%     
==========================================
  Files         147      147              
  Lines       11200    11247      +47     
==========================================
- Hits         8352     8351       -1     
- Misses       2848     2896      +48

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ggalloni · 2024-09-06T09:56:46Z

I left the validation of bools as it was and added an enforce_types attribute that will trigger the new code.

In this way, one can force type checking by setting enforce_types=True in any descendent class of CobayaComponent, without touching the old validation.

cmbant

Interesting to see this. Bit mixed feeling though, a bit complicated than hoping (for something is just checking), not very clear how robust it is.

cmbant · 2024-09-27T10:25:58Z

cobaya/model.py

@@ -589,7 +589,7 @@ def logpost(self,
        return self.logposterior(params_values, make_finite=make_finite,
                                 return_derived=False, cached=cached).logpost

-    def get_valid_point(self, max_tries: int, ignore_fixed_ref: bool = False,
+    def get_valid_point(self, max_tries: Union[int, str], ignore_fixed_ref: bool = False,


Has to be int as used in the code?

Hello @cmbant, thanks for the feedback on this!

Indeed I think that, technically, that could be a float, as it is used just for evaluations like if tries<max_tries. Still, I wouldn't feel comfortable with having a floating max_tries, right?

Also, that number comes from the value of a NumberWithUnits, already parsed, so that should be either a float or an int. Thus, I would drop the Union with str (going back as it was before).

cmbant · 2024-09-27T10:26:23Z

cobaya/cosmo_input/convert_cosmomc.py

@@ -37,7 +37,7 @@ def cosmomc_root_to_cobaya_info_dict(root: str, derived_to_input=()) -> InputDic
            name = name.replace('chi2_', 'chi2__')
        if name.startswith('minuslogprior') or name == 'chi2':
            continue
-        param_dict: ParamDict = {'latex': par.label}
+        param_dict: 'ParamDict' = {'latex': par.label}


Actual type should be better than deferred where possible

I removed completely deferring 'ParamDict'. This comes at the cost of moving the definition of ParamsDict in typing a few lines below, after the definition of ParamDict. Let me know what you think of this

cmbant · 2024-09-27T10:27:09Z

cobaya/component.py

@@ -331,6 +331,8 @@ class CobayaComponent(HasLogger, HasDefaults):
    _at_resume_prefer_new: List[str] = ["version"]
    _at_resume_prefer_old: List[str] = []

+    enforce_types: bool = False


Probably should start with _ as not something we want changed by yaml

I am not sure about this. Indeed, adding the underscore we cannot enforce types neither through the yaml nor through the info dict (as I do in the test for instance). So, I guess this becomes a question of what default behavior we want to use. If the default value of enforce_types is False, the user can only enforce types on components he/she defines with that flag to True. But, for example, types of built-in samplers and such cannot be checked (except if we add in their definition the opposite flag). This may be a good choice since I guess that types of built-in stuff are checked anyway, in a way of the other. Vice versa, with enforce_types = True by default, everything will be explicitly checked and there is no need for the user to access that.

Am I missing something?

Would probably have to be False by default to keep people from hitting annoying errors and compatibility with existing external likelihoods. Was thinking of this more as an option for likelihood developers who support accurate type hint in their code.

Ok great, then I'll make it private and default at False 👍

cmbant · 2024-09-27T19:59:51Z

cobaya/component.py

+            elif expected_type is int: # for numpy integers
+                if value == float('inf'): # for infinite values parsed as floats
+                    return isinstance(value, float)
+                return isinstance(value, int) or "numpy.int" in str(type(value))


Can simplify some of this using generic numbers.Number types?

Yes, I was thinking about using some more generic types, I will commit that soon.

cmbant · 2024-09-27T20:03:09Z

cobaya/parameterization.py

@@ -42,7 +42,7 @@ def is_derived_param(info_param: ParamInput) -> bool:
    return expand_info_param(info_param).get("derived", False) is not False


-def expand_info_param(info_param: ParamInput, default_derived=True) -> ParamDict:
+def expand_info_param(info_param: ParamInput, default_derived=True) -> 'ParamDict':


Ops, probably a search and replace gone wrong...

cmbant · 2024-09-27T20:03:21Z

cobaya/parameterization.py

@@ -76,7 +76,7 @@ def expand_info_param(info_param: ParamInput, default_derived=True) -> ParamDict
    return info_param


-def reduce_info_param(info_param: ParamDict) -> ParamInput:
+def reduce_info_param(info_param: 'ParamDict') -> ParamInput:


I removed all of these, see above

cmbant · 2024-09-27T20:06:55Z

cobaya/samplers/mcmc/mcmc.py

@@ -56,11 +56,11 @@ class MCMC(CovmatSampler):

    # instance variables from yaml
    burn_in: NumberWithUnits
-    learn_every: NumberWithUnits
-    output_every: NumberWithUnits
+    learn_every: Union[NumberWithUnits, str]


Not sure why only some of these changed. In terms of usage, optional str is probably not very helpful since should be converted at read in, maybe NumberWithUnits and more flexible check

I changed the only ones causing errors, but, following your suggestion, I moved the problem on the type-checking side as we know the NumberWithUnits can come in as a string.

…nto checking_types

ggalloni · 2024-10-02T10:34:48Z

For assessing the robustness of this, I am not sure how to test it. An idea would be to switch the default of _enforce_types to True, let tests here run and, if successful, set it back to False. So at least we know that everything internal is working as expected. What do you think?

ggalloni added 5 commits September 3, 2024 17:31

Allow all types for checking

930db00

Handle exceptions

71fc109

Add str when allowed

e4c2971

Clean imports

dc48e84

Handle ParamDict

7a568e7

Try skipping test for now

7bf99d8

ggalloni added 5 commits September 5, 2024 15:35

Separate bool validation and optional type enforcing

61a0405

ParamDict -> ForwardRef['ParamDict'] for type check

af5c727

Handle ClassVar

a16b804

Add test for type checking

726f522

Test for compatibility

bbb1f6b

ggalloni added 2 commits September 10, 2024 14:09

Test Optional

8e62704

Merge branch 'master' into checking_types

39b75b4

cmbant reviewed Sep 27, 2024

View reviewed changes

ggalloni added 7 commits October 2, 2024 12:22

Remove deferred types

3fe9b84

Remove ForwardRef handling

b985a40

Handle ints and floats with generic types

6ecc79a

Allow NumberWithUnits to be a str

00bcbb3

Remove useless type

5988a15

Merge branch 'checking_types' of https://github.com/ggalloni/cobaya i…

4cdd11e

…nto checking_types

Change enforce_types to private attribute

436e70d

Clean

19e4143

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance type checking #382

Enhance type checking #382

ggalloni commented Sep 4, 2024

ggalloni commented Sep 4, 2024

codecov-commenter commented Sep 4, 2024 •

edited

Loading

ggalloni commented Sep 6, 2024

cmbant left a comment

cmbant Sep 27, 2024

ggalloni Oct 2, 2024

cmbant Sep 27, 2024

ggalloni Oct 2, 2024 •

edited

Loading

cmbant Sep 27, 2024

ggalloni Oct 2, 2024

cmbant Oct 2, 2024

ggalloni Oct 2, 2024

cmbant Sep 27, 2024

ggalloni Oct 2, 2024

cmbant Sep 27, 2024

ggalloni Oct 2, 2024

cmbant Sep 27, 2024

ggalloni Oct 2, 2024

cmbant Sep 27, 2024

ggalloni Oct 2, 2024

ggalloni commented Oct 2, 2024

Enhance type checking #382

Are you sure you want to change the base?

Enhance type checking #382

Conversation

ggalloni commented Sep 4, 2024

ggalloni commented Sep 4, 2024

codecov-commenter commented Sep 4, 2024 • edited Loading

Codecov Report

ggalloni commented Sep 6, 2024

cmbant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggalloni Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggalloni commented Oct 2, 2024

codecov-commenter commented Sep 4, 2024 •

edited

Loading

ggalloni Oct 2, 2024 •

edited

Loading