Runtime value checking for LaunchSettings and BatchSettings #740

juliaputko · 2024-10-09T22:57:41Z

No description provided.

codecov · 2024-10-09T23:01:34Z

Codecov Report

Attention: Patch coverage is 15.38462% with 11 lines in your changes missing coverage. Please review.

Project coverage is 37.90%. Comparing base (cce16e6) to head (acd56da).
Report is 19 commits behind head on smartsim-refactor.

Files with missing lines	Patch %	Lines
smartsim/settings/batch_settings.py	0.00%	5 Missing ⚠️
smartsim/settings/launch_settings.py	40.00%	3 Missing ⚠️
smartsim/settings/arguments/launch/dragon.py	0.00%	1 Missing ⚠️
smartsim/settings/arguments/launch/local.py	0.00%	1 Missing ⚠️
smartsim/settings/arguments/launch/mpi.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##           smartsim-refactor     #740      +/-   ##
=====================================================
- Coverage              40.45%   37.90%   -2.56%     
=====================================================
  Files                    110      109       -1     
  Lines                   7326     6583     -743     
=====================================================
- Hits                    2964     2495     -469     
+ Misses                  4362     4088     -274

Files with missing lines	Coverage Δ
smartsim/settings/arguments/launch/dragon.py	`54.83% <0.00%> (-25.81%)`	⬇️
smartsim/settings/arguments/launch/local.py	`50.00% <0.00%> (-37.50%)`	⬇️
smartsim/settings/arguments/launch/mpi.py	`43.02% <0.00%> (-52.33%)`	⬇️
smartsim/settings/launch_settings.py	`64.93% <40.00%> (ø)`
smartsim/settings/batch_settings.py	`46.42% <0.00%> (ø)`

... and 28 files with indirect coverage changes

MattToast

Looks great so far! Just some initial feedback for you to take a look at while we expand this out into {Batch,Launch}Arguments classes as well!

smartsim/settings/batch_settings.py

smartsim/settings/launch_settings.py

tests/temp_tests/test_settings/test_batchSettings.py

tests/temp_tests/test_settings/test_launchSettings.py

amandarichardsonn

Awesome work as always! One larger suggestion is to move the setter testing to the associated launcher and scheduler test files!

smartsim/settings/arguments/batch/lsf.py

smartsim/settings/arguments/batch/pbs.py

smartsim/settings/arguments/batch/slurm.py

tests/temp_tests/test_settings/test_batchSettings.py

MattToast

Couple of small nits, but otherwise looks about ready to go!!

smartsim/settings/arguments/batch/pbs.py

smartsim/settings/batch_settings.py

smartsim/settings/launch_settings.py

tests/temp_tests/test_settings/test_batchSettings.py

tests/temp_tests/test_settings/test_launchSettings.py

smartsim/settings/arguments/batch/pbs.py

amandarichardsonn

Some small comments! One larger comment is that the arguments/launch modules haven't been given runtime value checks, consider adding to this PR! Unless this is only the batch PR

smartsim/settings/arguments/batch/lsf.py

smartsim/settings/arguments/batch/pbs.py

smartsim/settings/arguments/batch/slurm.py

amandarichardsonn · 2024-10-31T19:23:15Z

smartsim/settings/launch_settings.py

@@ -126,6 +126,18 @@ def __init__(
            """The launcher type"""


In the modules arguments/launch/... there is a spelling error where Overwriting is written as Overwritting - could you fix this in this PR?

cant find this error

amandarichardsonn · 2024-10-31T19:26:00Z

smartsim/settings/launch_settings.py

@@ -126,6 +126,18 @@ def __init__(
            """The launcher type"""
        except ValueError:
            raise ValueError(f"Invalid launcher type: {launcher}")
+


I notice that there hasnt been Value Checking added to arguments/launch/ modules, but all Value checking has been added to batch modules, is launch_setting value checking also going into this PR?

amandarichardsonn · 2024-10-31T19:28:48Z

tests/temp_tests/test_settings/test_lsfScheduler.py

+def test_batch_arguments_type_set_smts():
+    bs = BatchSettings(batch_scheduler="lsf", env_vars={"ENV": "VAR"})
+    with pytest.raises(TypeError, match="smts argument was not of type int"):
+        bs.batch_args.set_smts("invalid")


consider using pytest.parameterize (I spelt it so wrong) but tests look great!

…rguments

MattToast

Similar question to Amanda about whether or not we want to cover smartsim.settings.arguments.* in this PR as well, and a couple of small nits, but otherwise looks about ready to go on my end!

MattToast · 2024-10-31T23:48:14Z

smartsim/settings/arguments/batch/lsf.py

+        if key in self._batch_args and key != self._batch_args[key]:
+            logger.warning(f"Overwriting argument '{key}' with value '{value}'")


Just sanity checking, should this be

Suggested change

if key in self._batch_args and key != self._batch_args[key]:

logger.warning(f"Overwriting argument '{key}' with value '{value}'")

if key in self._batch_args and value != self._batch_args[key]:

logger.warning(f"Overwriting argument '{key}' with value '{value}'")

?

MattToast · 2024-10-31T23:51:24Z

smartsim/settings/arguments/batch/pbs.py

+        if key in self._batch_args and key != self._batch_args[key]:
+            logger.warning(f"Overwriting argument '{key}' with value '{value}'")


Ditto here

Suggested change

if key in self._batch_args and key != self._batch_args[key]:

logger.warning(f"Overwriting argument '{key}' with value '{value}'")

if key in self._batch_args and value != self._batch_args[key]:

logger.warning(f"Overwriting argument '{key}' with value '{value}'")

?

MattToast · 2024-10-31T23:53:23Z

smartsim/settings/arguments/batch/slurm.py

+        if key in self._batch_args and key != self._batch_args[key]:
+            logger.warning(f"Overwriting argument '{key}' with value '{value}'")


One more here

Suggested change

if key in self._batch_args and key != self._batch_args[key]:

logger.warning(f"Overwriting argument '{key}' with value '{value}'")

if key in self._batch_args and value != self._batch_args[key]:

logger.warning(f"Overwriting argument '{key}' with value '{value}'")

?

MattToast · 2024-10-31T23:54:28Z

smartsim/settings/arguments/launch/alps.py

@@ -224,5 +224,5 @@ def set(self, key: str, value: str | None) -> None:
            )
            return
        if key in self._launch_args and key != self._launch_args[key]:


Not technically part of this PR, but I think this should also read

Suggested change

if key in self._launch_args and key != self._launch_args[key]:

if key in self._launch_args and value != self._launch_args[key]:

?

MattToast · 2024-10-31T23:55:03Z

smartsim/settings/arguments/launch/dragon.py

@@ -71,7 +71,7 @@ def set(self, key: str, value: str | None) -> None:
        """
        set_check_input(key, value)
        if key in self._launch_args and key != self._launch_args[key]:


Also somehow got past a previous review

Suggested change

if key in self._launch_args and key != self._launch_args[key]:

if key in self._launch_args and value != self._launch_args[key]:

?

MattToast · 2024-10-31T23:57:28Z

smartsim/settings/arguments/launch/slurm.py

@@ -349,5 +351,5 @@ def set(self, key: str, value: str | None) -> None:
            )
            return
        if key in self._launch_args and key != self._launch_args[key]:


Whoops! I lied!! Last one for real

Suggested change

if key in self._launch_args and key != self._launch_args[key]:

if key in self._launch_args and value != self._launch_args[key]:

MattToast · 2024-11-01T00:00:10Z

smartsim/settings/batch_settings.py

@@ -140,6 +152,16 @@ def env_vars(self) -> StringArgument:
    @env_vars.setter
    def env_vars(self, value: t.Dict[str, str | None]) -> None:
        """Set the environment variables."""
+
+        if not (
+            isinstance(value, t.Dict)


Nit: for runtime checks (and static checks with python>=3.9), it is generally preferred to use dict over t.Dict

Suggested change

isinstance(value, t.Dict)

isinstance(value, dict)

MattToast · 2024-11-01T00:01:30Z

smartsim/settings/launch_settings.py

+
+        if launch_args is not None:
+            if not (
+                isinstance(launch_args, t.Dict)


Ditto pedantic "dict v t.Dict" nit

Suggested change

isinstance(launch_args, t.Dict)

isinstance(launch_args, dict)

MattToast · 2024-11-01T20:28:48Z

smartsim/settings/batch_settings.py

+            and all(
+                isinstance(key, str) and isinstance(val, (str, type(None)))
+                for key, val in value.items()
+            )
+        ):
+            raise TypeError("env_vars argument was not of type dict of str and str")


Minor nit: runtime check takes a dict[str, str | None], but the error message claims to only take a dict[str, str]

Suggested change

and all(

isinstance(key, str) and isinstance(val, (str, type(None)))

for key, val in value.items()

)

):

raise TypeError("env_vars argument was not of type dict of str and str")

and all(

isinstance(key, str) and isinstance(val, (str, type(None)))

for key, val in value.items()

)

):

raise TypeError("env_vars argument was not of type dict of str and str or None")

MattToast · 2024-11-01T20:30:18Z

smartsim/settings/launch_settings.py

+            and all(
+                isinstance(key, str) and isinstance(val, (str, type(None)))
+                for key, val in value.items()
+            )
+        ):
+            raise TypeError("env_vars argument was not of type dict of str and str")


Ditto dict[str, str | None] vs dict[str, str] discrepancy

Suggested change

and all(

isinstance(key, str) and isinstance(val, (str, type(None)))

for key, val in value.items()

)

):

raise TypeError("env_vars argument was not of type dict of str and str")

and all(

isinstance(key, str) and isinstance(val, (str, type(None)))

for key, val in value.items()

)

):

raise TypeError("env_vars argument was not of type dict of str and str or None")

juliaputko requested review from amandarichardsonn and MattToast October 9, 2024 22:57

MattToast requested changes Oct 10, 2024

View reviewed changes

amandarichardsonn suggested changes Oct 15, 2024

View reviewed changes

MattToast requested changes Oct 19, 2024

View reviewed changes

ankona reviewed Oct 28, 2024

View reviewed changes

smartsim/settings/arguments/batch/pbs.py Outdated Show resolved Hide resolved

amandarichardsonn self-requested a review October 31, 2024 17:33

amandarichardsonn suggested changes Oct 31, 2024

View reviewed changes

juliaputko force-pushed the settingsvaluecheck branch from acd56da to d83d7cd Compare October 31, 2024 23:16

juliaputko added 5 commits October 31, 2024 16:16

tests and value checks for batch and launch settings

3e8395a

addressing review comments; added value checks for batch and launch a…

39e8e1c

…rguments

more value checking, move tests to launcher specific tests

abb06a2

PR review comments addressed

14d4955

spelling update, minor test changes

d83d7cd

juliaputko changed the base branch from smartsim-refactor to v1.0 October 31, 2024 23:17

MattToast requested changes Nov 1, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runtime value checking for LaunchSettings and BatchSettings #740

Runtime value checking for LaunchSettings and BatchSettings #740

juliaputko commented Oct 9, 2024

codecov bot commented Oct 9, 2024 •

edited

Loading

MattToast left a comment

amandarichardsonn left a comment

MattToast left a comment

amandarichardsonn left a comment

amandarichardsonn Oct 31, 2024

juliaputko Oct 31, 2024

amandarichardsonn Oct 31, 2024

amandarichardsonn Oct 31, 2024

MattToast left a comment

MattToast Oct 31, 2024

MattToast Oct 31, 2024

MattToast Oct 31, 2024

MattToast Oct 31, 2024

MattToast Oct 31, 2024

MattToast Oct 31, 2024

MattToast Nov 1, 2024

MattToast Nov 1, 2024

MattToast Nov 1, 2024

MattToast Nov 1, 2024

		if key in self._batch_args and key != self._batch_args[key]:
		logger.warning(f"Overwriting argument '{key}' with value '{value}'")

	if key in self._launch_args and key != self._launch_args[key]:
	if key in self._launch_args and value != self._launch_args[key]:

	isinstance(launch_args, t.Dict)
	isinstance(launch_args, dict)

Runtime value checking for LaunchSettings and BatchSettings #740

Are you sure you want to change the base?

Runtime value checking for LaunchSettings and BatchSettings #740

Conversation

juliaputko commented Oct 9, 2024

codecov bot commented Oct 9, 2024 • edited Loading

Codecov Report

MattToast left a comment

Choose a reason for hiding this comment

amandarichardsonn left a comment

Choose a reason for hiding this comment

MattToast left a comment

Choose a reason for hiding this comment

amandarichardsonn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MattToast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 9, 2024 •

edited

Loading