Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pylint error fixes #766

Open
wants to merge 63 commits into
base: v1.0
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
85a022e
pushing just to update
amandarichardsonn May 2, 2024
ffaa9b3
updates
amandarichardsonn May 3, 2024
8210991
pushing updates
amandarichardsonn May 7, 2024
6e595fb
pushing mypy errors
amandarichardsonn May 8, 2024
821ffe3
Merge branch 'develop' into smartsim-refactor
amandarichardsonn May 15, 2024
f2ba3a7
Merge branch 'develop' of https://github.com/CrayLabs/SmartSim into s…
MattToast May 20, 2024
919e65a
Merge branch 'develop' of https://github.com/CrayLabs/SmartSim into s…
amandarichardsonn May 21, 2024
0c0ab7b
Merge branch 'smartsim-refactor' of https://github.com/CrayLabs/Smart…
amandarichardsonn May 21, 2024
f3a58d4
Merge remote-tracking branch 'upstream/develop' into smartsim-refactor
MattToast May 23, 2024
3ed3ef7
Merge branch 'develop' of https://github.com/CrayLabs/SmartSim into s…
amandarichardsonn May 23, 2024
e96f893
Merge branch 'smartsim-refactor' of https://github.com/CrayLabs/Smart…
amandarichardsonn May 23, 2024
e31c837
Move SmartSim v0.X.Y test to `tests/_legacy` dir (#575)
MattToast May 24, 2024
2cefbcf
Merge branch 'develop' into smartsim-refactor
amandarichardsonn May 28, 2024
ce35746
Merge branch 'smartsim-refactor' of https://github.com/CrayLabs/Smart…
amandarichardsonn May 28, 2024
d105df6
Fix Inconsistency in run_arg Formatting (#588)
juliaputko May 28, 2024
1cf41b1
Ignore build docs on the smartsim-refactor branch (#604)
juliaputko May 29, 2024
89214d6
Renaming Model to Application (#579)
juliaputko May 30, 2024
f21b6f8
Removal of Factory Methods from Experiment API (#581)
juliaputko Jun 3, 2024
883824b
Renaming Orchestrator and Database to Feature Store (#597)
juliaputko Jun 4, 2024
b64af80
Job and MPMD Job Implementation (#603)
juliaputko Jun 7, 2024
d058213
BaseJobGroup, JobGroup, ColocatedJobGroup skeleton (#599)
amandarichardsonn Jun 11, 2024
edb8364
BatchSettings, LaunchSettings, Command, CommandList and LaunchCommand…
amandarichardsonn Jun 13, 2024
e9d8eca
Merge remote-tracking branch 'upstream/develop' into smartsim-refactor
MattToast Jun 26, 2024
7e3e92c
Add Type Check Back to Refactor CI (#630)
MattToast Jul 8, 2024
c2164ca
Compound Entity and Ensemble (#605)
MattToast Jul 10, 2024
9b511f1
Command Generation (#615)
juliaputko Jul 24, 2024
b965c3d
Link up SmartSim Refactor CI (#646)
MattToast Jul 24, 2024
df6ed98
Merge remote-tracking branch 'upstream/develop' into smartsim-refactor
MattToast Jul 25, 2024
bd21957
Dispatch Jobs to Launchers based on Settings (#620)
MattToast Jul 27, 2024
a2c1251
Fix test overwriting a test configuration file (#648)
MattToast Jul 31, 2024
77eaf4d
Simple Status Checking (#655)
MattToast Aug 9, 2024
e44e099
Rename Status Enums (#663)
MattToast Aug 9, 2024
52cd8ec
Refactor of Generation class and inject path into Launch process (#650)
amandarichardsonn Aug 13, 2024
ddde9c5
Move `smartsim.settings.dispatch` module under `smartsim._core` (#659)
juliaputko Aug 14, 2024
347458f
merge develop into smartsim-refactor
juliaputko Aug 14, 2024
bf348a0
remove where install, build_py and Distribution were used in setup.py
juliaputko Aug 15, 2024
7ddb12f
Merge develop into smartsim-refactor (#668)
juliaputko Aug 15, 2024
0569c05
Move `ShellLauncher` (#658)
juliaputko Aug 21, 2024
a0cc447
Removed Slurm specific methods from LaunchArguments (#662)
juliaputko Aug 22, 2024
f6928e5
Redis removal (#656)
juliaputko Aug 22, 2024
f9a86d9
Unit tests for ShellLauncher & additional shell launch context (#671)
amandarichardsonn Aug 28, 2024
ec7677a
Remove dead attributes in application (#673)
juliaputko Aug 29, 2024
5611a16
Poll Based Waiting for Job Completion (#670)
MattToast Aug 29, 2024
4a4f43a
Refine SmartSimEntity Interface (#688)
amandarichardsonn Aug 29, 2024
c1faadd
Experiment Can Stop Launched Jobs (#677)
MattToast Aug 30, 2024
0175b6b
Ensemble Refactor (#686)
juliaputko Sep 5, 2024
c169878
Remove Problematic Type Union (#694)
MattToast Sep 5, 2024
f49fde3
Camel Case to Snake Case (#711)
amandarichardsonn Sep 18, 2024
3bfdff9
Removals of EntityList, EntitySequence, JobManager, and Controller (#…
juliaputko Sep 19, 2024
f748789
Short task to update BatchSettings (#704)
amandarichardsonn Sep 20, 2024
4faf95c
Ensemble Move short-task (#714)
amandarichardsonn Sep 23, 2024
4d9ab27
Replace TaggedFilesHierarchy with os.walk and implement configure_dir…
amandarichardsonn Sep 27, 2024
dbf7b72
Allow for sequences into exp.start(), and unpack iterables (#712)
juliaputko Oct 4, 2024
2cbd3be
Runtime value checking of public API methods (#707)
juliaputko Oct 7, 2024
a39246e
Application.files refactor (#732)
amandarichardsonn Oct 16, 2024
9fb5e3c
pylint error fixes, and temporary pylint suppressions to be removed a…
juliaputko Oct 25, 2024
94f078d
resolve merge conflicts, fix pylint errors
juliaputko Oct 28, 2024
4701e8c
Fix typos in documentation (#758)
jsta Oct 29, 2024
7e68e63
Merge branch 'pylintssrf' of https://github.com/juliaputko/SmartSim i…
juliaputko Oct 29, 2024
8bb01e1
invalid name fix
juliaputko Oct 29, 2024
cd7e3c5
fixed invalid name
juliaputko Oct 29, 2024
78396cd
the last of the pylint fixes
juliaputko Oct 31, 2024
94f0608
Merge branch 'v1pylintfix' of github.com:juliaputko/SmartSim into pyl…
juliaputko Oct 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Ensemble Refactor (#686)
Refactor Ensemble parameters

[ committed by @juliaputko ]
[ reviewed by @MattToast, @mellis13  ]
  • Loading branch information
juliaputko authored Sep 5, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 0175b6ba0477a20a285156fbb7b6dfd493862fa5
5 changes: 3 additions & 2 deletions smartsim/entity/application.py
Original file line number Diff line number Diff line change
@@ -26,6 +26,7 @@

from __future__ import annotations

import collections
import copy
import textwrap
import typing as t
@@ -262,7 +263,7 @@ def _build_exe_args(exe_args: t.Union[str, t.Sequence[str], None]) -> t.List[str
if not (
isinstance(exe_args, str)
or (
isinstance(exe_args, list)
isinstance(exe_args, collections.abc.Sequence)
and all(isinstance(arg, str) for arg in exe_args)
)
):
@@ -271,7 +272,7 @@ def _build_exe_args(exe_args: t.Union[str, t.Sequence[str], None]) -> t.List[str
if isinstance(exe_args, str):
return exe_args.split()

return exe_args
return list(exe_args)

def print_attached_files(self) -> None:
"""Print a table of the attached files on std out"""
176 changes: 168 additions & 8 deletions smartsim/entity/ensemble.py
Original file line number Diff line number Diff line change
@@ -32,7 +32,7 @@
import os.path
import typing as t

from smartsim.entity import _mock, entity, strategies
from smartsim.entity import entity, strategies
from smartsim.entity.application import Application
from smartsim.entity.files import EntityFiles
from smartsim.entity.strategies import ParamSet
@@ -59,23 +59,183 @@ def __init__(
max_permutations: int = -1,
replicas: int = 1,
) -> None:
"""Initialize an ``Ensemble`` of application instances

:param name: name of the ensemble
:param exe: executable to run
:param exe_args: executable arguments
:param exe_arg_parameters: parameters and values to be used when configuring entities
:param files: files to be copied, symlinked, and/or configured prior to
execution
:param file_parameters: parameters and values to be used when configuring
files
:param permutation_strategy: strategy to control how the param values are applied to the Ensemble
:param max_permutations: max parameter permutations to set for the ensemble
:param replicas: number of identical entities to create within an Ensemble
"""
self.name = name
self.exe = os.fspath(exe)
"""The name of the ensemble"""
self._exe = os.fspath(exe)
"""The executable to run"""
self.exe_args = list(exe_args) if exe_args else []
self.exe_arg_parameters = (
"""The executable arguments"""
self._exe_arg_parameters = (
copy.deepcopy(exe_arg_parameters) if exe_arg_parameters else {}
)
self.files = copy.deepcopy(files) if files else EntityFiles()
self.file_parameters = dict(file_parameters) if file_parameters else {}
self.permutation_strategy = permutation_strategy
self.max_permutations = max_permutations
self.replicas = replicas
"""The parameters and values to be used when configuring entities"""
self._files = copy.deepcopy(files) if files else EntityFiles()
"""The files to be copied, symlinked, and/or configured prior to execution"""
self._file_parameters = (
copy.deepcopy(file_parameters) if file_parameters else {}
)
"""The parameters and values to be used when configuring files"""
self._permutation_strategy = permutation_strategy
"""The strategy to control how the param values are applied to the Ensemble"""
self._max_permutations = max_permutations
"""The maximum number of entities to come out of the permutation strategy"""
self._replicas = replicas
"""How many identical entities to create within an Ensemble"""

@property
def exe(self) -> str:
"""Return executable to run.

:returns: application executable to run
"""
return self._exe

@exe.setter
def exe(self, value: str | os.PathLike[str]) -> None:
"""Set executable to run.

:param value: executable to run
"""
self._exe = os.fspath(value)

@property
def exe_args(self) -> t.List[str]:
"""Return a list of attached executable arguments.

:returns: application executable arguments
"""
return self._exe_args

@exe_args.setter
def exe_args(self, value: t.Sequence[str]) -> None:
"""Set the executable arguments.

:param value: executable arguments
"""
self._exe_args = list(value)

@property
def exe_arg_parameters(self) -> t.Mapping[str, t.Sequence[t.Sequence[str]]]:
"""Return the executable argument parameters

:returns: executable arguments parameters
"""
return self._exe_arg_parameters

@exe_arg_parameters.setter
def exe_arg_parameters(
self, value: t.Mapping[str, t.Sequence[t.Sequence[str]]]
) -> None:
"""Set the executable arguments.

:param value: executable arguments
"""
self._exe_arg_parameters = copy.deepcopy(value)

@property
def files(self) -> EntityFiles:
"""Return files to be copied, symlinked, and/or configured prior to
execution.

:returns: files
"""
return self._files

@files.setter
def files(self, value: EntityFiles) -> None:
"""Set files to be copied, symlinked, and/or configured prior to
execution.

:param value: files
"""
self._files = copy.deepcopy(value)

@property
def file_parameters(self) -> t.Mapping[str, t.Sequence[str]]:
"""Return file parameters.

:returns: application file parameters
"""
return self._file_parameters

@file_parameters.setter
def file_parameters(self, value: t.Mapping[str, t.Sequence[str]]) -> None:
"""Set the file parameters.

:param value: file parameters
"""
self._file_parameters = dict(value)

@property
def permutation_strategy(self) -> str | strategies.PermutationStrategyType:
"""Return the permutation strategy

:return: permutation strategy
"""
return self._permutation_strategy

@permutation_strategy.setter
def permutation_strategy(
self, value: str | strategies.PermutationStrategyType
) -> None:
"""Set the permutation strategy

:param value: permutation strategy
"""
self._permutation_strategy = value

@property
def max_permutations(self) -> int:
"""Return the maximum permutations

:return: max permutations
"""
return self._max_permutations

@max_permutations.setter
def max_permutations(self, value: int) -> None:
"""Set the maximum permutations

:param value: the maxpermutations
"""
self._max_permutations = value

@property
def replicas(self) -> int:
"""Return the number of replicas

:return: number of replicas
"""
return self._replicas

@replicas.setter
def replicas(self, value: int) -> None:
"""Set the number of replicas

:return: the number of replicas
"""
self._replicas = value

def _create_applications(self) -> tuple[Application, ...]:
"""Concretize the ensemble attributes into a collection of
application instances.
"""
permutation_strategy = strategies.resolve(self.permutation_strategy)

combinations = permutation_strategy(
self.file_parameters, self.exe_arg_parameters, self.max_permutations
)
52 changes: 51 additions & 1 deletion tests/test_ensemble.py
Original file line number Diff line number Diff line change
@@ -26,11 +26,13 @@

import itertools
import typing as t
from glob import glob
from os import path as osp

import pytest

from smartsim.entity import _mock
from smartsim.entity.ensemble import Ensemble
from smartsim.entity.files import EntityFiles
from smartsim.entity.strategies import ParamSet
from smartsim.settings.launchSettings import LaunchSettings

@@ -40,6 +42,54 @@
_2x2_EXE_ARG = {"EXE": [["a"], ["b", "c"]], "ARGS": [["d"], ["e", "f"]]}


@pytest.fixture
def get_gen_configure_dir(fileutils):
yield fileutils.get_test_conf_path(osp.join("generator_files", "tag_dir_template"))


def test_exe_property():
e = Ensemble(name="test", exe="path/to/example_simulation_program")
exe = e.exe
assert exe == e.exe


def test_exe_args_property():
e = Ensemble("test", exe="path/to/example_simulation_program", exe_args="sleepy.py")
exe_args = e.exe_args
assert exe_args == e.exe_args


def test_exe_arg_parameters_property():
exe_arg_parameters = {"-N": 2}
e = Ensemble(
"test",
exe="path/to/example_simulation_program",
exe_arg_parameters=exe_arg_parameters,
)
exe_arg_parameters = e.exe_arg_parameters
assert exe_arg_parameters == e.exe_arg_parameters


def test_files_property(get_gen_configure_dir):
tagged_files = sorted(glob(get_gen_configure_dir + "/*"))
files = EntityFiles(tagged=tagged_files)
e = Ensemble("test", exe="path/to/example_simulation_program", files=files)
files = e.files
assert files == e.files


def test_file_parameters_property():
file_parameters = {"h": [5, 6, 7, 8]}
e = Ensemble(
"test",
exe="path/to/example_simulation_program",
file_parameters=file_parameters,
)
file_parameters = e.file_parameters

assert file_parameters == e.file_parameters


def user_created_function(
file_params: t.Mapping[str, t.Sequence[str]],
exe_arg_params: t.Mapping[str, t.Sequence[t.Sequence[str]]],