diff --git a/doc/developers/heron_templates.png b/doc/developers/heron_templates.png new file mode 100644 index 00000000..c1d5f8cd Binary files /dev/null and b/doc/developers/heron_templates.png differ diff --git a/doc/developers/templates.md b/doc/developers/templates.md new file mode 100644 index 00000000..fa7d917e --- /dev/null +++ b/doc/developers/templates.md @@ -0,0 +1,62 @@ +# The HERON Template Driver: A Guide for the HERON Developer + +The HERON template driver is the portion of HERON which constructs RAVEN workflows given the information provided in the HERON input file. +In HERON, a template consists of an XML file which outlines a RAVEN workflow and a Python class which configures that workflow with the provided case information. +The template driver determines which template is most appropriate for each case. +The goal of this guide to explain the design philosophy of the template driver and give guidance on how it should be modified when adding features to HERON. +An illustrative diagram of the major object types and their interactions is given in the figure below. +![HERON template diagram](heron_templates.png) + +The current template system was developed with a few guiding principles in mind: +1. XML nodes in the template XML should never be removed by the template's Python class. If a node is present in the XML, you can count in being able to access it anywhere in the template class. +2. Subclasses of `RavenSnippet` should handle all XML operations within the block of XML described by the snippet class. Subclasses of `RavenTemplate` should handle orchestrating the connections among these snippets. +3. Use a flat workflow whenever possible. While any valid HERON case can be run with the bilevel template, the overhead of starting a new instance of RAVEN for each inner workflow iteration can add significantly slow down analyses. + +Also, if you're editing anything in the template drivers: +- Use type hints THOROUGHLY +- Favor properties over getter/setter methods + +## Templates +There are currently three main "flavors" of templates in the HERON templating system: +- Bilevel templates: workflows with an outer workflow for varying system capacity and economic variables and an inner workflow for evaluating the system dispatch over multiple time histories. The bilevel templates are further broken down by case mode ("opt" or "sweep") and time history source (sythetic or static). +- "Flat" templates: workflows which can be collapsed to either an inner or outer workflow. +- Debug template: a special template for HERON's "debug" mode. + +## Flat Template Limitations +Some cases which mathematically could be flat workflows cannot currently be implemented as such due to implementation issues in RAVEN or HERON. +- "opt" mode workflows with a single time history. The limitation is the RAVEN MultiRun step accepts either an Optimizer or a Sampler, but not both. To sample the time history (static or synthetic) requires the use of a sampler in the MultiRun step. +- Some workflows with uncertain economic parameters could be flat workflows, but the cashflows they describe are quantified in the HERON dispatch manager. There is currently no way to sample an uncertain economic parameter without running the dispatch optimization. + +## Should I make a new template? +The templating system is designed to make creating new templates a somewhat easy task. +However, a balance must be struck between configuring existing templates and creating new templates. +When is it appropriate to make a new template? + +Don't make a new template if... +- Substituting one algorithm for another (e.g. Bayesian optimization vs. gradient descent for optimization) +- Exposing options for an algorithm or entity that is already used by one or more templates + +Make a new template if... +- Adding significant new functionality, like new workflow types or new HERON case modes +- There is little overlap between the desired workflow and existing templates +- Adding a feature affects many parts of the template XML + +The final decision is left to the best judgement of the developer. +However, creating a new template likely represents a signficant development effort and would benefit from consultation with the core HERON development team. + +## So you want to... +An approximate guide on steps to take to implement new features. + +### Expose an existing RAVEN feature to the HERON user +1. Create a new RavenSnippet subclass for the feature if one does not yet exist. Expose subelement and attribute options as class properties. +2. Add unit tests for the snippet class. +3. Determine which templates can make use of the feature. If using the feature would require removing a node from the template XML, the template node should be removed and the feature should be added to the workflow from the python class. + +### Add a new HERON case mode +1. If the case mode will be run as a bilevel workflow, a new template file and class will likely need to be made for the bilevel outer template (currently split out by different modes). +2. If some cases of the mode could be run as a flat template, implement that as appropriate. This could be modifying the existing `FlatMultiConfigTemplate` template or creating a new template. Add this new template to the `TemplateDriver` as appropriate. + +### Make new kind of workflow +1. A new template very likely needs to be made. Create one or more template XML files and their corresponding `RavenTemplate` classes to configure them. +2. Consider which features of the workflow are useful in the other templates. Refactor as necessary. +3. Add these templates to the `TemplateDriver`. diff --git a/doc/guide/heron_guide.md b/doc/guide/heron_guide.md index 3913bcfb..230bcd4a 100644 --- a/doc/guide/heron_guide.md +++ b/doc/guide/heron_guide.md @@ -156,9 +156,9 @@ Note that in a typical HERON analysis, on the order of two million dispatch opti ### Custom User Specified Functions -HERON allows users to create their own functions that perform computations during simulation runtime. +HERON allows users to create their own functions that perform computations during simulation runtime. -Currently, these functions can only deal with computations that do not occur during the dispatch optimization. For example, a user can write a function that determines the `` parameter of a component's cashflow because cashflows are not computed during the inner dispatch optimization. +Currently, these functions can only deal with computations that do not occur during the dispatch optimization. For example, a user can write a function that determines the `` parameter of a component's cashflow because cashflows are not computed during the inner dispatch optimization. Currently, a user would _not_ be able to write a custom transfer function that informs the dispatcher on how resources are transformed while moving between components of the specified system. This is because transfer functions are required during the dispatch of the system and would require the user to write the function in a way that could be interpreted by our underlying optimization library. To be more specific, a user would **not** be able to use a custom function within a `` XML node in the HERON input file. **While this feature is not currently available, it may be made available in the future.** @@ -171,9 +171,9 @@ Users can write custom functions, but they must follow the API conventions to en A custom function utilized in a HERON input file requires two input parameters that are always returned by the function: * `data`: A Python dictionary containing information related to associated component that is calling the function. -* `meta`: A Python dictionary containing information pertaining to the case as a whole. +* `meta`: A Python dictionary containing information pertaining to the case as a whole. -It is possible to specify ancillary functions in the python file that do not follow the API conventions, but understand that functions called from the HERON input file will require this specification. +It is possible to specify ancillary functions in the python file that do not follow the API conventions, but understand that functions called from the HERON input file will require this specification. For example, suppose a user wanted to write a function that computed the reference price for a particular component based the current year of the project. In the input file, under the `` node, the user would write: @@ -200,13 +200,13 @@ def get_price(data, meta): year = meta['HERON']['active_index']['year'] if year <=10: multiplier = 3 - else: + else: multiplier = 1.5 result = 1000 * multiplier return {"reference_price": result}, meta ``` -In the above code block, the function starts by accessing data from the `meta` parameter to determine what the current year is within the simulation. Then the function determines the multiplier based on the current year of the simulation. If the simulation is within the first ten years of the project timeline, then it sets a higher multiplier, otherwise it sets the multiplier lower. Finally, the function stores the newly computed `reference_price` into a dictionary that is returned by the function. This value will then be used as the `` within the component that this function is called from within the input file. +In the above code block, the function starts by accessing data from the `meta` parameter to determine what the current year is within the simulation. Then the function determines the multiplier based on the current year of the simulation. If the simulation is within the first ten years of the project timeline, then it sets a higher multiplier, otherwise it sets the multiplier lower. Finally, the function stores the newly computed `reference_price` into a dictionary that is returned by the function. This value will then be used as the `` within the component that this function is called from within the input file. diff --git a/src/Cases.py b/src/Cases.py index ce142d07..9e505c5d 100644 --- a/src/Cases.py +++ b/src/Cases.py @@ -46,22 +46,22 @@ class Case(Base): # > 'optimization_default' - 'min' or 'max' for optimization # > 'percent' (only for percentile) - list of percentiles to return # > 'threshold' (only for sortinoRatio, gainLossRatio, expectedShortfall, valueAtRisk) - threshold value for calculation - stats_metrics_meta = {'expectedValue': {'prefix': 'mean', 'optimization_default': 'max'}, - 'minimum': {'prefix': 'min', 'optimization_default': 'max'}, - 'maximum': {'prefix': 'max', 'optimization_default': 'max'}, - 'median': {'prefix': 'med', 'optimization_default': 'max'}, - 'variance': {'prefix': 'var', 'optimization_default': 'min'}, - 'sigma': {'prefix': 'std', 'optimization_default': 'min'}, - 'percentile': {'prefix': 'perc', 'optimization_default': 'max', 'percent': ['5', '95']}, - 'variationCoefficient': {'prefix': 'varCoeff', 'optimization_default': 'min'}, - 'skewness': {'prefix': 'skew', 'optimization_default': 'min'}, - 'kurtosis': {'prefix': 'kurt', 'optimization_default': 'min'}, - 'samples': {'prefix': 'samp'}, - 'sharpeRatio': {'prefix': 'sharpe', 'optimization_default': 'max'}, - 'sortinoRatio': {'prefix': 'sortino', 'optimization_default': 'max', 'threshold': 'median'}, - 'gainLossRatio': {'prefix': 'glr', 'optimization_default': 'max', 'threshold': 'median'}, - 'expectedShortfall': {'prefix': 'es', 'optimization_default': 'min', 'threshold': ['0.05']}, - 'valueAtRisk': {'prefix': 'VaR', 'optimization_default': 'min', 'threshold': ['0.05']}} + stats_metrics_meta = {'expectedValue': {'prefix': 'mean', 'optimization_default': 'max'}, + 'minimum': {'prefix': 'min', 'optimization_default': 'max'}, + 'maximum': {'prefix': 'max', 'optimization_default': 'max'}, + 'median': {'prefix': 'med', 'optimization_default': 'max'}, + 'variance': {'prefix': 'var', 'optimization_default': 'min'}, + 'sigma': {'prefix': 'std', 'optimization_default': 'min'}, + 'percentile': {'prefix': 'perc', 'optimization_default': 'max', 'percent': ['5', '95']}, + 'variationCoefficient': {'prefix': 'varCoeff', 'optimization_default': 'min'}, + 'skewness': {'prefix': 'skew', 'optimization_default': 'min'}, + 'kurtosis': {'prefix': 'kurt', 'optimization_default': 'min'}, + 'samples': {'prefix': 'samp'}, + 'sharpeRatio': {'prefix': 'sharpe', 'optimization_default': 'max'}, + 'sortinoRatio': {'prefix': 'sortino', 'optimization_default': 'max', 'threshold': 'median'}, + 'gainLossRatio': {'prefix': 'glr', 'optimization_default': 'max', 'threshold': 'median'}, + 'expectedShortfall': {'prefix': 'es', 'optimization_default': 'min', 'threshold': ['0.05']}, + 'valueAtRisk': {'prefix': 'VaR', 'optimization_default': 'min', 'threshold': ['0.05']}} # creating a similar dictionary, this time with the optimization defaults flipped # (Levelized Cost does the opposite optimization for all of these stats) @@ -77,21 +77,21 @@ class Case(Base): # economic metrics that can be returned by sweep results OR alongside optimization results # TODO: might be important to index the stats_metrics_meta... does VaR of IRR make sense? # NOTE: the keys for this meta dictionary are the XML Input names - economic_metrics_meta = {'NPV': {'output_name': 'NPV', - 'TEAL_in_name': 'NPV', + economic_metrics_meta = {'NPV': {'output_name': 'NPV', + 'TEAL_in_name': 'NPV', 'TEAL_out_name': 'NPV', 'stats_map': stats_metrics_meta}, - 'PI': {'output_name': 'PI', - 'TEAL_in_name': 'PI', + 'PI': {'output_name': 'PI', + 'TEAL_in_name': 'PI', 'TEAL_out_name': 'PI', 'stats_map': stats_metrics_meta}, - 'IRR': {'output_name': 'IRR', - 'TEAL_in_name': 'IRR', + 'IRR': {'output_name': 'IRR', + 'TEAL_in_name': 'IRR', 'TEAL_out_name': 'IRR', 'stats_map': stats_metrics_meta}, - 'LC': {'output_name': 'LC_Mult', #this is how it will appear in CSV - 'TEAL_in_name': 'NPV_search', #this is how TEAL recognizes it - 'TEAL_out_name': 'NPV_mult', #this is how TEAL outputs it (don't know why) + 'LC': {'output_name': 'LC_Mult', #this is how it will appear in CSV + 'TEAL_in_name': 'NPV_search', #this is how TEAL recognizes it + 'TEAL_out_name': 'NPV_mult', #this is how TEAL outputs it (don't know why) 'stats_map': flipped_stats_metrics_meta}} # the keys of the meta dictionary are the names used in XML input economic_metrics_input_names = list(em_name for em_name,_ in economic_metrics_meta.items()) @@ -959,8 +959,15 @@ def _read_optimization_settings(self, node): # add other information to opt_settings dictionary (type is only information implemented) opt_settings[sub_name] = sub.value - if 'stats_metric' not in list(opt_settings): - opt_settings['stats_metric'] = {'name':self._default_stats_metric, 'tol':1e-4} + if 'stats_metric' not in opt_settings: + opt_settings['stats_metric'] = {'name': self._default_stats_metric, 'tol': 1e-4} + + # Set optimization type ("min" or "max") based on default by economic metric if not provided + if 'type' not in opt_settings: + opt_metric = opt_settings['opt_metric'] + stats_metric = opt_settings['stats_metric']['name'] + opt_settings['type'] = self.economic_metrics_meta[opt_metric]['stats_map'][stats_metric]['optimization_default'] + return opt_settings def _read_result_statistics(self, node): @@ -1068,14 +1075,9 @@ def _append_econ_metrics(self, new_metric, first=False): self._econ_metrics[new_metric] = self.economic_metrics_meta[new_metric] else: # we are updating the stored economic metric dictionary with new entries via an ordered dict + self._econ_metrics[new_metric] = self.economic_metrics_meta[new_metric] if first: - # there has to be a better way, but OrderedDict has no "prepend" method - new_dict = OrderedDict() - new_dict[new_metric] = self.economic_metrics_meta[new_metric] - new_dict.update(self._econ_metrics) - self._econ_metrics = new_dict - else: - self._econ_metrics[new_metric] = self.economic_metrics_meta[new_metric] + self._econ_metrics.move_to_end(new_metric, last=False) # last=False means move to beginning def determine_inner_objective(self, components): """ @@ -1354,35 +1356,31 @@ def npv_target(self): return self._npv_target #### API #### - def write_workflows(self, components, sources, loc): + def write_workflows(self, components, sources, dest_dir): """ Writes workflows for this case to XMLs on disk. @ In, components, HERON components, components for the simulation @ In, sources, HERON sources, sources for the simulation - @ In, loc, str, location in which to write files + @ In, dest_dir, str, directory in which to write files @ Out, None """ - # load templates - template_class = self._load_template() - inner, outer = template_class.createWorkflow(self, components, sources) - - template_class.writeWorkflow((inner, outer), loc) + # Load templates, create RAVEN workflows, and write those workflows using a TemplateDriver + driver = self._make_template_driver() + driver.create_workflow(self, components, sources) + driver.write_workflow(dest_dir, self, components, sources) #### UTILITIES #### - def _load_template(self): + def _make_template_driver(self): """ Loads template files for modification @ In, None - @ Out, template_class, RAVEN Template, instantiated Template class + @ Out, template_class, TemplateDriver, instantiated TemplateDriver class """ src_dir = os.path.dirname(os.path.realpath(__file__)) - heron_dir = os.path.abspath(os.path.join(src_dir, '..')) - template_dir = os.path.abspath(os.path.join(heron_dir, 'templates')) - template_name = 'template_driver' + heron_dir = os.path.abspath(os.path.join(src_dir, "..")) # import template module sys.path.append(heron_dir) - module = importlib.import_module(f'templates.{template_name}', package="HERON") + module = importlib.import_module("templates.template_driver", package="HERON") # load template, perform actions - template_class = module.Template(messageHandler=self.messageHandler) - template_class.loadTemplate(template_dir) - return template_class + driver = module.TemplateDriver(messageHandler=self.messageHandler) + return driver diff --git a/src/Components.py b/src/Components.py index 2daf45ec..f56b63af 100644 --- a/src/Components.py +++ b/src/Components.py @@ -200,6 +200,19 @@ def read_input(self, xml, mode="opt"): self.raiseAnError(IOError, f' node missing from component "{self.name}"!') CashFlowUser.read_input(self, econ_node) + def get_uncertain_cashflow_params(self): + """ + Get all uncertain economic parameters + @ In, None + @ Out, params, dict, the uncertain parameters + """ + params = {} + for cf in self.get_cashflows(): + uncertain = cf.get_uncertain_params() + params |= {f"{self.name}_{k}": v for k, v in uncertain.items()} + return params + + class HeronInteraction(DoveInteraction): """ Base class for component interactions (e.g. Producer, Storage, Demand) diff --git a/src/DispatchManager.py b/src/DispatchManager.py index 43fcf60d..a7858ff8 100644 --- a/src/DispatchManager.py +++ b/src/DispatchManager.py @@ -900,5 +900,3 @@ def run(self, raven, raven_dict): runner.override_time(override_time) # TODO setter dispatch, metrics, tot_activity = runner.run(raven_vars) runner.save_variables(raven, dispatch, metrics, tot_activity) - - diff --git a/src/Economics.py b/src/Economics.py index d33cf12b..c343548a 100644 --- a/src/Economics.py +++ b/src/Economics.py @@ -467,13 +467,11 @@ def set_reference_price(self, node): price_is_levelized = bool(levelized_cost) return price_is_levelized - - # Not none set it to default 1 def get_period(self): """ Getter for Recurring cashflow period type. @ In, None - @ Out, period, str, 'hourly' or 'yearly' + @ Out, period, str | None, 'hour' or 'year' or None """ return self._period @@ -574,6 +572,19 @@ def calculate_params(self, values_dict): params = {'alpha': a, 'driver': D, 'ref_driver': Dp, 'scaling': x, 'cost': cost} # TODO float(cost) except in pyomo it's not a float return params + def get_uncertain_params(self): + """ + Gets any of the cashflow equation parameters which are random variables + @ In, None + @ Out, uncertain_params, dict[ValuedParam], the uncertain cashflow parameters + """ + params = ["_driver", "_alpha", "_reference", "_scale"] + uncertain_params = {} + for param_name in params: + if (param := getattr(self, param_name)).type == "RandomVariable": + uncertain_params[param_name[1:]] = param + return uncertain_params + ####### # API # ####### @@ -656,3 +667,11 @@ def is_npv_exempt(self): @ Out, npv_exempt, bool, is cashflow exempt from NPV calculations? """ return self._npv_exempt + + def is_uncertain(self): + """ + Does the cashflow have any uncertain parameters? + @ In, None + @ Out, uncertain, bool, is cashflow a random variable? + """ + return len(self.get_uncertain_params()) > 0 diff --git a/src/Placeholders.py b/src/Placeholders.py index 8f8676fb..071f4bc7 100644 --- a/src/Placeholders.py +++ b/src/Placeholders.py @@ -8,6 +8,7 @@ import sys import abc import copy +import functools import HERON.src._utils as hutils from HERON.src.base import Base @@ -142,10 +143,11 @@ def is_type(self, typ): @ In, typ, str, type to check against @ Out, is_type, bool, True if matching request """ - # maybe it's not anything we know about - if typ not in ['ARMA', 'Function', 'ROM', 'CSV']: - return False - return eval(f'isinstance(self, {typ})') + return self.type == typ + + @property + def type(self) -> str: + return self._type def get_variable(self): """ @@ -504,7 +506,7 @@ def checkValid(self, case, components, sources): @ Out, None """ self.raiseAMessage(f'Checking CSV at "{self._target_file}"') - structure = hutils.get_csv_structure(self._target_file, case.get_year_name(), case.get_time_name()) + structure = self.get_structure(case) interpolated = 'macro' in structure clustered = bool(structure['clusters']) # segmented = bool(structure['segments']) # TODO @@ -532,3 +534,14 @@ def checkValid(self, case, components, sources): f'"{self.name}" will be extended to project life ({project_life}) macro steps using .' ) self.needs_multiyear = project_life + + @functools.cache + def get_structure(self, case) -> dict: + """ + Reads the CSV file to determine the structure of the time series samples it contains. + @ In, None + @ Out, structure, dict, the number of samples + """ + # NOTE: The result is cached to avoid needing to read the CSV file repeatedly, and the caching is done here in the + # Placeholder instead of the utility function so the caching is done on a per-file basis. + return hutils.get_csv_structure(self._target_file, case.get_year_name(), case.get_time_name()) diff --git a/src/Testers/HeronIntegrationTester.py b/src/Testers/HeronIntegrationTester.py index b5f6eb3f..aeb1efc7 100644 --- a/src/Testers/HeronIntegrationTester.py +++ b/src/Testers/HeronIntegrationTester.py @@ -95,7 +95,13 @@ def get_heron_command(self, cmd): cmd += ' bash.exe ' python = self._get_python_command() # python-command is for running HERON; python_command_for_raven is for running RAVEN inner - cmd += f' {self.heron_driver} --python-command="{python}" --python_command_for_raven="{python}" {heron_inp}' + # NOTE: Adding the --python_command_for_raven argument causes a node to be added in a RAVEN block + # in outer.xml. We only want this to appear if "python" contains something beyond a typical python executable + # (for running coverage scripts, in particular). + if len(python.split()) == 1 and os.path.basename(python) == "python": # should be just a python command + cmd += f' {self.heron_driver} --python-command="{python}" {heron_inp}' + else: # play it safe and use the arguments + cmd += f' {self.heron_driver} --python-command="{python}" --python_command_for_raven="{python}" {heron_inp}' return cmd, heron_inp def get_raven_command(self, cmd, heron_inp): diff --git a/src/_utils.py b/src/_utils.py index 0d33f573..7591c028 100644 --- a/src/_utils.py +++ b/src/_utils.py @@ -9,17 +9,7 @@ import xml.etree.ElementTree as ET import warnings import pickle -try: - from functools import cache -except ImportError: - from functools import lru_cache - def cache(user_function): - """ - use lru_cache for older versions of python - @ In, user_function, function - @ Out, user_function, function that caches values - """ - return lru_cache(maxsize=None)(user_function) +from functools import cache from os import path @@ -184,6 +174,9 @@ def get_csv_structure(fpath, macro_var, micro_var): # to find the environment. data = pd.read_csv(fpath) structure = {} + + structure['num_samples'] = data['RAVEN_sample_ID'].unique().size + if macro_var in data.columns: macro_steps = pd.unique(data[macro_var].values) structure['macro'] = { diff --git a/src/main.py b/src/main.py index 11e80502..bda34e18 100755 --- a/src/main.py +++ b/src/main.py @@ -187,6 +187,7 @@ def main(): if sim._case._workflow == 'standard': if args.python_cmd_raven is not None: + print("creating workflow") sim.create_raven_workflow(python_cmd_raven=args.python_cmd_raven) else: sim.create_raven_workflow() diff --git a/templates/__init__.py b/templates/__init__.py index 5e3ed630..4778d509 100644 --- a/templates/__init__.py +++ b/templates/__init__.py @@ -1,3 +1,2 @@ - # Copyright 2020, Battelle Energy Alliance, LLC # ALL RIGHTS RESERVED diff --git a/templates/bilevel_templates.py b/templates/bilevel_templates.py new file mode 100644 index 00000000..221ff52b --- /dev/null +++ b/templates/bilevel_templates.py @@ -0,0 +1,648 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Templates for bilevel RAVEN workflows (i.e. RAVEN-runs-RAVEN workflows) + + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-23 +""" +from pathlib import Path +import xml.etree.ElementTree as ET +import shutil + +from .imports import RAVEN_LOC +from .heron_types import HeronCase, Component, Source +from .naming_utils import get_capacity_vars, get_component_activity_vars, get_opt_objective + +from .raven_template import RavenTemplate +from .snippets.runinfo import RunInfo +from .snippets.variablegroups import VariableGroup +from .snippets.databases import NetCDF +from .snippets.dataobjects import PointSet +from .snippets.distributions import Distribution +from .snippets.models import RavenCode +from .snippets.outstreams import PrintOutStream, OptPathPlot +from .snippets.samplers import SampledVariable, Sampler, Grid, MonteCarlo, CustomSampler +from .snippets.steps import MultiRun + + +class BilevelTemplate(RavenTemplate): + """ Coordinates information between inner and outer templates for bilevel workflows """ + + def __init__(self, mode: str, has_static_history: bool, has_synthetic_history: bool) -> None: + """ + Constructor + @ In, case, HeronCase, the HERON case object + @ In, source, list[Source], sources + """ + super().__init__() + if has_static_history and has_synthetic_history: + raise ValueError("Bilevel HERON workflows expect either a static history source () or a synthetic history " + "source () but not both! Check your input file.") + + self.inner = InnerTemplateStaticHistory() if has_static_history else InnerTemplateSyntheticHistory() + + if mode == "sweep": + self.outer = OuterTemplateSweep() + elif mode == "opt": + self.outer = OuterTemplateOpt() + else: + raise ValueError(f"Unsupported case mode '{mode}' in Bilevel workflow template.") + + @property + def template_name(self) -> dict[str, str]: + """ + Getter property for the template template_name attributes. Allows for compatibility with acces strategy for + template classes which use only one template file. + @ In, None + @ Out, dict[str, str], inner and outer template names + """ + return {"inner": self.inner.template_name, "outer": self.outer.template_name} + + @property + def write_name(self) -> dict[str, str]: + """ + Getter property for the template write_name attributes. Allows for compatibility with acces strategy for template + classes which use only one template file. + @ In, None + @ Out, dict[str, str], inner and outer template write names + """ + return {"inner": self.inner.write_name, "outer": self.outer.write_name} + + def loadTemplate(self, filename: dict[str, str], path: str) -> None: + """ + Loads template file statefully. + @ In, filename, dict[str, str], dictionary of names of files to load (xml) + @ In, path, str, path to file relative to HERON/templates/ + @ Out, None + """ + self.inner.loadTemplate(filename["inner"], path) + self.outer.loadTemplate(filename["outer"], path) + + def createWorkflow(self, **kwargs) -> dict[str, ET.Element]: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, dict[str, ET.Element], modified template XML + """ + self.inner.createWorkflow(**kwargs) + + # set the path to the inner sampler so the outer knows where to send aliased variables + self.outer.inner_sampler = self.inner.get_sampler_path() + self.outer.createWorkflow(**kwargs) + + # Coordinate across templates. The outer workflow needs to know where the inner workflow will save the dispatch + # results to perform additional processing. + case = kwargs["case"] + disp_results_name = self.inner.get_dispatch_results_name() + self.outer.set_inner_data_name(disp_results_name, case.data_handling["inner_to_outer"]) + + return {"inner": self.inner.template_xml, "outer": self.outer.template_xml} + + def writeWorkflow(self, template: dict[str, ET.Element], destination: dict[str, str], run: bool = False) -> None: + """ + Writes a template to file. + @ In, template, dict[str, ET.Element], XML trees to write + @ In, destination, dict[str, str], paths to write the templates to + @ In, run, bool, optional, if True then run the workflow after writing? good idea? + @ Out, errors, int, 0 if successfully wrote [and run] and nonzero if there was a problem + """ + for name, xml in template.items(): + super().writeWorkflow(xml, destination[name], run) + + # copy "write_inner.py", which has the denoising and capacity fixing algorithms + conv_filename = "write_inner.py" + write_inner_dir = Path(__file__).parent + dest_dir = Path(next(iter(destination.values()))).parent + conv_src = write_inner_dir / conv_filename + conv_file = dest_dir / conv_filename + shutil.copyfile(str(conv_src), str(conv_file)) + print(f"Wrote '{conv_filename}' to '{destination}'") + + @property + def template_xml(self) -> dict[str, ET.Element]: + """ + Getter property for the template XML ET.Element tree + @ In, None + @ Out, _templates, dict[str, ET.Element], the XML trees + """ + return {"inner": self.inner.template_xml, "outer": self.outer.template_xml} + + def get_write_path(self, dest_dir: str) -> dict[str, str]: + """ + Get the path of to write the template to + @ In, dest_dir, str, the directory to write the file to + @ Out, paths, dict[str, str], the path (directory + file name) to write to for each template + """ + return {"inner": self.inner.get_write_path(dest_dir), "outer": self.outer.get_write_path(dest_dir)} + +class OuterTemplate(RavenTemplate): + """ Base class for modifying the outer workflow in bilevel workflows """ + write_name = "outer.xml" + + def __init__(self) -> None: + """ + Constructor + @ In, None + @ Out, None + """ + super().__init__() + self.inner_sampler = None + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + + case = kwargs["case"] + case_name = self.namingTemplates["jobname"].format(case=case.name, io="o") + self._set_case_name(case_name) + self._initialize_runinfo(case) + + components = kwargs["components"] + + # Configure the RAVEN model + self._configure_raven_model(case, components) + # Populate the capacities and outer_results variable groups + self._configure_variable_groups(case, components) + + def set_inner_data_name(self, name: str, inner_to_outer: str) -> None: + """ + Give the RAVEN code model the place to look for the inner's data + @ In, name, str, the name of the data objectw + @ In, inner_to_outer, str, the type of file used to pass the data ("csv" or "netcdf") + @ Out, None + """ + model = self._template.find("Models/Code[@subType='RAVEN']") # type: RavenCode + model.set_inner_data_handling(name, inner_to_outer) + + def _initialize_runinfo(self, case: HeronCase) -> None: + """ + Initializes the RunInfo node of the workflow + @ In, case, Case, the HERON Case object + @ Out, None + """ + run_info = self._template.find("RunInfo") # type: RunInfo + + # parallel + if case.outerParallel > 0: + # set outer batchsize and InternalParallel + run_info.batch_size = case.outerParallel + run_info.use_internal_parallel = True + else: + run_info.batch_size = 1 + + if case.useParallel: + # Fills in parallel settings for template RunInfo from case. Also appliespre-sets for known + # hostnames (e.g. sawtooth, bitterroot), as specified in the HERON/templates/parallel/*.xml files. + run_info.set_parallel_run_settings(case.parallelRunInfo) + + if case.innerParallel: + run_info.num_mpi = case.innerParallel + + def _configure_raven_model(self, case: HeronCase, components: list[Component]) -> RavenCode: + """ + Configures the inner RAVEN code. The bilevel outer template MUST have a node defined. + @ In, case, HeronCase, the HERON case + @ In, components, list[Component], the case components + @ Out, raven, RavenCode, the RAVEN code node + """ + raven = self._template.find("Models/Code[@subType='RAVEN']") # type: RavenCode + + # Find the RAVEN executable to use + exec_path = RAVEN_LOC / "raven_framework" + if exec_path.resolve().exists(): + executable = str(exec_path.resolve()) + elif shutil.which("raven_framework") is not None: + executable = "raven_framework" + else: + raise RuntimeError(f"raven_framework not in PATH and not at {exec_path}") + raven.executable = executable + + # custom python command for running raven (for example, "coverage run") + if cmd := case.get_py_cmd_for_raven(): + raven.python_command = cmd + + # Add alias for the number of denoises + raven.add_alias("denoises", loc=self.inner_sampler) + + # Add variable aliases for Inner + for component in components: + raven.add_alias(component.name, suffix="capacity", loc=self.inner_sampler) + + # Add label aliases for Inner + for label in case.get_labels(): + raven.add_alias(label, suffix="label", loc=self.inner_sampler) + + return raven + + def _configure_variable_groups(self, case: HeronCase, components: list[Component]) -> None: + """ + Fills out variable groups with capacity and results names + @ In, case, HeronCase, the HERON case + @ In, components, list[Component], the case components + @ Out, None + """ + # Set up some helpful variable groups + capacities_vargroup = self._template.find("VariableGroups/Group[@name='GRO_capacities']") + capacities_vars = list(get_capacity_vars(components, self.namingTemplates["variable"])) + capacities_vargroup.variables.extend(capacities_vars) + + results_vargroup = self._template.find("VariableGroups/Group[@name='GRO_outer_results']") + results_vars = self._get_statistical_results_vars(case, components) + results_vargroup.variables.extend(results_vars) + + def _set_batch_size(self, batch_size: int, case: HeronCase) -> None: + """ + Sets the batch size in RunInfo and sets internalParallel to True + @ In, batch_size, int, the batch size + @ In, case, HeronCase, the HERON case + @ Out, None + """ + case.outerParallel = batch_size + run_info = self._template.find("RunInfo") + run_info.batch_size = batch_size + run_info.internal_parallel = True + + +class OuterTemplateOpt(OuterTemplate): + """ Sets up the outer workflow for optimization mode """ + template_name = "outer_opt.xml" + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + case = kwargs["case"] + components = kwargs["components"] + sources = kwargs["sources"] + + # Define XML blocks for optimization: optimizer, sampler, ROM, etc. + opt_strategy = case.get_opt_strategy() + if opt_strategy == "BayesianOpt": + optimizer = self._create_bayesian_opt(case, components) + elif opt_strategy == "GradientDescent": + optimizer = self._create_gradient_descent(case, components) + else: + raise ValueError(f"Template does not recognize optimization strategy {opt_strategy}.") + + # Set optimizer data object + results_data = self._template.find("DataObjects/PointSet[@name='opt_eval']") + optimizer.target_evaluation = results_data + + # Set optimizer objective function + objective = get_opt_objective(case) + optimizer.objective = objective + results = self._template.find("VariableGroups/Group[@name='GRO_outer_results']") # type: VariableGroup + if objective not in results.variables: + results.variables.insert(0, objective) + + # Add case labels to the optimizer + self._add_labels_to_sampler(optimizer, case.get_labels()) + + # Add the optimizer and any custom function files to the main MultiRun step + multirun = self._template.find("Steps/MultiRun[@name='optimize']") # type: MultiRun + for func in self._get_function_files(sources): + multirun.add_input(func) + multirun.add_optimizer(optimizer) + + # Add the optimization objective to the opt_path plot variables + opt_path_plot = self._template.find("OutStreams/Plot[@subType='OptPath']") # type: OptPathPlot + opt_path_plot.variables.append(objective) + + # Update the parallel settings based on the number of sampled variables if the number of outer parallel runs + # was not specified before. + if case.outerParallel == 0 and case.useParallel: + batch_size = optimizer.num_sampled_vars + 1 + self._set_batch_size(batch_size, case) + + +class OuterTemplateSweep(OuterTemplate): + """ Sets up the outer workflow for sweep mode """ + template_name = "outer_sweep.xml" + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + case = kwargs["case"] + components = kwargs["components"] + + # Populate the sampled and constant capacities in the Grid sampler + sampler = self._template.find("Samplers/Grid") # type: Grid + variables, consts = self._create_sampler_variables(case, components) + for sampled_var, vals in variables.items(): + sampler.add_variable(sampled_var) + sampled_var.use_grid(construction="custom", kind="value", values=sorted(vals)) + for var_name, val in consts.items(): + sampler.add_constant(var_name, val) + + # Number of "denoises" for the sampler is the number of samples it should take + sampler.denoises = case.get_num_samples() + + # If there are any case labels, make a variable group for those and add it to the "grid" PointSet. + # These labels also need to get added to the sampler as constants. + grid_results = self._template.find("DataObjects/PointSet[@name='grid']") # type: PointSet + labels = case.get_labels() + if labels: + vargroup = self._create_case_labels_vargroup(labels) + self._add_snippet(vargroup) + grid_results.outputs.append(vargroup.name) + self._add_labels_to_sampler(sampler, labels) + + # Update the parallel settings based on the number of sampled variables if the number of outer parallel runs + # was not specified before. + if case.outerParallel == 0 and case.useParallel: + batch_size = sampler.num_sampled_vars + 1 + self._set_batch_size(batch_size, case) + + +class InnerTemplate(RavenTemplate): + """ Template for the inner workflow of a bilevel problem """ + write_name = "inner.xml" + + def __init__(self) -> None: + """ + Constructor + @ In, None + @ Out, None + """ + super().__init__() + self._dispatch_results_name = "" # str, keeps track of the name of the Database or OutStream used pass dispatch + # data to the outer workflow + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + case = kwargs["case"] + components = kwargs["components"] + + case_name = self.namingTemplates["jobname"].format(case=case.name, io="i") + self._set_case_name(case_name) + self._initialize_runinfo(case) + + # Set the index variable names for the time index names + self._set_time_vars(case.get_time_name(), case.get_year_name()) + + # Figure out econ metrics are being used for the case + # - econ metrics (from case obj), total activity variables (assembled from components list) + # - add to output groups GRO_dispatch_out, GRO_timeseries_out_scalar + # - add to metrics data object (arma_metrics PointSet) + activity_vars = get_component_activity_vars(components, self.namingTemplates["tot_activity"]) + econ_vars = case.get_econ_metrics(nametype="output") + output_vars = econ_vars + activity_vars + self._template.find("VariableGroups/Group[@name='GRO_dispatch_out']").variables.extend(output_vars) + self._template.find("VariableGroups/Group[@name='GRO_timeseries_out_scalar']").variables.extend(output_vars) + self._template.find("DataObjects/PointSet[@name='arma_metrics']").outputs.extend(output_vars) + + # Figure out what result statistics are being used + vg_final_return = self._template.find("VariableGroups/Group[@name='GRO_metrics_stats']") + results_vars = self._get_statistical_results_vars(case, components) + vg_final_return.variables.extend(results_vars) + + # Fill out the econ postprocessor statistics + econ_pp = self._template.find("Models/PostProcessor[@name='statistics']") + for stat, variable in self._get_stats_for_econ_postprocessor(case, econ_vars, activity_vars): + econ_pp.append(stat.to_element(variable)) + + # Work out how the inner results should be routed back to the outer + self._handle_data_inner_to_outer(case) + + def get_sampler_path(self) -> str: + """ + Getter for the pipe-delimited path to the first sampler in the workflow + @ In, None + @ Out, path, str, the path to the sampler + """ + sampler = self._template.find("Samplers")[0] + path = f"Samplers|{sampler.tag}@name:{sampler.name}" + return path + + def _handle_data_inner_to_outer(self, case: HeronCase) -> None: + """ + Set up either a Database or DataObject for the outer workflow to read + @ In, case, HeronCase, the HERON case object + @ Out, None + """ + # Work out how the inner results should be routed back to the outer + metrics_stats = self._template.find("DataObjects/PointSet[@name='metrics_stats']") + write_metrics_stats = self._template.find("Steps/IOStep[@name='database']") + self._dispatch_results_name = "disp_results" + data_handling = case.data_handling["inner_to_outer"] + if data_handling == "csv": + disp_results = PrintOutStream(self._dispatch_results_name) + disp_results.source = metrics_stats + else: # default to NetCDF handling + disp_results = NetCDF(self._dispatch_results_name) + disp_results.read_mode = "overwrite" + self._add_snippet(disp_results) + write_metrics_stats.add_output(disp_results) + + def get_dispatch_results_name(self) -> str: + """ + Gets the name of the Database or OutStream used to export the dispatch results to the outer workflow + @ In, None + @ Out, disp_results_name, str, the name of the dispatch results object + """ + if not self._dispatch_results_name: + raise ValueError("No dispatch results object name has been set! Perhaps the inner workflow hasn't " + "been created yet?") + return self._dispatch_results_name + + def _initialize_runinfo(self, case: HeronCase) -> None: + """ + Initializes the RunInfo node of the workflow + @ In, case, Case, the HERON Case object + @ Out, None + """ + run_info = self._template.find("RunInfo") + + # parallel settings + if case.innerParallel > 0: + run_info.use_internal_parallel = True + run_info.batch_size = case.innerParallel + else: + run_info.batch_size = 1 + + def _add_case_labels_to_sampler(self, case_labels: dict[str, str], sampler: Sampler) -> None: + """ + Adds case labels to relevant variable groups + @ In, case_labels, dict[str, str], the case labels + @ In, sampler, Sampler, the sampler to add labels to + @ Out, None + """ + if not case_labels: + return + + vg_case_labels = VariableGroup("GRO_case_labels") + self._add_snippet(vg_case_labels) + self._template.find("VariableGroups/Group[@name='GRO_timeseries_in_scalar']").variables.append(vg_case_labels.name) + self._template.find("VariableGroups/Group[@name='GRO_dispatch_in_scalar']").variables.append(vg_case_labels.name) + for k, label_val in case_labels.items(): + label_name = self.namingTemplates["variable"].format(unit=k, feature="label") + vg_case_labels.variables.append(label_name) + sampler.add_constant(label_name, label_val) + + def _set_time_vars(self, time_name: str, year_name: str) -> None: + """ + Update variable groups and data objects to have the correct time variable names. + @ In, time_name, str, name of time variable + @ In, year_name, str, name of year variable + @ Out, None + """ + group = self._template.find("VariableGroups/Group[@name='GRO_dispatch']") + group.variables.extend([time_name, year_name]) + + for time_index in self._template.findall("DataObjects/DataSet/Index[@var='Time']"): + time_index.set("var", time_name) + + for year_index in self._template.findall("DataObjects/DataSet/Index[@var='Year']"): + year_index.set("var", year_name) + + def _add_uncertain_econ_params(self, + sampler: Sampler, + variables: list[SampledVariable], + distributions: list[Distribution]) -> VariableGroup: + """ + Add uncertain economic parameter variables to the sampler and appropriate variable groups + @ In, sampler, Sampler, the sampler + @ In, variables, list[SampledVariable], variables to be sampled + @ In, distributions, list[Distribution], distributions to be sampled from + @ Out, vg_econ_uq, VariableGroup, a VariableGroup with the economic parameter names + """ + vg_econ_uq = self._template.find("VariableGroups/Group[@name='GRO_UQ']") + if vg_econ_uq is None: + vg_econ_uq = VariableGroup("GRO_UQ") + self._add_snippet(vg_econ_uq) + # Add the SampledVariable and Distribution nodes to the appropriate locations + for samp_var, dist in zip(variables, distributions): + self._add_snippet(dist) + vg_econ_uq.variables.append(samp_var.name) + sampler.add_variable(samp_var) + return vg_econ_uq + + def _add_constant_caps_to_sampler(self, sampler: Sampler, components: list[Component]) -> None: + """ + Add capacity values to the sampler as constants + @ In, sampler, Sampler, the sampler + @ In, components, list[Component], the case components + @ Out, None + """ + capacities_vargroup = self._template.find("VariableGroups/Group[@name='GRO_capacities']") # type: VariableGroup + capacities_vars = get_capacity_vars(components, self.namingTemplates["variable"]) + capacities_vargroup.variables.extend(list(capacities_vars)) + for k, v in capacities_vars.items(): + val = "" if isinstance(v, list) else v # empty string is overwritten by capacity from outer in write_inner.py + sampler.add_constant(k, val) + + +class InnerTemplateSyntheticHistory(InnerTemplate): + """ Template for the inner workflow of a bilevel problem that uses a static history source """ + template_name = "inner_synth.xml" + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + case = kwargs["case"] + components = kwargs["components"] + sources = kwargs["sources"] + + # Add ARMA ROMs to ensemble model + ensemble_model = self._template.find("Models/EnsembleModel") + self._add_time_series_roms(ensemble_model, case, sources) + + # Determine which variables are sampled by the Monte Carlo sampler + mc = self._template.find("Samplers/MonteCarlo[@name='mc_arma_dispatch']") # type: MonteCarlo + # default sampler init + mc.init_seed = 42 + mc.init_limit = 3 + # Add capacities as constants to the sampler + self._add_constant_caps_to_sampler(mc, components) + + # Add case labels to sampler and variable groups, if any labels have been provided + self._add_case_labels_to_sampler(case.get_labels(), mc) + + # See if there are any uncertain cashflow parameters that need to get added to the sampler + sampled_vars, distributions = self._get_uncertain_cashflow_params(components) + if len(sampled_vars) > 0: + # Create a VariableGroup for the uncertain econ parameters + vg_econ_uq = self._add_uncertain_econ_params(mc, sampled_vars, distributions) + self._template.find("VariableGroups/Group[@name='GRO_dispatch_in_scalar']").variables.append(vg_econ_uq.name) + self._template.find("VariableGroups/Group[@name='GRO_timeseries_in_scalar']").variables.append(vg_econ_uq.name) + + +class InnerTemplateStaticHistory(InnerTemplate): + """ Template for the inner workflow of a bilevel problem """ + template_name = "inner_static.xml" + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + case = kwargs["case"] + components = kwargs["components"] + sources = kwargs["sources"] + + # Create the custom sampler used to provide static history data to the model + custom_sampler = CustomSampler("static_history_sampler") + self._configure_static_history_sampler(custom_sampler, case, sources) + + # Add case labels to the sampler + self._add_case_labels_to_sampler(case.get_labels(), custom_sampler) + + # Add the outer capacities as constants here + # - component capacities (constants) + # - add variables to GRO_capacities + capacities_vargroup = self._template.find("VariableGroups/Group[@name='GRO_capacities']") # type: VariableGroup + capacities_vars = get_capacity_vars(components, self.namingTemplates["variable"]) + capacities_vargroup.variables.extend(list(capacities_vars)) + for k, v in capacities_vars.items(): + val = "" if isinstance(v, list) else v # empty string is overwritten by capacity from outer in write_inner.py + custom_sampler.add_constant(k, val) + + # See if there are any uncertain cashflow parameters. If so, we need to create a MonteCarlo sampler to sample + # from those distributions and tie the MonteCarlo and CustomSampler samplers together with an EnsembleForward + # sampler. + sampled_vars, distributions = self._get_uncertain_cashflow_params(components) + if len(sampled_vars) > 0: + # Create a MonteCarlo sampler + mc = MonteCarlo("mc") + mc.init_seed = 42 + mc.init_limit = case.get_num_samples() + # Create a VariableGroup for the uncertain econ parameters + vg_econ_uq = self._add_uncertain_econ_params(mc, sampled_vars, distributions) + self._template.find("VariableGroups/Group[@name='GRO_dispatch_in_scalar']").variables.append(vg_econ_uq.name) + self._template.find("VariableGroups/Group[@name='GRO_timeseries_in_scalar']").variables.append(vg_econ_uq.name) + + # Combine the MonteCarlo and CustomSampler samplers in an EnsembleForward sampler. + ensemble_sampler = self._create_ensemble_forward_sampler(custom_sampler, mc) + self._add_snippet(ensemble_sampler) + + sampler = ensemble_sampler + else: + self._add_snippet(custom_sampler) + sampler = custom_sampler + + # Set the sampler to be used in the main MultiRun + multirun = self._template.find("Steps/MultiRun[@name='arma_sampling']") + multirun.add_sampler(sampler) diff --git a/templates/cash.xml b/templates/cash.xml deleted file mode 100644 index 2ff8bced..00000000 --- a/templates/cash.xml +++ /dev/null @@ -1,9 +0,0 @@ - - - - 0.00 - 0.000 - 0.00 - CONP|CA - - \ No newline at end of file diff --git a/templates/debug_template.py b/templates/debug_template.py new file mode 100644 index 00000000..f32493a8 --- /dev/null +++ b/templates/debug_template.py @@ -0,0 +1,282 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + A template for HERON's debug mode + + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-23 +""" +from .raven_template import RavenTemplate + +from .snippets.models import EnsembleModel +from .snippets.outstreams import HeronDispatchPlot, TealCashFlowPlot +from .snippets.runinfo import RunInfo +from .snippets.samplers import MonteCarlo, CustomSampler +from .snippets.variablegroups import VariableGroup + +from .heron_types import HeronCase, Component, Source +from .naming_utils import get_capacity_vars, get_component_activity_vars, get_cashflow_names +from .xml_utils import find_node + + +class DebugTemplate(RavenTemplate): + """ Sets up a flat RAVEN run for debug mode """ + template_name = "debug.xml" + write_name = "outer.xml" + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + case = kwargs["case"] + components = kwargs["components"] + sources = kwargs["sources"] + + # RunInfo initialization + case_name = self.namingTemplates["jobname"].format(case=case.name, io="o") + self._set_case_name(case_name) + self._initialize_runinfo(case) + + # First, handle aspects of the workflow common across all debug runs + self._update_vargroups(case, components, sources) + self._update_dataset_indices(case) + + # Add optional plots + debug_iostep = self._template.find("Steps/IOStep[@name='debug_output']") + if case.debug["dispatch_plot"]: + disp_plot = self._make_dispatch_plot(case) + self._add_snippet(disp_plot) + debug_iostep.add_output(disp_plot) + + if case.debug["cashflow_plot"]: + cashflow_plot = self._make_cashflow_plot() + self._add_snippet(cashflow_plot) + debug_iostep.add_output(cashflow_plot) + + # Then, figure out how things will be sampled for the debug run. + # We need to handle 3 separate sampler configurations: + # 1. MonteCarlo sampler: Used for sampling from time series ROM and distributions for swept/optimized + # capacities without a debug value and uncertain cashflow parameters. + # 2. CustomSampler sampler: Used for getting a static history from a CSV file. + # 3. EnsembleForward sampler: Used for combining the two where needed. + # If there is a MonteCarlo sampler being used, we'll add any constants (like capacities with debug values) + # there. Otherwise, if only a CustomSampler is used, we'll add them to the CustomSampler. + + # What time series sources does our case have? + has_arma_source = any(s.is_type("ARMA") for s in sources) + has_csv_source = any(s.is_type("CSV") for s in sources) + + # What variables need to be sampled and which are constants? + cap_vars, cap_consts = self._create_sampler_variables(case, components) # capacities + has_sampled_capacities = len(cap_vars) > 0 + + # Are there any uncertain cashflow parameters which need to get sampled? + cashflow_vars, cashflow_dists = self._get_uncertain_cashflow_params(components) + has_uncertain_cashflows = len(cashflow_vars) > 0 + + # Create the Monte Carlo sampler, if needed + if any([has_arma_source, has_sampled_capacities, has_uncertain_cashflows]): + # Okay, we know we need it, so let's make it. + monte_carlo = MonteCarlo("mc") + + # Set number of samples for sampler + monte_carlo.denoises = case.get_num_samples() + monte_carlo.init_limit = case.get_num_samples() + + # Set up case to use synthetic history ROM + if has_arma_source: + self._use_time_series_rom(monte_carlo, case, sources) + + # Add capacities to sampler + for sampled_var in cap_vars: + monte_carlo.add_variable(sampled_var) + for var_name, val in cap_consts.items(): + monte_carlo.add_constant(var_name, val) + + # Add uncertain cashflow parameters + if has_uncertain_cashflows: + vg_econ_uq = find_node(self._template, "VariableGroups/Group[@name='GRO_UQ']") # type: VariableGroup + self._template.find("VariableGroups/Group[@name='GRO_dispatch_in_scalar']").variables.append(vg_econ_uq.name) + self._template.find("VariableGroups/Group[@name='GRO_timeseries_in_scalar']").variables.append(vg_econ_uq.name) + # Add the SampledVariable and Distribution nodes to the appropriate locations + for samp_var, dist in zip(cashflow_vars, cashflow_dists): + self._add_snippet(dist) + vg_econ_uq.variables.append(samp_var.name) + monte_carlo.add_variable(samp_var) + else: + monte_carlo = None + + # The CustomSampler is only needed if there is a static history from CSV + if has_csv_source: + custom_sampler = CustomSampler("static_hist_sampler") + self._configure_static_history_sampler(custom_sampler, case, sources) + else: + custom_sampler = None + + # If only a CustomSampler if being used, the capacity constants need to be added to the custom sampler + if monte_carlo is None and custom_sampler is not None: + for var_name, val in cap_consts.items(): + custom_sampler.add_constant(var_name, val) + + # If we need both the MonteCarlo sampler and the CustomSampler, add them both to an EnsembleForward sampler so they + # can be used together. + multirun_step = self._template.find("Steps/MultiRun[@name='debug']") + if monte_carlo and custom_sampler: + ensemble_sampler = self._create_ensemble_forward_sampler([monte_carlo, custom_sampler], name="ensemble_sampler") + self._add_snippet(ensemble_sampler) + multirun_step.add_sampler(ensemble_sampler) + elif monte_carlo is not None: + self._add_snippet(monte_carlo) + multirun_step.add_sampler(monte_carlo) + elif custom_sampler is not None: + self._add_snippet(custom_sampler) + multirun_step.add_sampler(custom_sampler) + else: + raise ValueError("Nothing that requires a sampler was found.") + + # Add the model and file inputs to the main multirun step + multirun = self._template.find("Steps/MultiRun[@name='debug']") + model = self._template.find("Models/EnsembleModel") or self._template.find("Models/ExternalModel") + multirun.add_model(model) + for func in self._get_function_files(sources): + multirun.add_input(func) + + def _initialize_runinfo(self, case: HeronCase) -> None: + """ + Initializes the RunInfo node of the workflow + @ In, case_name, str, optional, the case name + @ Out, None + """ + run_info = self._template.find("RunInfo") # type: RunInfo + + # Use the outer parallel settings for flat run modes + batch_size = min(case.outerParallel, 1) * min(case.innerParallel, 1) + run_info.use_internal_parallel = batch_size > 1 + + if case.useParallel: + # Fills in parallel settings for template RunInfo from case. Also applies pre-sets for known + # hostnames (e.g. sawtooth, bitterroot), as specified in the HERON/templates/parallel/*.xml files. + run_info.set_parallel_run_settings(case.parallelRunInfo) + + def _use_time_series_rom(self, sampler: MonteCarlo, case: HeronCase, sources: list[Source]) -> None: + """ + Sets the workflow up to sample a time history from a PickledROM model + @ In sampler, MonteCarlo, a MonteCarlo sampler snippet + @ In, case, HeronCase, the HERON case + @ In, sources, list[Source], external models, data, and functions + @ Out, None + """ + # If a time series PickledROM is being used, it will need to be combined with the HERON.DispatchManager external + # model with an EnsembleModel. + ensemble = EnsembleModel("sample_and_dispatch") + self._add_snippet(ensemble) + + # Load the time series ROM(s) from file and add to the ensemble model. + # Includes steps to load and print metadata for the pickled time series ROMs + self._add_time_series_roms(ensemble, case, sources) + + # Fetch the dispatch model and add it to the ensemble. The model and associated data objects already exist + # in the template XML, so we find those and add them to the model. + dispatcher = self._template.find("Models/ExternalModel[@subType='HERON.DispatchManager']") + dispatcher_assemb = dispatcher.to_assembler_node("Model") + # FIXME: I don't know why this is the case with RAVEN, but the dispatch_placeholder data object MUST come before + # any function nodes, or it errors out. This is bad XML practice, which should be independent of order! + disp_placeholder = self._template.find("DataObjects/PointSet[@name='dispatch_placeholder']") + dispatcher_assemb.append(disp_placeholder.to_assembler_node("Input")) # THIS COMES FIRST + for func in self._get_function_files(sources): + dispatcher_assemb.append(func.to_assembler_node("Input")) # THEN ADD THESE + disp_eval = self._template.find("DataObjects/DataSet[@name='dispatch_eval']") + dispatcher_assemb.append(disp_eval.to_assembler_node("TargetEvaluation")) + ensemble.append(dispatcher_assemb) + + # A scaling constant needs to be added to the MonteCarlo sampler for the + sampler.add_constant("scaling", 1.0) + + def _update_vargroups(self, case: HeronCase, components: list[Component], sources: list[Source]) -> None: + """ + Updates existing variable group nodes with index and variable names + @ In, case, HeronCase, the HERON case object + @ In, components, list[Component], the case components + @ In, sources, list[Source], the case data sources + @ Out, None + """ + # Fill out capacities vargroup + capacities_vargroup = self._template.find("VariableGroups/Group[@name='GRO_capacities']") + capacities_vars = list(get_capacity_vars(components, self.namingTemplates["variable"], debug=True)) + capacities_vargroup.variables.extend(capacities_vars) + + # Add time indices to GRO_time_indices + self._template.find("VariableGroups/Group[@name='GRO_time_indices']").variables = [ + case.get_time_name(), + case.get_year_name() + ] + + # Dispatch variables + dispatch_vars = get_component_activity_vars(components, self.namingTemplates["dispatch"]) + self._template.find("VariableGroups/Group[@name='GRO_full_dispatch']").variables.extend(dispatch_vars) + + # Cashflows + cfs = get_cashflow_names(components) + self._template.find("VariableGroups/Group[@name='GRO_cashflows']").variables.extend(cfs) + + # Time history sources + group = self._template.find("VariableGroups/Group[@name='GRO_debug_synthetics']") # type: VariableGroup + for source in filter(lambda x: x.type in ["ARMA", "CSV"], sources): + synths = source.get_variable() + group.variables.extend(synths) + + # Figure out which econ metrics are being used for the case + activity_vars = get_component_activity_vars(components, self.namingTemplates["tot_activity"]) + econ_vars = case.get_econ_metrics(nametype="output") + output_vars = econ_vars + activity_vars + self._template.find("VariableGroups/Group[@name='GRO_dispatch_out']").variables.extend(output_vars) + self._template.find("VariableGroups/Group[@name='GRO_timeseries_out_scalar']").variables.extend(output_vars) + + def _update_dataset_indices(self, case: HeronCase) -> None: + """ + Update the Index node variables for all DataSet nodes to correctly reflect the provided macro, micro, and cluster + index names + @ In, case, HeronCase, the HERON case + @ Out, None + """ + # Configure dispatch DataSet indices + time_name = case.get_time_name() + year_name = case.get_year_name() + cluster_name = self.namingTemplates["cluster_index"] + + for time_index in self._template.findall(".//DataSet/Index[@var='Time']"): + time_index.set("var", time_name) + + for year_index in self._template.findall(".//DataSet/Index[@var='Year']"): + year_index.set("var", year_name) + + for cluster_index in self._template.findall(".//DataSet/Index[@var='_ROM_Cluster']"): + cluster_index.set("var", cluster_name) + + def _make_dispatch_plot(self, case: HeronCase) -> HeronDispatchPlot: + """ + Make a HERON dispatch plot + @ In, case, HeronCase, the HERON case + @ Out, disp_plot, HeronDispatchPlot, the dispatch plot node + """ + disp_plot = HeronDispatchPlot("dispatchPlot") + dispatch_dataset = self._template.find("DataObjects/DataSet[@name='dispatch']") + disp_plot.source = dispatch_dataset + disp_plot.macro_variable = case.get_year_name() + disp_plot.micro_variable = case.get_time_name() + disp_plot.signals.append("GRO_debug_synthetics") + return disp_plot + + def _make_cashflow_plot(self) -> TealCashFlowPlot: + """ + Make a TEAL cashflow plot + @ In, None, + @ Out, cashflow_plot, TealCashFlowPlot, the cashflow plot node + """ + cashflow_plot = TealCashFlowPlot("cashflow_plot") + cashflows = self._template.find("DataObjects/HistorySet[@name='cashflows']") + cashflow_plot.source = cashflows + return cashflow_plot diff --git a/templates/decorators.py b/templates/decorators.py new file mode 100644 index 00000000..9b07ef2a --- /dev/null +++ b/templates/decorators.py @@ -0,0 +1,327 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Decorators + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-11 +""" +from typing import Any + + +def _coerce_to_list(value: Any) -> list[Any]: + """ + Try to make value a list that makes sense. Special handling for strings is taken to avoid splitting a + string into characters. If commas are present in the string, it is assumed the string has comma-delimited + values. + @ In, value, Any, the value to coerce + @ Out, coerced, list[Any], the coerced value object + """ + if not value: + # empty Collections (list, set, tuple, str, dict, ...) or None are all falsy + coerced = [] + elif isinstance(value, str): + # strings: either a comma-delimited list of values in the string or just a single item + if "," in value: + coerced = [s.strip() for s in value.split(",")] + else: + coerced = [value] + else: + # Just try coercing to a list otherwise and hope for the best. This should work without throwing + # an exception for anything iterable. + coerced = list(value) + return coerced + + +class ListWrapper(list): + """ + A wrapper class which emulates a list (and subclasses list for duck typing) which interfaces with a property + """ + def __init__(self, property_instance, obj): + """ + Constructor + @ In, property_instance, listproperty, a listproperty object + @ In, obj, Any, some object that can be made to be list-like (see _coerce_to_list function) + @ Out, None + """ + self.property_instance = property_instance + self.obj = obj + + def _get_list(self): + """ + Private getter for wrapped list + @ In, None + @ Out, obj, list, obj as a list + """ + return _coerce_to_list(self.property_instance.fget(self.obj)) + + def _set_list(self, value): + """ + Private setter for the obj list + @ In, value, coercible to list, object to set obj + @ Out, None + """ + self.property_instance.fset(self.obj, _coerce_to_list(value)) + + def __getitem__(self, index): + """ + Get item from list + @ In, index, int, index of item to get + @ Out, item, Any, the list item + """ + return self._get_list()[index] + + def __setitem__(self, index, value): + """ + Get item in list + @ In, index, int, index of item to set + @ In, value, Any, the value to store at the list index + @ Out, None + """ + lst = self._get_list() + lst[index] = value + self._set_list(lst) + + def __delitem__(self, index): + """ + Delete item from list + @ In, index, int, the index of the item to delete + @ Out, None + """ + lst = self._get_list() + del lst[index] + self._set_list(lst) + + def append(self, obj): + """ + Append to list + @ In, obj, Any, the object to append + @ Out, None + """ + lst = self._get_list() + lst.append(obj) + self._set_list(lst) + + def extend(self, iterable): + """ + Extend list with iterable + @ In, iterable, Iterable, the iterable with which to extend the lsit + @ Out, None + """ + lst = self._get_list() + lst.extend(iterable) + self._set_list(lst) + + def insert(self, index, obj): + """ + Insert an object into the list + @ In, index, int, the index to insert at + @ In, obj, Any, the object to insert + @ Out, None + """ + lst = self._get_list() + lst.insert(index, obj) + self._set_list(lst) + + def remove(self, value): + """ + Remove an object from the list + @ In, value, Any, the value to remove + @ Out, None + """ + lst = self._get_list() + lst.remove(value) + self._set_list(lst) + + def pop(self, index=-1): + """ + Remove an object from the list by index and return the object + @ In, index, int, optional, the index to pull + @ Out, val, Any, the popped item + """ + lst = self._get_list() + val = lst.pop(index) + self._set_list(lst) + return val + + def clear(self): + """ + Clear the list + @ In, None + @ Out, None + """ + self._set_list([]) + + def index(self, value): + """ + Get the index of a value in the list + @ In, value, Any, the value to find in the list + @ Out, index, int, the index of value + """ + return self._get_list().index(value) + + def count(self, value): + """ + Count the number of occurrences of a value + @ In, value, Any, the value to count + @ Out, count, int, the number of occurrences + """ + return self._get_list().count(value) + + def sort(self, *, key=None, reverse=False): + """ + Sort the list + @ In, key, Callable[[Any], Any], optional, a function of one argument that is used to extract + a comparison key from the values of the list + @ In, reverse, bool, optional, if the sort should be done in descending order + @ Out, None + """ + lst = self._get_list() + lst.sort(key=key, reverse=reverse) + self._set_list(lst) + + def reverse(self): + """ + Reverse the list + @ In, None + @ Out, None + """ + lst = self._get_list() + lst.reverse() + self._set_list(lst) + + def copy(self): + """ + Make a copy of the list + @ In, None + @ Out, copy, list, the copy of the list + """ + return self._get_list().copy() + + def __len__(self): + """ + The length of the list + @ In, None + @ Out, length, int, the length + """ + return len(self._get_list()) + + def __iter__(self): + """ + Get an iterator over the list + @ In, None + @ Out, iter, Iterable, an iterator for the list + """ + return iter(self._get_list()) + + def __repr__(self): + """ + A representation of the list + @ In, None + @ Out, repr, the list representation + """ + return repr(self._get_list()) + + def __eq__(self, other): + """ + Equality comparison + @ In, other, list, list to compare to + @ Out, equal, bool, if the lists are equal + """ + return self._get_list() == other + + def __contains__(self, value): + """ + Does list contain value? + @ In, value, Any, value to look for + @ Out, contains, bool, if the value is in the list + """ + return value in self._get_list() + + +class listproperty: + """ + A approximation of the built-in "property" function/decorator, with additional logic for getting/setting values + which are lists (or more precisely, ListWrapper objects) in a way that allows for list operations (e.g. append, + extend, insert) on the property. + """ + def __init__(self, fget=None, fset=None, fdel=None, doc=None): + """ + Constructor + @ In, fget, Callable, optional, getter function + @ In, fset, Callable, optional, setter function + @ In, fdel, Callable, optional, deleter function + @ In, doc, str, optional, a docstring description of the property + """ + self.fget = fget + self.fset = fset + self.fdel = fdel + if doc is None and fget is not None: + doc = fget.__doc__ + self.__doc__ = doc + + def __set_name__(self, owner, name): + """ + Set the name of the property + @ In, owner, type, the class owning the property (unused, included for consistency with 'property' builtin) + @ In, name, str, the name of the property + @ Out, None + """ + self.__name__ = name + + def __get__(self, obj, objtype=None): + """ + Get the property value + @ In, obj, Any, the instance from which the property is accessed + @ In, objtype, type, optional, the type of the instance (unused, included for consistency with 'property' builtin) + @ Out, value, Any, the value of the property + """ + if obj is None: + return self + if self.fget is None: + raise AttributeError("unreadable attribute") + return ListWrapper(self, obj) + + def __set__(self, obj, value): + """ + Set the property value + @ In, obj, Any, the instance on which the property is set + @ In, value, Any, the value to be set + @ Out, None + """ + if self.fset is None: + raise AttributeError("can't set attribute") + self.fset(obj, value) + + def __delete__(self, obj): + """ + Delete the property value + @ In, obj, Any, the instance from which the property is deleted + @ Out, None + """ + if self.fdel is None: + raise AttributeError("can't delete attribute") + self.fdel(obj) + + def getter(self, fget): + """ + Set getter function for property + @ In, fget, Callable, getter function + @ Out, obj, self with set getter + """ + return type(self)(fget, self.fset, self.fdel, self.__doc__) + + def setter(self, fset): + """ + Set setter function for property + @ In, fget, Callable, setter function + @ Out, obj, self with set setter + """ + return type(self)(self.fget, fset, self.fdel, self.__doc__) + + def deleter(self, fdel): + """ + Set deleter function for property + @ In, fdel, Callable, deleter function + @ Out, obj, self with set deleter + """ + return type(self)(self.fget, self.fset, fdel, self.__doc__) diff --git a/templates/flat_templates.py b/templates/flat_templates.py new file mode 100644 index 00000000..95f6a1db --- /dev/null +++ b/templates/flat_templates.py @@ -0,0 +1,118 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Templates for workflows which can be "flat" RAVEN workflows (no need for RAVEN-runs-RAVEN) + + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-23 +""" +import os + +from .heron_types import HeronCase, Component, Source + +from .raven_template import RavenTemplate + +from .snippets.dataobjects import PointSet +from .snippets.runinfo import RunInfo +from .snippets.samplers import CustomSampler, EnsembleForward, Grid +from .snippets.steps import MultiRun +from .snippets.variablegroups import VariableGroup + +from .naming_utils import get_capacity_vars + + +class FlatMultiConfigTemplate(RavenTemplate): + """ + A template for RAVEN workflows which do not consider uncertainty from sources which affect the system dispatch (one + time history, no uncertain variable costs for dispatchable components). Many system configurations may be considered. + """ + template_name = "flat_multi_config.xml" + write_name = "outer.xml" + + # With static histories, the stats that are used shouldn't require multiple samples. Therefore, we drop the + # sigma and variance default stat names for the static history case here. + DEFAULT_STATS_NAMES = { + "opt": ["expectedValue", "median"], + "sweep": ["maximum", "minimum", "percentile", "samples"] + } + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + super().createWorkflow(**kwargs) + case = kwargs["case"] + components = kwargs["components"] + sources = kwargs["sources"] + + case_name = self.namingTemplates["jobname"].format(case=case.name, io="o") + self._set_case_name(case_name) + self._initialize_runinfo(case) + + # Set up some helpful variable groups + capacities_vargroup = self._template.find("VariableGroups/Group[@name='GRO_capacities']") # type: VariableGroup + capacities_vars = list(get_capacity_vars(components, self.namingTemplates["variable"])) + capacities_vargroup.variables.extend(capacities_vars) + + results_vargroup = self._template.find("VariableGroups/Group[@name='GRO_results']") # type: VariableGroup + results_vars = self._get_deterministic_results_vars(case, components) + results_vargroup.variables.extend(results_vars) + + # Define a sampler for handling the static history + static_hist_sampler = self._template.find("Samplers/EnsembleForward/CustomSampler") # type: CustomSampler + self._configure_static_history_sampler(static_hist_sampler, case, sources, scaling=None) + + # Define a grid sampler, a data object to store the sweep results, and an outstream to print those results + grid_sampler = self._template.find("Samplers/EnsembleForward/Grid") # type: Grid + grid_results = self._template.find("DataObjects/PointSet[@name='grid']") # type: PointSet + + variables, consts = self._create_sampler_variables(case, components) + for sampled_var, vals in variables.items(): + grid_sampler.add_variable(sampled_var) + sampled_var.use_grid(construction="custom", kind="value", values=sorted(vals)) + + ensemble_sampler = self._template.find("Samplers/EnsembleForward") # type: EnsembleForward + for var_name, val in consts.items(): + ensemble_sampler.add_constant(var_name, val) + + # If there are any case labels, make a variable group for those and add it to the "grid" PointSet. + # These labels also need to get added to the sampler as constants. + labels = case.get_labels() + if labels: + vargroup = self._create_case_labels_vargroup(labels) + self._add_snippet(vargroup) + grid_results.outputs.append(vargroup.name) + self._add_labels_to_sampler(grid_sampler, labels) + + # Use a MultiRun to run to the model over the grid points + multirun = self._template.find("Steps/MultiRun[@name='sweep']") # type: MultiRun + for func in self._get_function_files(sources): + multirun.add_input(func) + + # Update the parallel settings based on the number of sampled variables if the number of outer parallel runs + # was not specified before. + if case.outerParallel == 0 and case.useParallel: + sampler = self._template.find("Samplers/Grid") + run_info = self._template.find("RunInfo") + case.outerParallel = sampler.num_sampled_vars + 1 + run_info.batch_size = case.outerParallel + run_info.internal_parallel = True + + def _initialize_runinfo(self, case: HeronCase) -> None: + """ + Initializes the RunInfo node of the workflow + @ In, case, Case, the HERON Case object + @ Out, None + """ + run_info = self._template.find("RunInfo") # type: RunInfo + + # parallel + batch_size = min(case.outerParallel, 1) * min(case.innerParallel, 1) + run_info.use_internal_parallel = batch_size > 1 + + if case.useParallel: + # Fills in parallel settings for template RunInfo from case. Also appliespre-sets for known + # hostnames (e.g. sawtooth, bitterroot), as specified in the HERON/templates/parallel/*.xml files. + run_info.set_parallel_run_settings(case.parallelRunInfo) diff --git a/templates/heron_types.py b/templates/heron_types.py new file mode 100644 index 00000000..133493df --- /dev/null +++ b/templates/heron_types.py @@ -0,0 +1,15 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Alias types of major HERON classes for easier type hinting + + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-23 +""" +from typing import TypeAlias + +# load utils +from .imports import Case, Placeholder, Component, ValuedParam + +HeronCase: TypeAlias = Case +Source: TypeAlias = Placeholder diff --git a/templates/imports.py b/templates/imports.py new file mode 100644 index 00000000..c4c4d4ef --- /dev/null +++ b/templates/imports.py @@ -0,0 +1,31 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Imports from ravenframework and HERON.src + + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-23 +""" +import sys +from pathlib import Path + + +# import HERON +sys.path.append(str(Path(__file__).parent.parent.parent)) +from HERON.src.base import Base +from HERON.src.Cases import Case +from HERON.src.Components import Component +from HERON.src.Placeholders import Placeholder +from HERON.src.ValuedParams import ValuedParam +import HERON.src._utils as hutils +sys.path.pop() + +# where is ravenframework? +RAVEN_LOC = Path(hutils.get_raven_loc()) + +# import needed ravenframework modules +sys.path.append(str(RAVEN_LOC)) +from ravenframework.utils import xmlUtils +from ravenframework.InputTemplates.TemplateBaseClass import Template +from ravenframework.Distributions import returnInputParameter +sys.path.pop() diff --git a/templates/inner.xml b/templates/inner.xml deleted file mode 100644 index 2fcb7db4..00000000 --- a/templates/inner.xml +++ /dev/null @@ -1,138 +0,0 @@ - - - - . - arma_sampling, summarize, database - 1 - - - - - dispatch_placeholder - sample_and_dispatch - mc_arma_dispatch - arma_metrics - - - disp_full - disp_full - - - arma_metrics - statistics - metrics_stats - - - - metrics_stats - disp_results - - - - - - GRO_dispatch_in, GRO_dispatch_out - GRO_dispatch_in_scalar, GRO_dispatch_in_Time - - GRO_capacities, scaling - - GRO_armasamples_in, GRO_armasamples_out - GRO_armasamples_in_scalar - GRO_armasamples_out_scalar - scaling, GRO_capacities - - - - - - stepwise - - - - - - - GRO_armasamples_in - GRO_armasamples_out - - - - - - GRO_final_return - - - GRO_full_dispatch, GRO_dispatch, GRO_cashflows - GRO_full_dispatch, GRO_dispatch_in_Time - GRO_full_dispatch, GRO_dispatch_in_Time - GRO_full_dispatch, GRO_dispatch_in_Time - GRO_cashflows - - - GRO_dispatch_in - GRO_dispatch_in_Time - GRO_dispatch_in_Time - - - GRO_dispatch_in_scalar - - - - - - - - - - - GRO_dispatch, GRO_armasamples - - - - dispatch - dispatch_placeholder - dispatch_eval - - - - - - - - - ../../heron.lib - - - - - - - - - 42 - 3 - - 1.0 - - - - - - - - - - - csv - arma_samples - - - csv - metrics_stats - - - - diff --git a/templates/naming_utils.py b/templates/naming_utils.py new file mode 100644 index 00000000..b3a63761 --- /dev/null +++ b/templates/naming_utils.py @@ -0,0 +1,191 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Utility functions for getting variable names, cashflow names, objective function names, and so on. + + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-23 +""" +import itertools +from dataclasses import dataclass +from typing import Any +import xml.etree.ElementTree as ET + +from .heron_types import HeronCase, Component + + +@dataclass(frozen=True) +class Statistic: + """ + A dataclass for building statistic names and ET.Elements. Hopefully this helps cut down repeated parsing of variable + names and statistis meta info from case. + """ + name: str + prefix: str + threshold : str | None = None + percent: str | None = None + + def to_metric(self, variable: str) -> str: + """ + Get the name for this statistic of variable + @ In, variable, str, the variable name (e.g. NPV) + @ Out, varname, str, the name of a variable's statistic (e.g. mean_NPV) + """ + param = self.threshold or self.percent # threshold, percent, or None + parts = [self.prefix, param, variable] if param else [self.prefix, variable] + varname = "_".join(parts) + return varname + + def to_element(self, variable: str) -> ET.Element: + """ + Get the statistic as an ET.Element, as is given to BasicStatistics and EconomicRatio PostProcessor models + @ In, variable, str, the variable name (e.g. NPV) + @ Out, element, ET.Element, the variable statistic element + """ + element = ET.Element(self.name, prefix=self.prefix) + if self.threshold: + element.set("threshold", self.threshold) + if self.percent: + element.set("percent", self.percent) + element.text = variable + return element + +def get_statistics(stat_names: list[str], stat_meta: dict) -> list[Statistic]: + """ + Create Statistic objects for each statistic in stat_names + @ In, stat_names, list[str], names of statistics + @ In, stat_meta, dict, statistics meta data + @ Out, stats, list[Statistic], Statistic objects for each statistic of interest + """ + stats = [] + + for name in stat_names: + meta = stat_meta[name] + prefix = meta["prefix"] + percent = meta.get("percent", None) + if not isinstance(percent, list): + percent = [percent] + threshold = meta.get("threshold", None) + if not isinstance(threshold, list): + threshold = [threshold] + + for perc, thresh in itertools.product(percent, threshold): + new_stat = Statistic(name=name, prefix=prefix, threshold=thresh, percent=perc) + stats.append(new_stat) + + return stats + +def get_result_stats(names: list[str], stats: list[str], case: HeronCase) -> list[str]: + """ + Constructs the names of the statistics requested for output + @ In, names, list[str], result metric names (economics, component activities) + @ In, stats, list[str], statistic names + @ In, case, HeronCase, defining Case instance + @ Out, names, list[str], list of names of statistics requested for output + """ + stats_objs = get_statistics(stats, case.stats_metrics_meta) + stat_names = [stat.to_metric(name) for stat, name in itertools.product(stats_objs, names)] + return stat_names + +def get_capacity_vars(components: list[Component], name_template, *, debug=False) -> dict[str, Any]: + """ + Get dispatch variable names + @ In, components, list[Component], list of HERON components + @ In, name_template, str, naming template for dispatch variable name, expecting + keywords "component", "tracker", and "resource" + @ In, debug, bool, optional, keyword-only argument for whether or not to use the debug value of a capacity variable + @ Out, variables, dict[str, Any], variable name-value pairs + """ + variables = {} + + for component in components: + name = component.name + # treat capacity + ## we just need to make sure everything we need gets into the dispatch ensemble model. + ## For each interaction of each component, that means making sure the Function, ARMA, or constant makes it. + ## Constants from outer (namely sweep/opt capacities) are set in the MC Sampler from the outer + ## The Dispatch needs info from the Outer to know which capacity to use, so we can't pass it from here. + capacity = component.get_capacity(None, raw=True) + + if capacity.is_parametric(): + cap_name = name_template.format(unit=name, feature='capacity') + values = capacity.get_value(debug=debug) + variables[cap_name] = values + elif capacity.type in ['StaticHistory', 'SyntheticHistory', 'Function', 'Variable']: + # capacity is limited by a signal, so it has to be handled in the dispatch; don't include it here. + # OR capacity is limited by a function, and we also can't handle it here, but in the dispatch. + pass + else: + raise NotImplementedError + + return variables + +def get_component_activity_vars(components: list[Component], name_template: str) -> list[str]: + """ + Get dispatch variable names + @ In, components, list[Component], list of HERON components + @ In, name_template, str, naming template for dispatch variable name, expecting + keywords "component", "tracker", and "resource" + @ Out, variables, list[str], list of variable names + """ + variables = [] + + for component in components: + name = component.name + for tracker in component.get_tracking_vars(): + resource_list = sorted(list(component.get_resources())) + for resource in resource_list: + var_name = name_template.format(component=name, tracker=tracker, resource=resource) + variables.append(var_name) + + return variables + +def get_opt_objective(case: HeronCase) -> str: + """ + Get the name of the optimization objective + @ In, case, HeronCase, the HERON case object + @ Out, objective, str, the name of the objective + """ + # What statistic is used for the objective? + opt_settings = case.get_optimization_settings() + try: + statistic = opt_settings["stats_metric"]["name"] + except (KeyError, TypeError): + # FIXME: What about cases with only 1 history (not statistical)? + statistic = "expectedValue" # default to expectedValue + + meta = case.stats_metrics_meta[statistic] + stat_name = meta["prefix"] + param = meta.get("percent", None) or meta.get("threshold", None) + if isinstance(param, list): + param = param[0] + if param: + stat_name += f"_{param}" + + # What variable does the metric act on? + target_var, _ = case.get_opt_metric() + target_var_output_name = case.economic_metrics_meta[target_var]["output_name"] + + objective = f"{stat_name}_{target_var_output_name}" + return objective + +def get_cashflow_names(components: list[Component]) -> list[str]: + """ + Loop through components and collect all the full cashflow names + @ In, components, list[Component], list of HERON Component instances for this run + @ Out, cfs, list[str], list of cashflow full names e.g. {comp}_{cf}_CashFlow + """ + cfs = [] + for comp in components: + comp_name = comp.name + for cashflow in comp.get_cashflows(): + # User has specified to leave this cashflow out of the NPV calculation. Skip it. + if cashflow.is_npv_exempt(): + continue + cf_name = cashflow.name + name = f"{comp_name}_{cf_name}_CashFlow" + cfs.append(name) + if cashflow.get_depreciation() is not None: + cfs.append(f"{comp_name}_{cf_name}_depreciation") + cfs.append(f"{comp_name}_{cf_name}_depreciation_tax_credit") + return cfs diff --git a/templates/outer.xml b/templates/outer.xml deleted file mode 100644 index 4f877299..00000000 --- a/templates/outer.xml +++ /dev/null @@ -1,181 +0,0 @@ - - - - . - - 1 - - - - - inner_workflow - heron_lib - raven - grid - grid - sweep - - - inner_workflow - heron_lib - raven - cap_opt - opt_eval - opt_soln - opt_soln - - - opt_soln - opt_path - opt_soln - - - - - - - - - - - - - - GRO_capacities - GRO_outer_results - - - GRO_capacities - GRO_outer_results - - - trajID - iteration, accepted, GRO_capacities, GRO_outer_results - - - - - - ~/projects/raven/raven_framework - disp_results - - - - Samplers|MonteCarlo@name:mc_arma_dispatch|constant@name:denoises - - - - mean_NPV - 1e-8 - 5 - True - Custom - (Constant*Matern) - True - False - - - - - - - - - 1 - - - - - - - - - mean_NPV - 1 - opt_eval - - 800 - every - max - - - - - - - 2 - 1.5 - 0.2 - - - - - - - 1 - 1e-4 - 1e-8 - - - - mean_NPV - 1 - opt_eval - - 100 - max - every - - LHS_samp - gpROM - - 1 - Internal - - - 1e-5 - 4 - - - - differentialEvolution - 30 - - - differentialEvolution - 30 - 1 - 20 - Constant - - - differentialEvolution - 30 - 0.98 - Constant - - - - - - - inner.xml - ../heron.lib - - - - - csv - grid - - - csv - opt_soln - trajID - - - opt_soln - GRO_capacities, mean_NPV - - - diff --git a/templates/raven_template.py b/templates/raven_template.py new file mode 100644 index 00000000..46a57d09 --- /dev/null +++ b/templates/raven_template.py @@ -0,0 +1,854 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + RAVEN workflow templates + + @author: Jacob Bryan (@j-bryan) + @date: 2024-10-29 +""" +import os +import re +import glob +from pathlib import Path +import itertools as it +import xml.etree.ElementTree as ET + +from .imports import xmlUtils, Template +from .heron_types import HeronCase, Component, Source, ValuedParam +from .naming_utils import get_result_stats, get_component_activity_vars, get_opt_objective, get_statistics, Statistic +from .xml_utils import add_node_to_tree, stringify_node_values + +from .snippets.base import RavenSnippet +from .snippets.runinfo import RunInfo +from .snippets.steps import Step, IOStep +from .snippets.samplers import Sampler, SampledVariable, Grid, Stratified, CustomSampler, EnsembleForward +from .snippets.optimizers import BayesianOptimizer, GradientDescent +from .snippets.models import GaussianProcessRegressor, PickledROM, EnsembleModel +from .snippets.distributions import Distribution, Uniform +from .snippets.outstreams import PrintOutStream +from .snippets.dataobjects import DataObject, PointSet, DataSet +from .snippets.variablegroups import VariableGroup +from .snippets.files import File +from .snippets.factory import factory as snippet_factory + + +# NOTE: Leave this here! Moving this to xml_utils.py will cause a circular import problem with snippets.factory.py +def parse_to_snippets(node: ET.Element) -> ET.Element: + """ + Builds an XML tree that looks exactly like node but with RavenSnippet objects where defined. + @ In, node, ET.Element, the node to parse + @ Out, parsed: ET.Element, the parsed XML node + """ + # Base case: The node matches a registered RavenSnippet class. RavenSnippets know how to represent + # their entire contiguous block of XML, so no further recursion is necessary once a valid RavenSnippet + # is found. + if snippet_factory.has_registered_class(node): + snippet = snippet_factory.from_xml(node) + return snippet + + # If the node doesn't match a registered RavenSnippet class, copy over the node to the + parsed = ET.Element(node.tag, node.attrib) + parsed.text = node.text + parsed.tail = node.tail + + # Recurse over node children (if any) + for child in node: + parsed_child = parse_to_snippets(child) + parsed.append(parsed_child) + + return parsed + +class RavenTemplate(Template): + """ Template class for RAVEN workflows """ + # Default stats abbreviations. Different run modes have different defaults + DEFAULT_STATS_NAMES = { + "opt": ["expectedValue", "sigma", "median"], + "sweep": ["maximum", "minimum", "percentile", "samples", "variance"] + } + + # Prefixes for financial metrics only + FINANCIAL_PREFIXES = ["sharpe", "sortino", "es", "VaR", "glr"] + FINANCIAL_STATS_NAMES = ["sharpeRatio", "sortinoRatio", "expectedShortfall", "valueAtRisk", "gainLossRatio"] + + def __init__(self) -> None: + """ + Constructor + @ In, None + @ Out, None + """ + super().__init__() + # Naming templates + self.addNamingTemplates({"jobname" : "{case}_{io}", + "stepname" : "{action}_{subject}", + "variable" : "{unit}_{feature}", + "dispatch" : "Dispatch__{component}__{tracker}__{resource}", + "tot_activity" : "TotalActivity__{component}__{tracker}__{resource}", + "data object" : "{source}_{contents}", + "distribution" : "{variable}_dist", + "ARMA sampler" : "{rom}_sampler", + "lib file" : "heron.lib", # TODO use case name? + "cashfname" : "_{component}{cashname}", + "re_cash" : "_rec_{period}_{driverType}{driverName}", + "cluster_index" : "_ROM_Cluster", + "metric_name" : "{stats}_{econ}", + "statistic" : "{prefix}_{name}" + }) + self._template = None + + ######################## + # PUBLIC API FUNCTIONS # + ######################## + + def loadTemplate(self, filename: str, path: str) -> None: + """ + Loads template file statefully. + @ In, filename, str, name of file to load (xml) + @ In, path, str, path to file relative to HERON/templates/ + @ Out, None + """ + super().loadTemplate(filename, path) + self._template = parse_to_snippets(self._template) + + def createWorkflow(self, **kwargs) -> None: + """ + Create a workflow for the specified Case and its components and sources + @ In, kwargs, dict, keyword arguments + @ Out, None + """ + # Universal workflow settings + self._set_verbosity(kwargs["case"].get_verbosity()) + + def writeWorkflow(self, template: ET.Element, destination: str, run: bool = False) -> None: + """ + Writes a template to file. + @ In, template, xml.etree.ElementTree.Element, file to write + @ In, destination, str, path and filename to write to + @ In, run, bool, optional, if True then run the workflow after writing? good idea? + @ Out, errors, int, 0 if successfully wrote [and run] and nonzero if there was a problem + """ + # Ensure all node attribute values and text are expressed as strings. Errors are thrown if any of these aren't + # strings. Enforcing this here allows flexibility with how node values are stored and manipulated before write + # time, such as storing values as lists or numeric types. For example, text fields which are a comma-separated + # list of values can be stored in the RavenSnippet object as a list, and new items can be inserted into that + # list as needed, then the list can be converted to a string only now at write time. + stringify_node_values(template) + + # Remove any unused top-level nodes (Models, Samplers, etc.) to keep things looking clean + for node in template: + if len(node) == 0: + template.remove(node) + + super().writeWorkflow(template, destination, run) + print(f"Wrote '{self.write_name}' to '{destination}'") + + @property + def template_xml(self) -> ET.Element: + """ + Getter property for the template XML ET.Element tree + @ In, None + @ Out, _template, ET.Element, the XML tree + """ + return self._template + + def get_write_path(self, dest_dir: str) -> str: + """ + Get the path of to write the template to + @ In, dest_dir, str, the directory to write the file to + @ Out, path, str, the path (directory + file name) to write to + """ + write_name = getattr(self, "write_name", None) + if not write_name: + raise ValueError(f"Template class {self.__class__.__name__} object has no 'write_name' attribute.") + path = os.path.join(dest_dir, write_name) + return path + + ##################### + # SNIPPET UTILITIES # + ##################### + def _add_snippet(self, snippet: RavenSnippet, parent: str | ET.Element | None = None) -> None: + """ + Add an XML snippet to the template XML + @ In, snippet, RavenSnippet, the XML snippet to add + @ In, parent, str | ET.Element | None, the parent node to add the snippet + @ Out, None + """ + if isinstance(snippet, ET.Element) and not isinstance(snippet, RavenSnippet): + raise TypeError(f"The XML block to be added is not a RavenSnippet object. Received type: {type(snippet)}. " + "Perhaps something went wrong when parsing the template XML, and the correct RavenSnippet " + "subclass wasn't found?") + if snippet is None: + raise ValueError("Received None instead of a RavenSnippet object. Perhaps something went wrong when finding " + "an XML node?") + + # If a parent node was provided, just append the snippet to its parent node. + if isinstance(parent, ET.Element): + parent.append(snippet) + return + + # Otherwise, figure out where to put the XML snippet. Either a string for a parent node (maybe doesn't exist yet) + # was provided, or the desired location is inferred from the snippet class (e.g. Models, DataObjects, etc.). + if parent and isinstance(parent, str): + parent_path = parent + else: + # Find parent node based on snippet "class" attribute + parent_path = snippet.snippet_class + + if parent_path is None: + raise ValueError(f"The path to a parent node for node {snippet} could not be determined!") + + # Make the parent node if it doesn't exist. This is helpful if it's unknown if top-level nodes (Models, Optimizers, + # Steps, etc.) exist without having to add a check everywhere a snippet needs to get added. + add_node_to_tree(snippet, parent_path, self._template) + + ############################## + # FEATURE BUILDING UTILITIES # + ############################## + # These functions help set options and build workflow features. They are roughly organized by which portion of the + # RAVEN template they modify. + + # Global attributes + def _set_verbosity(self, verbosity: str) -> None: + """ + Sets the verbosity attribute of the root Simulation node + @ In, verbosity, str, the verbosity level + @ Out, None + """ + self._template.set("verbosity", verbosity) + + def _set_case_name(self, name: str) -> None: + """ + Sets the JobName and WorkingDir values in the RunInfo block to the given name + @ In, name, str, case name to use + @ Out, None + """ + run_info = self._template.find("RunInfo") # type: RunInfo + run_info.job_name = name + run_info.working_dir = name + + def _add_step_to_sequence(self, step: Step, index: int | None = None) -> None: + """ + Add a step to the Sequence node + @ In, step, Step, the step to add + @ In, index, int, optional, the index to add the step at + @ Out, None + """ + run_info = self._template.find("RunInfo") # type: RunInfo + idx = index if index is not None else len(run_info.sequence) + run_info.sequence.insert(idx, step) + + # TODO refactor to fit snippet style + # from PR #397 + def _get_parallel_xml(self, hostname): + """ + Finds the xml file to go with the given hostname. + @ In, hostname, string with the hostname to search for + @ Out, xml, xml.eTree.ElementTree or None, if an xml file is found then use it, otherwise return None + """ + # Should this allow loading from another directory (such as one + # next to the input file?) + path = os.path.join(os.path.dirname(__file__),"parallel","*.xml") + filenames = glob.glob(path) + for filename in filenames: + cur_xml = ET.parse(filename).getroot() + regexp = cur_xml.attrib['hostregexp'] + if re.match(regexp, hostname): + return cur_xml + return None + + # Steps + def _load_file_to_object(self, source: Source, target: RavenSnippet) -> IOStep: + """ + Load a source file to a target object + @ In, source, Source, the source to load + @ In, target, RavenSnippet, the object to load to + @ Out, step, IOStep, the step used to do the loading + """ + # Get the file to load. Might already exist in the template XML + file = self._template.find("Files/Input[@name='{source.name}']") # type: File + if file is None: + file = File(source.name) + file.path = source._target_file + + # Create an IOStep to load the file to the target + step_name = self.namingTemplates["stepname"].format(action="read", subject=source.name) + step = IOStep(step_name) + step.append(file.to_assembler_node("Input")) + step.append(target.to_assembler_node("Output")) + + self._add_snippet(file) + self._add_snippet(step) + + return step + + @staticmethod + def _load_pickled_rom(source: Source) -> tuple[File, PickledROM, IOStep]: + """ + Loads a pickled ROM + @ In, source, Source, the ROM source + @ Out, file, File, a Files/Input snippet pointing to the ROM file + @ Out, rom, PickledROM, the model snippet + @ Out, step, IOStep, a step to load the model + """ + # Create the Files/Input node for the ROM source file + file = File(source.name) + file.path = source._target_file + + # Create ROM snippet + rom = PickledROM(source.name) + if source.needs_multiyear is not None: + rom.add_subelements({"Multicycle" : {"cycles": source.needs_multiyear}}) + if source.limit_interp is not None: + rom.add_subelements(maxCycles=source.limit_interp) + if source.eval_mode == 'clustered': + ET.SubElement(rom, "clusterEvalMode").text = "clustered" + + # Create an IOStep to load the ROM from the file + step = IOStep(f"read_{source.name}") + step.append(file.to_assembler_node("Input")) + step.append(rom.to_assembler_node("Output")) + + return file, rom, step + + @staticmethod + def _print_rom_meta(rom: RavenSnippet) -> tuple[DataSet, PrintOutStream, IOStep]: + """ + Print the metadata for a ROM, making the DataSet, Print OutStream, and IOStep to accomplish this. + @ In, rom, RavenSnippet, the ROM to print + @ Out, dataset, DataSet, the ROM metadata data object + @ Out, outstream, PrintOutStream, the outstream to print the data object to file + @ Out, step, IOStep, the step to print the ROM meta + """ + if rom.snippet_class != "Models": + raise ValueError("The RavenSnippet class provided is not a Model!") + + # Create the output data object + dataset = DataSet(f"{rom.name}_meta") + + # Create the outstream for the dataset + outstream = PrintOutStream(dataset.name) + outstream.source = dataset + + # create step + step = IOStep(f"print_{dataset.name}") + step.append(rom.to_assembler_node("Input")) + step.append(dataset.to_assembler_node("Output")) + step.append(outstream.to_assembler_node("Output")) + + return dataset, outstream, step + + # VariableGroups + def _create_case_labels_vargroup(self, labels: dict[str, str], name: str = "GRO_case_labels") -> VariableGroup: + """ + Create a variable group for case labels + @ In, labels, dict[str, str], the case labels + @ In, name, str, optional, the name of the group + @ Out, group, VariableGroup, the case labels variable group + """ + group = VariableGroup(name) + group.variables.extend(map(lambda label: f"{label}_label", labels.keys())) + return group + + def _get_statistical_results_vars(self, case: HeronCase, components: list[Component]) -> list[str]: + """ + Collects result metric names for statistical metrics. Should only be used with templates which have multiple + time series samples. + @ In, case, Case, HERON case + @ In, components, list[Component], HERON components + @ Out, var_names, list[str], list of variable names + """ + # Add statistics for economic metrics to variable group. Use all statistics. + default_names = self.DEFAULT_STATS_NAMES.get(case.get_mode(), []) + # This gets the unique values from default_names and the case result statistics dict keys. Set operations + # look cleaner but result in a randomly ordered list. Having a consistent ordering of statistics is beneficial + # from a UX standpoint. + stats_names = list(dict.fromkeys(default_names + list(case.get_result_statistics()))) + econ_metrics = case.get_econ_metrics(nametype="output") + stats_var_names = get_result_stats(econ_metrics, stats_names, case) + + # Add total activity statistics for variable group. Use only non-financial statistics. + non_fin_stat_names = [name for name in stats_names if name not in self.FINANCIAL_STATS_NAMES] + tot_activity_metrics = get_component_activity_vars(components, self.namingTemplates["tot_activity"]) + activity_var_names = get_result_stats(tot_activity_metrics, non_fin_stat_names, case) + + var_names = stats_var_names + activity_var_names + + # The optimization objective might not have made it into the list. Make sure it's there. + if case.get_mode() == "opt" and (objective := get_opt_objective(case)) not in var_names: + var_names.insert(0, objective) + + return var_names + + def _get_deterministic_results_vars(self, case: HeronCase, components: list[Component]) -> list[str]: + """ + Collects result metric names for deterministic cases + @ In, case, Case, HERON case + @ In, components, list[Component], HERON components + @ Out, var_names, list[str], list of variable names + """ + econ_metrics = case.get_econ_metrics(nametype="output") + tot_activity_metrics = get_component_activity_vars(components, self.namingTemplates["tot_activity"]) + var_names = econ_metrics + tot_activity_metrics + return var_names + + def _get_activity_metrics(self, components: list[Component]) -> list[str]: + """ + Gets the names of component activity metrics + @ In, components, list[Component], HERON components + @ Out, act_metrics, list[str], component activity metric names + """ + act_metrics = [] + for component in components: + for tracker in component.get_tracking_vars(): + resource_list = sorted(list(component.get_resources())) + for resource in resource_list: + # NOTE: Assumes the only activity metric we care about is total activity + default_stats_tot_activity = self.namingTemplates["tot_activity"].format(component=component.name, + tracker=tracker, + resource=resource) + act_metrics.append(default_stats_tot_activity) + return act_metrics + + # Models + def _add_time_series_roms(self, ensemble_model: EnsembleModel, case: HeronCase, sources: list[Source]) -> None: + """ + Create and modify snippets based on sources + @ In, case, Case, HERON case + @ In, sources, list[Source], case sources + @ Out, None + """ + dispatch_eval = self._template.find("DataObjects/DataSet[@name='dispatch_eval']") # type: DataSet + + # Gather any ARMA sources from the list of sources + arma_sources = [s for s in sources if s.is_type("ARMA")] + + # Add cluster index info to dispatch variable groups and data objects + if any(source.eval_mode == "clustered" for source in arma_sources): + vg_dispatch = self._template.find("VariableGroups/Group[@name='GRO_dispatch']") # type: VariableGroup + vg_dispatch.variables.append(self.namingTemplates["cluster_index"]) + dispatch_eval.add_index(self.namingTemplates["cluster_index"], "GRO_dispatch_in_Time") + + # Add models, steps, and their requisite data objects and outstreams for each case source + for source in arma_sources: + # An ARMA source is a pickled ROM that needs to be loaded. + # Load the ROM from file + source_file, pickled_rom, load_iostep = self._load_pickled_rom(source) + self._add_snippet(source_file) + self._add_snippet(pickled_rom) + self._add_snippet(load_iostep) + self._add_step_to_sequence(load_iostep, index=0) + + # Print the pickled ROM metadata + meta_dataset, meta_outstream, meta_iostep = self._print_rom_meta(pickled_rom) + self._add_snippet(meta_dataset) + self._add_snippet(meta_outstream) + self._add_snippet(meta_iostep) + self._add_step_to_sequence(meta_iostep, index=1) + + # Add loaded ROM to the EnsembleModel + inp_name = self.namingTemplates["data object"].format(source=source.name, contents="placeholder") + inp_do = PointSet(inp_name) + inp_do.inputs.append("scaling") + self._add_snippet(inp_do) + + eval_name = self.namingTemplates["data object"].format(source=source.name, contents="samples") + eval_do = DataSet(eval_name) + eval_do.inputs.append("scaling") + out_vars = source.get_variable() + eval_do.outputs.extend(out_vars) + eval_do.add_index(case.get_time_name(), out_vars) + eval_do.add_index(case.get_year_name(), out_vars) + if source.eval_mode == "clustered": + eval_do.add_index(self.namingTemplates["cluster_index"], out_vars) + self._add_snippet(eval_do) + + rom_assemb = pickled_rom.to_assembler_node("Model") + rom_assemb.append(inp_do.to_assembler_node("Input")) + rom_assemb.append(eval_do.to_assembler_node("TargetEvaluation")) + ensemble_model.append(rom_assemb) + + # update variable group with ROM output variable names + self._template.find("VariableGroups/Group[@name='GRO_dispatch_in_Time']").variables.extend(out_vars) + + def _get_stats_for_econ_postprocessor(self, + case: HeronCase, + econ_vars: list[str], + activity_vars: list[str]) -> list[tuple[Statistic, str]]: + """ + Get pairs of Statistic objects and metric/variable names to which to apply that statistic + @ In, case, HeronCase, the HERON case + @ In, econ_vars, list[str], economic metric names + @ In, activity_vars, list[str], activity variable names + @ Out, stats_to_add, list[tuple[Statistic, str]], statistics and the variables they act on + """ + # Econ metrics with all statistics names + # NOTE: This logic is borrowed from ravenTemplate._get_statistical_results_vars, but it's more useful to have the + # names, prefixes, and variable name separate here, not as one big string. Otherwise, we have to try to break that + # string back up, which would be sensitive to metric and variable naming conventions. We duplicate a little of the + # logic but get something more robust in return. + default_names = self.DEFAULT_STATS_NAMES.get(case.get_mode(), []) + stats_names = list(dict.fromkeys(default_names + list(case.get_result_statistics()))) + econ_stats = get_statistics(stats_names, case.stats_metrics_meta) + # Activity metrics with non-financial statistics + non_fin_stat_names = [name for name in stats_names if name not in self.FINANCIAL_STATS_NAMES] + activity_stats = get_statistics(non_fin_stat_names, case.stats_metrics_meta) + + # Collect the statistics to add to the postprocessor + stats_to_add = list(it.chain( + it.product(econ_stats, econ_vars), + it.product(activity_stats, activity_vars) + )) + + # The metric needed for the objective function might not have been added yet. + if case.get_mode() == "opt": + opt_settings = case.get_optimization_settings() + try: + statistic = opt_settings["stats_metric"]["name"] + except (KeyError, TypeError): + statistic = "expectedValue" # default to expectedValue + opt_stat = get_statistics([statistic], case.stats_metrics_meta)[0] + target_var, _ = case.get_opt_metric() + target_var_output_name = case.economic_metrics_meta[target_var]["output_name"] + if (opt_stat, target_var_output_name) not in stats_to_add: + stats_to_add.append((opt_stat, target_var_output_name)) + + return stats_to_add + + # Distributions and SampledVariables + def _create_new_sampled_capacity(self, var_name: str, capacities: list[float]) -> SampledVariable: + """ + Creates a uniform distribution and SampledVariable object for a given list of capacities + @ In, var_name, str, name of the variable + @ In, capacities, list[float], list of capacity values + @ Out, sampled_var, SampledVariable, variable to be sampled + """ + dist_name = self.namingTemplates["distribution"].format(variable=var_name) + dist = Uniform(dist_name) + min_cap = min(capacities) + max_cap = max(capacities) + dist.lower_bound = min_cap + dist.upper_bound = max_cap + self._add_snippet(dist) + + sampled_var = SampledVariable(var_name) + sampled_var.distribution = dist + + return sampled_var + + def _get_uncertain_cashflow_params(self, + components: list[Component]) -> tuple[list[SampledVariable], list[Distribution]]: + """ + Create SampledVariable and Distribution snippets for all uncertain cashflow parameters. + @ In, components, list[Component] + @ Out, sampled_vars, list[SampledVariable], objects to link sampler variables to distributions + @ Out, distributions, list[Distribution], distribution snippets + """ + sampled_vars = [] + distributions = [] + + # For each component, cashflow, and cashflow equation parameter, find any which are uncertain, and create + # distribution and sampled variable objects. + for component in components: + for cashflow in component.get_cashflows(): + for param_name, vp in cashflow.get_uncertain_params().items(): + unit_name = f"{component.name}_{cashflow.name}" + feat_name = self.namingTemplates["variable"].format(unit=unit_name, feature=param_name) + dist_name = self.namingTemplates["distribution"].format(variable=feat_name) + + # Reconstruct distribution XML node from valuedParam definition + dist_node = vp._vp.get_distribution() # type: ET.Element + dist_node.set("name", dist_name) + dist_snippet = snippet_factory.from_xml(dist_node) + distributions.append(dist_snippet) + + # Create sampled variable snippet + sampler_var = SampledVariable(feat_name) + sampler_var.distribution = dist_snippet + sampled_vars.append(sampler_var) + + return sampled_vars, distributions + + # Samplers + def _create_sampler_variables(self, + case: HeronCase, + components: list[Component]) -> tuple[dict[SampledVariable, list[float]], + dict[str, ValuedParam]]: + """ + Create the Distribution and SampledVariable objects and the list of constant capacities that need to + be added to samplers and optimizers. + @ In, case, Case, HERON case + @ In, components, list[Component], HERON components + @ Out, sampled_variables, dict[SampledVariable, list[float]], variable objects for the sampler/optimizer + @ Out, constants, dict[str, float], constant variables + """ + sampled_variables = {} + constants = {} + + # Make Distribution and SampledVariable objects for sampling dispatch variables + for key, value in case.dispatch_vars.items(): + var_name = self.namingTemplates["variable"].format(unit=key, feature="dispatch") + vals = value.get_value(debug=case.debug["enabled"]) + if isinstance(vals, list): + sampled_var = self._create_new_sampled_capacity(var_name, vals) + sampled_variables[sampled_var] = vals + + # Make Distribution and SampledVariable objects for capacity variables. Capacities with non-parametric + # ValuedParams are fixed values and are added instead as constants. + for component in components: + interaction = component.get_interaction() + name = component.name + var_name = self.namingTemplates["variable"].format(unit=name, feature="capacity") + cap = interaction.get_capacity(None, raw=True) # type: ValuedParam + + if not cap.is_parametric(): # we already know the value + continue + + vals = cap.get_value(debug=case.debug["enabled"]) + if isinstance(vals, list): # multiple values meaning either opt bounds or sweep values + sampled_var = self._create_new_sampled_capacity(var_name, vals) + sampled_variables[sampled_var] = vals + else: # just one value meaning it's a constant + constants[var_name] = vals + + return sampled_variables, constants + + def _add_labels_to_sampler(self, sampler: Sampler, labels: dict[str, str]) -> None: + """ + Add case labels as constants for a sampler or optimizer + @ In, sampler, Sampler, sampler to add labels to + @ In, case, Case, HERON case + @ Out, None + """ + for key, value in labels.items(): + var_name = self.namingTemplates["variable"].format(unit=key, feature="label") + sampler.add_constant(var_name, value) + + def _configure_static_history_sampler(self, + custom_sampler: CustomSampler, + case: HeronCase, + sources: list[Source], + scaling: int | None = 1.0) -> None: + """ + Configures a sampler and relevant data objects and variable groups for using static histories in the workflow + @ In, custom_sampler, CustomSampler, the sampler to use to sample the static histories + @ In, case, HeronCase, the HERON case, + @ In, sources, list[Source], the case sources + @ In, scaling, int, optional, the scaling constant for the custom_sampler + @ Out, None + """ + indices = [case.get_year_name(), case.get_time_name()] + cluster_index = self.namingTemplates["cluster_index"] + if case.debug["enabled"]: + indices.append(cluster_index) + + time_series_vargroup = self._template.find("VariableGroups/Group[@name='GRO_timeseries']") # type: VariableGroup + + for source in filter(lambda x: x.is_type("CSV"), sources): + # Add the source variables to the GRO_timeseries_in variable group + source_vars = source.get_variable() + self._template.find("VariableGroups/Group[@name='GRO_timeseries']").variables.extend(source_vars) + + # Create a new that will store the csv data + csv_dataset = DataSet(source.name) + csv_dataset.inputs.extend([case.get_time_name(), case.get_year_name()]) + csv_dataset.outputs.extend(source_vars) + for index in indices: + csv_dataset.add_index(index, source_vars) + self._add_snippet(csv_dataset) + + # Use an IOStep to load the CSV data into the DataSet + read_static = self._load_file_to_object(source, csv_dataset) + self._add_step_to_sequence(read_static, index=0) + + # Add variables to the custom sampler for the + custom_sampler.append(csv_dataset.to_assembler_node("Source")) + for var in it.filterfalse(custom_sampler.has_variable, it.chain(indices, source_vars)): + # NOTE: Being careful not to add duplicate time index variables to the custom sampler in case somebody tries + # to include multiple CSV sources. + custom_sampler.add_variable(SampledVariable(var)) + + # Add the static history variables to the dispatch model + new_vars = it.chain( + source_vars, + filter(lambda x: x not in time_series_vargroup.variables, indices) + ) + time_series_vargroup.variables.extend(new_vars) + + if custom_sampler.find("constant[@name='scaling']") is None and scaling is not None: + custom_sampler.add_constant("scaling", scaling) + + def _create_grid_sampler(self, + case: HeronCase, + components: list[Component], + capacity_vars: VariableGroup | str | list[str], + results_vars:VariableGroup | str | list[str]) -> tuple[Grid, PointSet]: + """ + Creates a grid sampler for sweep cases + @ In, case, the HERON case + @ In, components, list[Component], the case components + @ In, capacity_vars, VariableGroup | str | list[str], the capacity variable names + @ In, results_vars, VariableGroup | str | list[str], the result stat/metric names + @ Out, sampler, Grid, the grid sampler + @ Out, results_data, PointSet, a data object to hold the results data at each grid point + """ + # Define a PointSet for the results variables at each grid point + results_data = PointSet("grid") + + if isinstance(capacity_vars, list): + results_data.inputs.extend(capacity_vars) + else: + results_data.inputs.append(capacity_vars) + + if isinstance(results_vars, list): + results_data.outputs.extend(results_vars) + else: + results_data.outputs.append(results_vars) + + self._add_snippet(results_data) + + # Define grid sampler and build the variables and their distributions that it'll sample + sampler = Grid("grid") + variables, consts = self._create_sampler_variables(case, components) + for sampled_var, vals in variables.items(): + sampler.add_variable(sampled_var) + sampled_var.use_grid(construction="custom", kind="value", values=sorted(vals)) + for var_name, val in consts.items(): + sampler.add_constant(var_name, val) + + # Number of "denoises" for the sampler is the number of samples it should take + sampler.denoises = case.get_num_samples() + + # If there are any case labels, make a variable group for those and add it to the "grid" PointSet. + # These labels also need to get added to the sampler as constants. + labels = case.get_labels() + if labels: + vargroup = self._create_case_labels_vargroup(labels) + self._add_snippet(vargroup) + results_data.outputs.append(vargroup.name) + self._add_labels_to_sampler(sampler, labels) + + return sampler, results_data + + def _create_ensemble_forward_sampler(self, samplers: list[Sampler], name: str | None = None) -> EnsembleForward: + """ + Wraps a list of samplers in an EnsembleForward sampler + @ In, samplers, list[Sampler], the samplers to include in the ensemble + @ In, name, str, optional, the name of the ensemble sampler + @ Out, ensemble_sampler, EnsembleForward, the ensemble sampler; default: "ensemble_sampler" + """ + ensemble_sampler = EnsembleForward(name or "ensemble_sampler") + for sampler in samplers: + ensemble_sampler.append(sampler) + return ensemble_sampler + + # Optimizers + def _create_bayesian_opt(self, case: HeronCase, components: list[Component]) -> BayesianOptimizer: + """ + Set up the Bayesian optimization optimizer, LHS sampler, GPR model, and necessary distributions and data objects + for using Bayesian optimization. + @ In, case, HeronCase, the HERON case + @ In, components, list[Component], the case components + @ Out, optimizer, BayesianOptimizer, the Bayesian optimization node + """ + # Create major XML blocks + optimizer = BayesianOptimizer("cap_opt") + sampler = Stratified("LHS_samp") + gpr = GaussianProcessRegressor("gpROM") + + # Add blocks to XML template + self._add_snippet(optimizer) + self._add_snippet(sampler) + self._add_snippet(gpr) + + # Connect optimizer to sampler and ROM components + optimizer.set_sampler(sampler) + optimizer.set_rom(gpr) + + # Apply any specified optimization settings + opt_settings = case.get_optimization_settings() or {} # default to empty dict if None + optimizer.set_opt_settings(opt_settings) + # Set GPR kernel if provided + if opt_settings and (custom_kernel := opt_settings["algorithm"]["BayesianOpt"].get("kernel", None)): + gpr.custom_kernel = custom_kernel + + # Create sampler variables and their respective distributions + variables, consts = self._create_sampler_variables(case, components) + for sampled_var in variables: + sampled_var.use_grid(construction="equal", kind="CDF", steps=4, values=[0, 1]) + optimizer.add_variable(sampled_var) + sampler.add_variable(sampled_var) + for var_name, val in consts.items(): + optimizer.add_constant(var_name, val) + + # Set number of denoises + optimizer.denoises = case.get_num_samples() + + # Set GPR features list and target + for component in components: + name = component.name + interaction = component.get_interaction() + cap = interaction.get_capacity(None, raw=True) + if cap.is_parametric() and isinstance(cap.get_value(debug=case.debug["enabled"]) , list): + gpr.features.append(self.namingTemplates["variable"].format(unit=name, feature="capacity")) + gpr.target.append(get_opt_objective(case)) + + return optimizer + + def _create_gradient_descent(self, case: HeronCase, components: list[Component]) -> GradientDescent: + """ + Set up the gradient descent optimizer + @ In, case, HeronCase, the HERON case + @ In, components, list[Component], the case components + @ Out, optimizer, GradientDescent, the gradient descent optimizer node + """ + # Create necessary XML blocks + optimizer = GradientDescent("cap_opt") + self._add_snippet(optimizer) + + # Apply any specified optimization settings + opt_settings = case.get_optimization_settings() + optimizer.set_opt_settings(opt_settings) + optimizer.objective = get_opt_objective(case) + + # Set number of denoises + optimizer.denoises = case.get_num_samples() + + # Create sampler variables and their respective distributions + variables, consts = self._create_sampler_variables(case, components) + + for sampled_var, vals in variables.items(): + # initial value + min_val = min(vals) + max_val = max(vals) + delta = max_val - min_val + # start 5% away from zero + initial = min_val + 0.05 * delta if max_val > 0 else max_val - 0.05 * delta + sampled_var.initial = initial + optimizer.add_variable(sampled_var) + + for var_name, val in consts.items(): + optimizer.add_constant(var_name, val) + + return optimizer + + # Files + def _get_function_files(self, sources: list[Source]) -> list[File]: + """ + Get the File object for each source that is a Function + @ In, sources, list[Source], the sources + @ Out, files, list[File], the files + """ + # Add Function sources as Files + files = [] + for function in [s for s in sources if s.is_type("Function")]: + file = self._template.find(f"Files/Input[@name='{function.name}']") + if file is None: # Add function to if not found there + file = File(function.name) + path = Path(function._source) + # magic variable name that will get resolved later are like %VARNAME%/some/path + if not str(path).startswith("%"): + path = ".." / path + file.path = path + # file.path = Path(function._source) + self._add_snippet(file) + files.append(file) + return files diff --git a/templates/snippets/__init__.py b/templates/snippets/__init__.py new file mode 100644 index 00000000..03dea6c5 --- /dev/null +++ b/templates/snippets/__init__.py @@ -0,0 +1,17 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED + +from .base import RavenSnippet +from .databases import HDF5, NetCDF +from .dataobjects import DataSet, HistorySet, PointSet +from .distributions import Distribution +from .files import File +from .models import EconomicRatioPostProcessor, EnsembleModel, ExternalModel, GaussianProcessRegressor, HeronDispatchModel, PickledROM, RavenCode +from .optimizers import BayesianOptimizer, GradientDescent +from .outstreams import HeronDispatchPlot, OptPathPlot, PrintOutStream, TealCashFlowPlot +from .runinfo import RunInfo +from .samplers import CustomSampler, Grid, MonteCarlo, SampledVariable, Sampler, Stratified, EnsembleForward +from .steps import IOStep, MultiRun, PostProcess +from .variablegroups import VariableGroup + +from .factory import factory diff --git a/templates/snippets/base.py b/templates/snippets/base.py new file mode 100644 index 00000000..e5df8513 --- /dev/null +++ b/templates/snippets/base.py @@ -0,0 +1,153 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Base RavenSnippet class + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +from typing import Any +import xml.etree.ElementTree as ET + +from ..xml_utils import merge_trees + +class RavenSnippet(ET.Element): + """ + RavenSnippet class objects describe one contiguous snippet of RAVEN XML, inheriting from the + xml.etree.ElementTree.Element class. This base class contains methods for quickly building subtrees + and set and access common RAVEN node attributes. + """ + tag = None # XML tag associated with the snippet class + snippet_class = None # class of Entity described by the snippet (e.g. Models, Optimizers, DataObjects, etc.) + subtype = None # subtype of the snippet entity, does not need to be defined for all snippets + + @classmethod + def from_xml(cls, node: ET.Element, **kwargs) -> "RavenSnippet": + """ + Alternate constructor which instantiates a new RavenSnippet object from an existing XML node + @ In, node, ET.Element, the template node + @ In, kwargs, dict, keyword arguments + @ Out, snippet, RavenSnippet, the new snippet + """ + # Instantiate the snippet class and copy over attribs and text + snippet = cls() + snippet.attrib |= node.attrib + snippet.text = node.text + + # Merge the node's subtree into the snippet's subtree. Whether or not to overwrite equal nodes (overwrite=True), + # include both nodes (overwrite=False), and whether or not matching nodes must have matching attributes + # (match_attrib) can be contolled via keyword arguments. + merge_kwargs = {k: kwargs[k] for k in kwargs.keys() & {"overwrite", "match_attrib", "match_text"}} + snippet = merge_trees(snippet, node, **merge_kwargs) + + return snippet + + def __init__(self, name: str | None = None, subelements: dict[str, Any] | None = None) -> None: + """ + Constructor + @ In, name, str, optional, the name of the entity + @ In, subelements, dict[str, Any], optional, keyword settings which are added as XML child nodes + @ Out, None + """ + super().__init__(self.tag) + + # Update node attributes with provided values + # Arguments "name", "class_name", and "subtype_name" help to alias the problematic "class" attribute name and + # provide an easy interface to set the common attributes "name" and "subType". + if name is not None: + self.name = name + if self.subtype is not None: + self.attrib["subType"] = self.subtype + + if subelements: + self.add_subelements(subelements) + + def __repr__(self) -> str: + """ + Make a string representation of the snippet. If the "name" attribute is defined, return that. Otherwise, fall back + to the ET.Element implementation. + @ In, None + @ Out, repr, str, the string representation of the object + """ + if name := self.name: + return name + return super().__repr__() + + @property + def name(self) -> str | None: + """ + Name attribute getter + @ In, None + @ Out, name, str | None, the name + """ + return self.get("name", None) + + @name.setter + def name(self, value: str) -> None: + """ + Name attribute setter + @ In, value, str, the name to set + @ Out, None + """ + self.set("name", value) + + def to_assembler_node(self, tag: str) -> ET.Element: + """ + Creates an assembler node from the snippet, if possible. The "class" attribute must be defined. + @ In, tag, str, assembler node tag + """ + if not (self.snippet_class and self.name): + raise ValueError("The RavenSnippet object cannot be expressed as an Assembler node! The object must have " + "'name' and 'class' attributes defined to create an Assembler node. Current values: " + f"class='{self.snippet_class}', name='{self.name}'.") + + node = ET.Element(tag) + node.attrib["class"] = self.snippet_class + node.attrib["type"] = self.tag + node.text = self.name + + return node + + # Subtree building utilities + def add_subelements(self, subelements: dict[str, Any] | None = None, **kwargs) -> None: + """ + Add subelements by either providing a dict or keyword arguments. + @ In, subelements, dict[str, Any], optional, dict with new key-value settings pairs + @ In, kwargs, dict, optional, new settings provided as keyword arguments + @ Out, None + """ + parent = kwargs.pop("parent", self) + all_subs = kwargs | (subelements if subelements else {}) + for tag, value in all_subs.items(): + self._add_subelement(parent, tag, value) + + def _add_subelement(self, parent: ET.Element, tag: str, value: Any) -> None: + """ + Recursively build out subtree. Recurse over dicts, set child node text to string or numeric values, + or form comma separated lists for other iterative data types (list, numpy array, tuple, set, etc.). + @ In, parent, ET.Element, the parent node to append to + @ In, tag, str, the tag of the child node + @ In, value, Any, the value of the child node + """ + # If the value inherits from ET.Element, we can append the value to the parent directly. + if isinstance(value, ET.Element): + parent.append(value) + # If the value happens to be another entity, it has its own to_xml method. Use that instead of manually + # using the tag input to create the child node. + elif isinstance(value, RavenSnippet): + # has a to_xml method + child = value.to_xml() + parent.append(child) + # Otherwise, we'll create the child node ourselves. We handle several possible types of value: + # 1. If value is a dict, create an XML subtree using the dict key-value pairs. + # 2. If the value is iterable but not a string (so a list, numpy array, tuple, set, etc.), create a + # comma separated list of the values and set the node's text to that. + # 3. If the value is anything else (assumes can be cast to a reasonable string), just set the node + # text to that value. + else: + child = ET.SubElement(parent, tag) + if isinstance(value, dict): + for tag, value in value.items(): + self._add_subelement(child, tag, value) + else: + child.text = value diff --git a/templates/snippets/databases.py b/templates/snippets/databases.py new file mode 100644 index 00000000..360deb5c --- /dev/null +++ b/templates/snippets/databases.py @@ -0,0 +1,128 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Database snippets + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +from ..xml_utils import find_node +from ..decorators import listproperty +from .base import RavenSnippet + + +class Database(RavenSnippet): + """ Database snippet base class """ + snippet_class = "Databases" + + @property + def read_mode(self) -> str | None: + """ + Read mode getter + @ In, None + @ Out, read_mode, str | None, the database read mode + """ + return self.get("readMode", None) + + @read_mode.setter + def read_mode(self, value: str) -> None: + """ + Read mode setter + @ In, value, str, the database read mode + @ Out, None + """ + self.set("readMode", value) + + @property + def directory(self) -> str | None: + """ + Directory getter + @ In, None + @ Out, directory, str | None, the database directory + """ + return self.get("directory", None) + + @directory.setter + def directory(self, value: str) -> None: + """ + Directory setter + @ In, value, str, the database directory + @ Out, None + """ + self.set("directory", value) + + @property + def filename(self) -> str | None: + """ + File name getter + @ In, None + @ Out, filename, str | None, the database file name + """ + return self.get("filename", None) + + @filename.setter + def filename(self, value: str) -> None: + """ + File name setter + @ In, value, str, the database file name + @ Out, None + """ + self.set("filename", value) + + @listproperty + def variables(self) -> list[str]: + """ + Database variables getter + @ In, None + @ Out, variables, str | None, the database variables + """ + node = self.find("variables") + return getattr(node, "text", []) + + @variables.setter + def variables(self, value: list[str]) -> None: + """ + Database variables setter + @ In, value, str, the database variables + @ Out, None + """ + find_node(self, "variables").text = value + + ##################### + # Getters & Setters # + ##################### + def add_variable(self, *variables: str) -> None: + """ + Add variables to the database + @ In, *variables, str, variable names + @ Out, None + """ + self.variables.update(variables) + + +class NetCDF(Database): + """ NetCDF database snippet class """ + tag = "NetCDF" + + +class HDF5(Database): + """ HDF5 database snippet class """ + tag = "HDF5" + + @property + def compression(self) -> str | None: + """ + Compression getter + @ In, None + @ Out, compression, str | None, compression method + """ + return self.get("compression", None) + + @compression.setter + def compression(self, value: str) -> None: + """ + Compression setter + @ In, value, str, compression method + @ Out, None + """ + self.set("compression", value) diff --git a/templates/snippets/dataobjects.py b/templates/snippets/dataobjects.py new file mode 100644 index 00000000..a85956b2 --- /dev/null +++ b/templates/snippets/dataobjects.py @@ -0,0 +1,92 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + DataObject snippets + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import xml.etree.ElementTree as ET + +from ..xml_utils import find_node +from ..decorators import listproperty +from .base import RavenSnippet + + +class DataObject(RavenSnippet): + """ DataObject base class """ + snippet_class = "DataObjects" + + @listproperty + def inputs(self) -> list[str]: + """ + Getter for inputs list + @ In, None + @ Out, inputs, list[str], list of input variables + """ + node = self.find("Input") + return getattr(node, "text", []) + + @inputs.setter + def inputs(self, value: list[str]) -> None: + """ + Setter for inputs list + @ In, value, list[str], list of input variables + @ Out, None + """ + find_node(self, "Input").text = value + + @listproperty + def outputs(self) -> list[str]: + """ + Getter for outputs list + @ In, None + @ Out, outputs, list[str], list of output variables + """ + node = self.find("Output") + return getattr(node, "text", []) + + @outputs.setter + def outputs(self, value: list[str]) -> None: + """ + Setter for outputs list + @ In, value, list[str], list of output variables + @ Out, None + """ + find_node(self, "Output").text = value + + +class PointSet(DataObject): + """ PointSet snippet """ + tag = "PointSet" + + +class HistorySet(DataObject): + """ HistorySet snippet """ + tag = "HistorySet" + + +class DataSet(DataObject): + """ DataSet snippet """ + tag = "DataSet" + + def add_index(self, index_var: str, variables: str | list[str]) -> None: + """ + Add an index node to the snippet XML + @ In, index_var, str, name of the index variable + @ In, variables, str | list[str], names of the variable(s) indexed by the index_var + @ Out, None + """ + # Force to be a list + if not isinstance(variables, list): + variables = [str(variables)] + + # There shouldn't be any reason to have duplicate index nodes, so only add the indicated index variable + index_node = self.find(f"Index[@var='{index_var}']") + if index_node is None: + ET.SubElement(self, "Index", {"var": index_var}).text = variables + else: + # If there are any variables provided that aren't in the existing index node's text, add them here. + for var in variables: + if var not in index_node.text: + index_node.text.append(var) diff --git a/templates/snippets/distributions.py b/templates/snippets/distributions.py new file mode 100644 index 00000000..44595942 --- /dev/null +++ b/templates/snippets/distributions.py @@ -0,0 +1,90 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Distribution snippets + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import keyword +import re +import xml.etree.ElementTree as ET + +from ..xml_utils import find_node +from .base import RavenSnippet + + +class Distribution(RavenSnippet): + """ Distribution snippet base class """ + snippet_class = "Distributions" + + +def camel_to_snake(camel: str) -> str: + """ + Converts camelCase to snake_case, handling grouped capital letters + @ In, camel, str, a camel case string + @ Out, snake, str, a snake case string + """ + snake = re.sub(r"([A-Z]+)([A-Z][a-z])", r"\1_\2", camel) # Handle grouped capitals + snake = re.sub(r"([a-z\d])([A-Z])", r"\1_\2", snake) # Handle single capitals + snake = snake.lower() + return snake + + +def distribution_class_from_spec(spec) -> type[Distribution]: + """ + Make a new distribution class from the RAVEN input spec for that class + @ In, spec, RAVEN input spec + @ Out, distribution, NewDistribution, the new distribution class + """ + # It might be tempting to use this node_property function everywhere! I tried it, and here's why I reverted + # to manually defining properties: + # 1. Direct definition of properties in the classes they belong to make it very clear how the property + # is defined. This helps readability and maintainability. + # 2. This function gets very complicated as options are added to it, and in a way that's not easy to read. + # Separate definitions means these options can be handled on a case-by-case basis. + # 3. Linters don't pick up on what properties are available when they're added dynamically. Directly defining + # them with their classes make the developer experience better when using RavenSnippets when building + # templates. + # I've moved the function inside here to limit access to it from elsewhere in the code + def node_property(cls: ET.Element, prop_name: str, node_tag: str | None = None) -> None: + """ + Creates a class property that gets/sets a child node text value + @ In, cls, ET.Element, the ET.Element class or a subclass of it + @ In, prop_name, str, property name + @ In, node_tag, str | None, optional, tag or path of the node the property is tied to (default=prop_name) + @ Out, None + """ + if node_tag is None: + node_tag = prop_name + + def getter(self): + node = self.find(node_tag) + return None if node is None else node.text + + def setter(self, val): + find_node(self, node_tag).text = val + + def deleter(self): + if (node := self.find(node_tag)) is not None: + self.remove(node) + + doc = f"Accessor property for '{node_tag}' node text" + setattr(cls, prop_name, property(getter, setter, deleter, doc=doc)) + + class NewDistribution(Distribution): + """ Dynamically created Distribution class """ + tag = spec.getName() + + # Use the RAVEN input spec for the node to create class properties which set/get subelement values + for subnode in spec.subs: + subnode_tag = subnode.getName() + prop_name = camel_to_snake(subnode_tag) + # can't use name if it's in keywords.kwlist (reserved keywords), so add a trailing underscore to the name + if prop_name in keyword.kwlist: + prop_name += "_" + # Create a property to set the distribution parameters as "distribution.prop_name = value". This lets us keep a + # property-based interface like our other snippet classes. + node_property(NewDistribution, prop_name, subnode_tag) + + return NewDistribution diff --git a/templates/snippets/factory.py b/templates/snippets/factory.py new file mode 100644 index 00000000..462fd4c7 --- /dev/null +++ b/templates/snippets/factory.py @@ -0,0 +1,134 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + A RavenSnippet class factory + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import xml.etree.ElementTree as ET + +from .base import RavenSnippet +from . import distributions + +from ..imports import returnInputParameter + + +def get_all_subclasses(cls: type[RavenSnippet]) -> list[type[RavenSnippet]]: + """ + Recursively collect all of the classes that are a subclass of cls + @ In, cls, type[RavenSnippet], the class to retrieve sub-classes. + @ Out, getAllSubclasses, list[type[RavenSnippet]], list of classes which subclass cls + """ + return cls.__subclasses__() + [g for s in cls.__subclasses__() for g in get_all_subclasses(s)] + + +class SnippetFactory: + """ A factory for RavenSnippet classes, useful for converting XML to RavenSnippet objects """ + def __init__(self) -> None: + """ + Constructor + @ In, None + @ Out, None + """ + self.registered_classes = {} # dict[str, RavenSnippet] + # NOTE: Keys for registered classes are formatted like XPaths. This functionality isn't currently used, + # but could be useful, e.g. for finding any XML nodes which match a certain snippet class. + + def register_snippet_class(self, *cls: type[RavenSnippet]) -> None: + """ + Register a RavenSnippet subclass with the factory + @ In, *cls, type[RavenSnippet], classes to register + @ Out, None + """ + for snip_cls in cls: + # Can't put a snippet into XML if it doesn't have a defined tag. + # This prevents the factory from registering base classes. + if not snip_cls.tag: + continue + + key = self._get_snippet_class_key(snip_cls) + + # What if the key is already in the dict (duplicate name)? + if key in self.registered_classes and (existing_cls := self.registered_classes[key]) != snip_cls: + raise ValueError("A key collision has occurred when registering a RavenSnippet class! " + f"Key: {key}, Class to add: {snip_cls}, Existing class: {existing_cls}") + + self.registered_classes[key] = snip_cls + + def register_all_subclasses(self, cls: type[RavenSnippet]) -> None: + """ + Register all suclasses of a class with the factory + @ In, cls, type[RavenSnippet], base class to register + @ Out, None + """ + self.register_snippet_class(*get_all_subclasses(cls)) + + def from_xml(self, node: ET.Element) -> ET.Element: + """ + Produce a RavenSnippet object of the correct class with identical XML to an existing node + @ In, node, ET.Element, the existing XML node + @ Out, snippet, ET.Element, the matching RavenSnippet object, if one is registered + """ + # Find the registered class which matches the tag and required attributes + key = self._get_node_key(node) + try: + cls = self.registered_classes[key] + except KeyError: + # Not a registered type, so just toss the node back to the caller? + return node + + snippet = cls.from_xml(node) + return snippet + + def has_registered_class(self, node: ET.Element) -> bool: + """ + Does the node have a registered class associated with it? + @ In, node, ET.Element, the node to check + @ Out, is_registered, bool, has a matching registered class + """ + return self._get_node_key(node) in self.registered_classes + + # NOTE: I tried to combine the below methods since they're so similar. However, it gave me more trouble than expected + # because of sometimes having the class type instead of a class object. This makes it so many attributes are not + # accessible since the class hasn't yet been instantiated. Keeping them separate made it easier to avoid errors. + @staticmethod + def _get_snippet_class_key(cls: type[RavenSnippet]) -> str: + """ + Get the registry key for a snippet class + @ In, cls, type[RavenSnippet], the snippet class + @ Out, key, str, the registry key + """ + key = f"{cls.tag}" + # TODO: delineate snippet type by additional attributes? + if cls.subtype is not None: + key += f"[@subType='{cls.subtype}']" + return key + + @staticmethod + def _get_node_key(node: ET.Element) -> str: + """ + Get the registry key for an XML node + @ In, node, ET.Element, the XML node + @ Out, key, str, the registry key + """ + key = node.tag + if subtype := node.get("subType", None): + key += f"[@subType='{subtype}']" + return key + + +# There are many allowable distributions, each of which have their own properties. Rather than manually create classes +# for those, we can read from the RAVEN distributions input specs and dynamically create RavenSnippet classes for those +# distributions. + +dist_collection = returnInputParameter() +for sub in dist_collection.subs: + # We create a new distribution class for every RAVEN distribution class in the RAVEN input spec, then register that + # new class with the templates.snippets.distributions module so they can be imported as expected. + dist_class = distributions.distribution_class_from_spec(sub) + setattr(distributions, dist_class.tag, dist_class) + +# Register all RavenSnippet subclasses with the SnippetFactory +factory = SnippetFactory() +factory.register_all_subclasses(RavenSnippet) diff --git a/templates/snippets/files.py b/templates/snippets/files.py new file mode 100644 index 00000000..57557db9 --- /dev/null +++ b/templates/snippets/files.py @@ -0,0 +1,65 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + File snippets + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import xml.etree.ElementTree as ET + +from .base import RavenSnippet + + +class File(RavenSnippet): + """ Snippet class for input files """ + snippet_class = "Files" + tag = "Input" + + def to_assembler_node(self, tag: str) -> ET.Element: + """ + Refer to the snippet with an assembler node representation + @ In, tag, str, the tag of the assembler node to use + @ Out, node, ET.Element, the assembler node + """ + node = super().to_assembler_node(tag) + # Type is set as a node attribute and is not the main node tag, as is assumed in the default implementation. + # The "type" attribute should be an empty string if not set. + node.set("type", self.type or "") + return node + + @property + def type(self) -> str | None: + """ + type getter + @ In, None, + @ Out, type, str | None, the file type + """ + return self.get("type", None) + + @type.setter + def type(self, value: str) -> None: + """ + type setter + @ In, value, str, the type value to set + @ Out, None + """ + self.set("type", value) + + @property + def path(self) -> str | None: + """ + file path getter + @ In, None, + @ Out, path, str | None, the file path + """ + return self.text + + @path.setter + def path(self, value: str) -> None: + """ + file path setter + @ In, value, str, the file path to set + @ Out, None + """ + self.text = value diff --git a/templates/snippets/models.py b/templates/snippets/models.py new file mode 100644 index 00000000..6d4af0f2 --- /dev/null +++ b/templates/snippets/models.py @@ -0,0 +1,275 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Model snippets + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import xml.etree.ElementTree as ET + +from ..xml_utils import find_node +from ..decorators import listproperty +from .base import RavenSnippet + + +class RavenCode(RavenSnippet): + """ Sets up a model to run inner RAVEN workflow. """ + tag = "Code" + snippet_class = "Models" + subtype = "RAVEN" + + def __init__(self, name: str | None = None) -> None: + """ + Constructor + @ In, name, str, optional, the model name + @ Out, None + """ + super().__init__(name) + self._py_cmd = None + + def add_alias(self, name: str, suffix: str | None = None, loc: str | None = None) -> None: + """ + Add an alias node to the model. + @ In, name, str, the variable name to alias + @ In, suffix, str, optional, a suffix to append to 'name' + @ In, loc, str, optional, the location of the variable in the workflow + default: Samplers|MonteCarlo@name:mc_arma_dispatch + @ Out, None + """ + varname = name if not suffix else f"{name}_{suffix}" + if not loc: + loc = "Samplers|MonteCarlo@name:mc_arma_dispatch" # where this is pointing 9/10 times + alias_text = f"{loc}|constant@name:{varname}" + alias = ET.SubElement(self, "alias", {"variable": varname, "type": "input"}) + alias.text = alias_text + + def set_inner_data_handling(self, dest: str, dest_type: str) -> None: + """ + Set the inner-to-outer data handling source object. + @ In, dest, str, the name of the data object + @ In, dest_type, str, the type of object used to pass the data ("csv" or "netcdf") + @ Out, None + """ + if dest_type == "csv": + remove_tag = "outputDatabase" + keep_tag = "outputExportOutStreams" + elif dest_type == "netcdf": + remove_tag = "outputExportOutStreams" + keep_tag = "outputDatabase" + else: + raise ValueError("Model output export destination must be a CSV OutStream or " + f"a NetCDF Database. Received: {type(dest)}") + + if (remove_node := self.find(remove_tag)) is not None: + self.remove(remove_node) + + keep_node = self.find(keep_tag) + if keep_node is None: + keep_node = ET.SubElement(self, keep_tag) + keep_node.text = dest + + @property + def executable(self) -> str | None: + """ + RAVEN executable path getter + @ In, None + @ Out, executable, str | None, the executable path + """ + node = self.find("executable") + return None if node is None else node.text + + @executable.setter + def executable(self, value: str) -> None: + """ + RAVEN executable path setter + @ In, value, str, the executable path + @ Out, None + """ + find_node(self, "executable").text = value + + @property + def python_command(self) -> str | None: + """ + Python command getter + @ In, None + @ Out, py_cmd, str | None, the Python command + """ + return self._py_cmd + + @python_command.setter + def python_command(self, cmd: str) -> None: + """ + Python command setter + @ In, cmd, str, the python command + @ Out, None + """ + if self._py_cmd is None: + ET.SubElement(self, "clargs", {"type": "prepend", "arg": cmd}) + else: + node = self.find(f"clargs[@type='prepend' and @arg='{cmd}']") + node.set("arg", cmd) + self._py_cmd = cmd + + +class GaussianProcessRegressor(RavenSnippet): + """ A Gaussian Process Regressor model snippet """ + tag = "ROM" + snippet_class = "Models" + subtype = "GaussianProcessRegressor" + + def __init__(self, name: str | None = None) -> None: + """ + Constructor + @ In, name, str, optional, snippet name + @ Out, None + """ + # FIXME: Only custom_kernel setting exposed to HERON input + default_settings = { + "alpha": 1e-8, + "n_restarts_optimizer": 5, + "normalize_y": True, + "kernel": "Custom", + "custom_kernel": "(Constant*Matern)", + "anisotropic": True, + "multioutput": False + } + super().__init__(name, default_settings) + + @listproperty + def features(self) -> list[str]: + """ + Features list getter + @ In, None + @ Out, features, list[str], features list + """ + node = self.find("Features") + return getattr(node, "text", []) + + @features.setter + def features(self, value: list[str]) -> None: + """ + Features list setter + @ In, value, list[str], features list + @ Out, None + """ + find_node(self, "Features").text = value + + @listproperty + def target(self) -> str: + """ + Target getter + @ In, None + @ Out, target, str, target variables + """ + node = self.find("Target") + return getattr(node, "text", []) + + @target.setter + def target(self, value: list[str]) -> None: + """ + Target setter + @ In, value, list[str], target list + @ Out, None + """ + find_node(self, "Target").text = value + + @property + def custom_kernel(self) -> str: + """ + Custom kernel getter + @ In, None + @ Out, custom_kernel, str, the custom kernel + """ + node = self.find("custom_kernel") + return node.text + + @custom_kernel.setter + def custom_kernel(self, value: str) -> None: + """ + Custom kernel setter + @ Ine, value, str, the custom kernel + @ Out, None + """ + self.find("custom_kernel").text = value + + +class EnsembleModel(RavenSnippet): + """ EnsembleModel snippet class """ + tag = "EnsembleModel" + snippet_class = "Models" + subtype = "" + + +class EconomicRatioPostProcessor(RavenSnippet): + """ PostProcessor snippet for EconomicRatio postprocessors """ + tag = "PostProcessor" + snippet_class = "Models" + subtype = "EconomicRatio" + + def add_statistic(self, tag: str, prefix: str, variable: str, **kwargs) -> None: + """ + Add a statistic to the postprocessor + @ In, tag, str, the node tag (also the statistic name) + @ In, prefix, str, the statistic prefix + @ In, variable, str, the variable name + @ In, kwargs, dict, keyword arguments for additional attributes + @ Out, None + """ + # NOTE: This allows duplicate nodes to be added. It's good to avoid that but won't cause anything to crash. + ET.SubElement(self, tag, prefix=prefix, **kwargs).text = variable + + +class ExternalModel(RavenSnippet): + """ ExternalModel snippet class """ + tag = "ExternalModel" + snippet_class = "Models" + subtype = "" + + @classmethod + def from_xml(cls, node: ET.Element) -> "ExternalModel": + """ + Create a snippet class from a XML node + @ In, node, ET.Element, the XML node + @ Out, model, ExternalModel, an external model snippet object + """ + model = cls() + model.attrib |= node.attrib + for sub in node: + if sub.tag == "variables": + model.variables = [v.strip() for v in sub.text.split(",")] + else: + model.append(sub) + return model + + @listproperty + def variables(self) -> list[str]: + """ + Variables list getter + @ In, None + @ Out, variables, list[str], the model variables + """ + node = self.find("variables") + return getattr(node, "text", []) + + @variables.setter + def variables(self, value: list[str]) -> None: + """ + Variables list getter + @ In, value, list[str], the model variables + @ Out, None + """ + find_node(self, "variables").text = value + + +class HeronDispatchModel(ExternalModel): + """ ExternalModel snippet for HERON dispatch manager models """ + snippet_class = "Models" + subtype = "HERON.DispatchManager" + + +class PickledROM(RavenSnippet): + """ Pickled ROM snippet class """ + tag = "ROM" + snippet_class = "Models" + subtype = "pickledROM" diff --git a/templates/snippets/optimizers.py b/templates/snippets/optimizers.py new file mode 100644 index 00000000..59c754b5 --- /dev/null +++ b/templates/snippets/optimizers.py @@ -0,0 +1,303 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" +Optimization features + +@author: Jacob Bryan (@j-bryan) +@date: 2024-11-08 +""" +from ..xml_utils import find_node +from .base import RavenSnippet +from .samplers import Sampler +from .dataobjects import DataObject + + +# Inheriting from Sampler mimics RAVEN inheritance structure +class Optimizer(Sampler): + """ A base class for RAVEN optimizer XML snippets """ + snippet_class = "Optimizers" + + def set_opt_settings(self, opt_settings: dict) -> None: + """ + Set optimizer settings + @ In, opt_settings, dict, the optimizer settings + @ Out, None + """ + # convergence settings + convergence = find_node(self, "convergence") + for k, v in opt_settings.get("convergence", {}).items(): + find_node(convergence, k).text = v + + # persistence + if "persistence" in opt_settings: + find_node(convergence, "persistence").text = opt_settings["persistence"] + + # samplerInit settings + sampler_init = find_node(self, "samplerInit") + for name in opt_settings.keys() & {"limit", "type"}: # writeEvery not exposed in HERON input + find_node(sampler_init, name).text = opt_settings[name] + + @property + def objective(self) -> str | None: + """ + Optimization objective getter + @ In, None + @ Out, objective, str | None, the objective + """ + node = self.find("objective") + return None if node is None else node.text + + @objective.setter + def objective(self, value: str) -> None: + """ + Optimization objective getter + @ In, value, str, the objective + @ Out, None + """ + find_node(self, "objective").text = value + + @property + def target_evaluation(self) -> str | None: + """ + Target evaluation getter + @ In, None + @ Out, target_evaluation, str | None, the target evaluation data object name + """ + node = self.find("TargetEvaluation") + return None if node is None else node.text + + @target_evaluation.setter + def target_evaluation(self, value: DataObject) -> None: + """ + Target evaluation setter + @ In, value, DataObject, the target evaluation data object + @ Out, None + """ + if not getattr(value, "snippet_class", None) == "DataObjects": + raise TypeError(f"Optimizer evaluation target must be set with a DataObject object. Received '{type(value)}'") + assemb = value.to_assembler_node("TargetEvaluation") + # Copy assembler node info over to the existing TargetEvaluation node + target_eval = self.find("TargetEvaluation") + if target_eval is None: + self.append(assemb) + else: + target_eval.attrib.update(assemb.attrib) + target_eval.text = assemb.text + +######################### +# Bayesian Optimization # +######################### + +class BayesianOptimizer(Optimizer): + """ Bayesian optimizer snippet class """ + tag = "BayesianOptimizer" + + default_settings = { + "samplerInit": { + "limit": 100, + "type": "max", + "writeSteps": "every", + # "initialSeed": "" # initialSeed not included here so the RAVEN default internal seed is used if not provided + }, + "ModelSelection": { + "Duration": 1, + "Method": "Internal" + }, + "convergence": { + "acquisition": 1e-5, + "persistence": 4 + }, + # NOTE: Acquisition function defaults are handled by the the AcquisitionFunction class. Adding the "Acquisition" + # key here just ensures the creation of the child node. + "Acquisition": "" + } + + def __init__(self, name: str | None = None) -> None: + """ + Constructor + @ In, name, str, optional, entity name + @ Out, None + """ + super().__init__(name, self.default_settings) + + def set_opt_settings(self, opt_settings: dict) -> None: + """ + Sets optimization settings + @ In, opt_settings, dict, optimization settings + @ Out, None + """ + super().set_opt_settings(opt_settings) + + try: + bo_settings = opt_settings["algorithm"]["BayesianOpt"] + except KeyError: # use defaults + bo_settings = {} + + # acquisition function + acquisition = self.find("Acquisition") + if len(acquisition) > 0: + acquisition.clear() # drops all children, tag, text, attribs, etc. + acquisition.tag = "Acquisition" # replace cleared tag + default_acq_func = "ExpectedImprovement" + acq_func_name = bo_settings.get("acquisition", default_acq_func) + acq_funcs = { + "ExpectedImprovement": ExpectedImprovement, + "ProbabilityOfImprovement": ProbabilityOfImprovement, + "LowerConfidenceBound": LowerConfidenceBound + } + try: + # FIXME: No acquisition function parameters are exposed to the HERON user + acq_func = acq_funcs.get(acq_func_name, default_acq_func)() + acquisition.append(acq_func) + except KeyError as ke: + raise ValueError(f"Unrecognized acquisition function {acq_func_name}. Allowed: {acq_funcs.keys()}") from ke + + # random seed + if "seed" in bo_settings: + sampler_init = find_node(self, "samplerInit") + find_node(sampler_init, "initialSeed").text = bo_settings["seed"] + + # modelSelection + model_selection = find_node(self, "ModelSelection") + for k, v in bo_settings.get("ModelSelection", {}).items(): + find_node(model_selection, k).text = v + + def set_sampler(self, sampler: Sampler) -> None: + """ + Set the sampler used for initializing the optimizer + @ In, sampler, Sampler, the initialization sampler + @ Out, None + """ + sampler_node = find_node(self, "Sampler") + assemb_node = sampler.to_assembler_node("Sampler") + sampler_node.attrib = assemb_node.attrib + sampler_node.text = assemb_node.text + + def set_rom(self, rom: RavenSnippet) -> None: + """ + Set the ROM to be used as a surrogate in the Bayesian optimization + @ In, rom, RavenSnippet + """ + if rom.snippet_class != "Models": + raise TypeError(f"An object which is not a model (type: '{type(rom)}') was provided to the BayesianOptimizer " + "to be the ROM.") + model_node = find_node(self, "ROM") + assemb_node = rom.to_assembler_node("ROM") + model_node.attrib = assemb_node.attrib + model_node.text = assemb_node.text + + +class ExpectedImprovement(RavenSnippet): + """ Expected improvement acquisition function """ + tag = "ExpectedImprovement" + + default_settings = { + "optimizationMethod": "differentialEvolution", + "seedingCount": 30 + } + + def __init__(self) -> None: + """ + Constructor + @ In, None + @ Out, None + """ + super().__init__(subelements=self.default_settings) + +class ProbabilityOfImprovement(RavenSnippet): + """ Probability of improvement acquisition function """ + tag = "ProbabilityOfImprovement" + + default_settings = { + "optimizationMethod": "differentialEvolution", + "seedingCount": 30, + "epsilon": 1, + "rho": 20, + "transient": "Constant" + } + + def __init__(self) -> None: + """ + Constructor + @ In, None + @ Out, None + """ + super().__init__(subelements=self.default_settings) + +class LowerConfidenceBound(RavenSnippet): + """ Lower confidence bound acquisition function """ + tag = "LowerConfidenceBound" + + default_settings = { + "optimizationMethod": "differentialEvolution", + "seedingCount": 30, + "pi": 0.98, + "transient": "Constant" + } + + def __init__(self) -> None: + """ + Constructor + @ In, None + @ Out, None + """ + super().__init__(subelements=self.default_settings) + +#################### +# Gradient Descent # +#################### + +class GradientDescent(Optimizer): + """ Gradient descent optimizer snippet class """ + tag = "GradientDescent" + + default_settings = { + "samplerInit": { + "limit": 800, + "type": "max", + "writeSteps": "every" + }, + "gradient": { # CentralDifference, SPSA not exposed in HERON input + "FiniteDifference": "" # gradDistanceScalar option not exposed + }, + "convergence": { + "persistence": 1, + "gradient": 1e-4, + "objective": 1e-8 + }, + "stepSize": { + "GradientHistory": { + "growthFactor": 2, + "shrinkFactor": 1.5, + "initialStepScale": 0.2 + } + }, + "acceptance": { + "Strict": "" + } + } + + def __init__(self, name: str | None = None) -> None: + """ + Constructor + @ In, name, str, optional, entity name + @ Out, None + """ + super().__init__(name, self.default_settings) + + def set_opt_settings(self, opt_settings: dict) -> None: + """ + Sets the optimization settings + @ In, opt_settings, dict, optimization settings + @ Out, None + """ + super().set_opt_settings(opt_settings) + + try: + gd_settings = opt_settings["algorithm"]["GradientDescent"] + except KeyError: + return # nothing else to do + + # stepSize settings + for name in gd_settings.keys() & {"growthFactor", "shrinkFactor", "initialStepScale"}: + find_node(self, f"stepSize/GradientHistory/{name}").text = gd_settings[name] diff --git a/templates/snippets/outstreams.py b/templates/snippets/outstreams.py new file mode 100644 index 00000000..bb406177 --- /dev/null +++ b/templates/snippets/outstreams.py @@ -0,0 +1,162 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Define OutStreams for RAVEN workflows + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import xml.etree.ElementTree as ET + +from ..xml_utils import find_node +from ..decorators import listproperty +from .base import RavenSnippet +from .dataobjects import DataObject + + +class OutStream(RavenSnippet): + """ Base class for OutStream entities """ + snippet_class = "OutStreams" + + @property + def source(self) -> str | None: + """ + Source getter + @ In, None + @ Out, source, str | None, the data source + """ + node = self.find("source") + return None if node is None or not node.text else str(node.text) + + @source.setter + def source(self, value: str | DataObject) -> None: + """ + Source setter + @ In, value, str | DataObject, the source + @ Out, None + """ + find_node(self, "source").text = value + + +class PrintOutStream(OutStream): + """ OutStream snippet for Print OutStreams """ + tag = "Print" + + def __init__(self, name: str | None = None) -> None: + """ + Constructor + @ In, name, str, optional, entity name + @ Out None + """ + super().__init__(name) + ET.SubElement(self, "type").text = "csv" # The only supported output file type + + +class OptPathPlot(OutStream): + """ OutStream snippet for OptPath plots """ + tag = "Plot" + subtype = "OptPath" + + @listproperty + def variables(self) -> list[str]: + """ + Variables list getter + @ In, None + @ Out, variables, list[str], variables list + """ + node = self.find("vars") + return getattr(node, "text", []) + + @variables.setter + def variables(self, value: list[str]) -> None: + """ + Variables list setter + @ In, value, list[str], variables list + @ Out, None + """ + find_node(self, "vars").text = value + + +class HeronDispatchPlot(OutStream): + """ OutStream snippet for HERON dispatch plots """ + tag = "Plot" + subtype = "HERON.DispatchPlot" + + @classmethod + def from_xml(cls, node: ET.Element) -> "HeronDispatchPlot": + """ + Alternate construction from XML node + @ In, node, ET.Element, the XML node + @ Out, plot, HeronDispatchPlot, the corresponding snippet object + """ + plot = cls() + plot.attrib |= node.attrib + for sub in node: + if sub.tag == "signals": + plot.signals = [s.strip() for s in sub.text.split(",")] + else: + plot.append(sub) + return plot + + @property + def macro_variable(self) -> str | None: + """ + Getter for macro variable node + @ In, None + @ Out, macro_variable, str | None, macro variable name + """ + node = self.find("macro_variable") + return None if node is None else node.text + + @macro_variable.setter + def macro_variable(self, value: str) -> None: + """ + Setter for macro variable node + @ In, value, str, macro variable name + @ Out, None + """ + find_node(self, "macro_variable").text = value + + @property + def micro_variable(self) -> str | None: + """ + Getter for micro variable node + @ In, None + @ Out, micro_variable, str | None, micro variable name + """ + node = self.find("micro_variable") + return None if node is None else node.text + + @micro_variable.setter + def micro_variable(self, value: str) -> None: + """ + Setter for micro variable node + @ In, value, str, micro variable name + @ Out, None + """ + find_node(self, "micro_variable").text = value + + @listproperty + def signals(self) -> list[str]: + """ + Getter for signals node + @ In, None + @ Out, signals, str | None, signals variable names + """ + node = self.find("signals") + return getattr(node, "text", []) + + @signals.setter + def signals(self, value: list[str]) -> None: + """ + Setter for signals node + @ In, value, str, signals variable names + @ Out, None + """ + find_node(self, "signals").text = value + + +class TealCashFlowPlot(OutStream): + """ OutStream snippet for TEAL cashflow plots """ + tag = "Plot" + subtype = "TEAL.CashFlowPlot" diff --git a/templates/snippets/runinfo.py b/templates/snippets/runinfo.py new file mode 100644 index 00000000..d7605c7a --- /dev/null +++ b/templates/snippets/runinfo.py @@ -0,0 +1,215 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Snippet class for the RunInfo block + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import re +import socket +from pathlib import Path +import xml.etree.ElementTree as ET + +from .base import RavenSnippet +from .steps import Step +from ..xml_utils import find_node +from ..decorators import listproperty + + +class RunInfo(RavenSnippet): + """ Snippet class for the RAVEN XML RunInfo block """ + tag = "RunInfo" + + def set_parallel_run_settings(self, parallel_run_info: dict[str, str]) -> None: + """ + Set how to run in parallel + @ In, parallel_run_info, dict[str, str], settings for parallel execution + @ Out, None + """ + # Get special pre-sets for known computing environments + try: + hostname = socket.gethostbyaddr(socket.gethostname())[0] + except socket.gaierror: + hostname = "unknown" + parallel_xml = get_parallel_xml(hostname) + self._apply_parallel_xml(parallel_xml) + + # Handle "memory" setting first since its parent node is the "mode" node + if memory_val := parallel_run_info.pop("memory", None): + find_node(self, "mode/memory").text = memory_val + # All other settings get tacked onto the main RunInfo block + for tag, value in parallel_run_info.items(): + find_node(self, tag).text = value + + def _apply_parallel_xml(self, parallel_xml: ET.Element) -> None: + """ + Apply the parallel processing settings in parallel_xml to the snippet XML tree + @ In, parallel_xml, ET.Element, the XML tree with parallel settings + @ Out, None + """ + for child in parallel_xml.find("useParallel"): + self.append(child) + + # Add parallel method settings from parallel_xml, if present + outer_node = parallel_xml.find("outer") + if outer_node is None: + self.use_internal_parallel = True + else: + for child in outer_node: + self.append(child) + + @property + def job_name(self) -> str | None: + """ + Getter for job name + @ In, None + @ Out, job_name, str | None, the job name + """ + node = self.find("JobName") + return getattr(node, "text", None) + + @job_name.setter + def job_name(self, value: str) -> None: + """ + Setter for job name + @ In, value, str, the job name + @ Out, None + """ + find_node(self, "JobName").text = str(value) + + @property + def working_dir(self) -> str | None: + """ + Getter for working directory + @ In, None + @ Out, working_dir, str | None, the working directory + """ + node = self.find("WorkingDir") + return None if node is None or node.text is None else str(node.text) + + @working_dir.setter + def working_dir(self, value: str) -> None: + """ + Setter for working directory + @ In, value, str, the working directory + @ Out, None + """ + find_node(self, "WorkingDir").text = str(value) + + @property + def batch_size(self) -> int: + """ + Getter for batch size + @ In, None + @ Out, batch_size, int | None, the batch size + """ + node = self.find("batchSize") + return None if node is None else int(getattr(node, "text", 1)) + + @batch_size.setter + def batch_size(self, value: int) -> None: + """ + Setter for batch size + @ In, value, int, the batch size + @ Out, None + """ + find_node(self, "batchSize").text = int(value) + + @property + def use_internal_parallel(self) -> bool: + """ + Getter for internal parallel flag + @ In, None + @ Out, internal_parallel, bool, the internal parallel flag + """ + node = self.find("internalParallel") + return False if node is None else bool(getattr(node, "text", False)) + + @use_internal_parallel.setter + def use_internal_parallel(self, value: bool) -> None: + """ + Setter for internal parallel flag + @ In, value, bool, the internal parallel flag + @ Out, None + """ + # Set node text if True, remove node if False + node = find_node(self, "internalParallel") + if value: + node.text = value + else: + self.remove(node) + + @property + def num_mpi(self) -> int | None: + """ + Getter for number of MPI processes + @ In, None + @ Out, num_mpi, int | None, the number of processes + """ + node = self.find("NumMPI") + return None if node is None else int(getattr(node, "text", 1)) + + @num_mpi.setter + def num_mpi(self, value: int) -> None: + """ + Setter for number of MPI processes + @ In, value, int, the number of MPI processes + @ Out, None + """ + find_node(self, "NumMPI").text = int(value) + + @listproperty + def sequence(self) -> list[str]: + """ + Getter for the step sequence + @ In, None + @ Out, sequence, list[str], list of steps + """ + node = self.find("Sequence") + return getattr(node, "text", []) + + @sequence.setter + def sequence(self, value: list[str | Step]) -> None: + """ + Setter for the step sequence + @ In, value, str, the step sequence + @ Out, None + """ + find_node(self, "Sequence").text = [str(v).strip() for v in value] + +##################### +# UTILITY FUNCTIONS # +##################### + +def get_default_parallel_settings() -> ET.Element: + """ + The default parallelization settings. Used when the hostname doesn't match any parallel settings + XMLs found in HERON/templates/parallel. + @ In, None + @ Out, parallel, ET.Element, the default parallel settings + """ + parallel = ET.Element("parallel") + use_parallel = ET.SubElement(parallel, "useParallel") + mode = ET.SubElement(use_parallel, "mode") + mode.text = "mpi" + mode.append(ET.Element("runQSUB")) + return parallel + + +def get_parallel_xml(hostname: str) -> ET.Element: + """ + Finds the xml file to go with the given hostname. + @ In, hostname, string with the hostname to search for + @ Out, xml, xml.eTree.ElementTree, if an xml file is found then use it, otherwise return the default settings + """ + # Should this allow loading from another directory (such as one next to the input file?) + path = Path(__file__).parent.parent / "parallel" + for filename in path.glob("*.xml"): + cur_xml = ET.parse(filename).getroot() + regexp = cur_xml.attrib["hostregexp"] + print(f"Checking {filename} regexp {regexp} for hostname {hostname}") + if re.match(regexp, hostname): + print("Success!") + return cur_xml + return get_default_parallel_settings() diff --git a/templates/snippets/samplers.py b/templates/snippets/samplers.py new file mode 100644 index 00000000..96e5d6fa --- /dev/null +++ b/templates/snippets/samplers.py @@ -0,0 +1,218 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Sampler snippets + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +from typing import Any +import xml.etree.ElementTree as ET + +from ..xml_utils import find_node +from .base import RavenSnippet +from .distributions import Distribution + + +class SampledVariable(RavenSnippet): + """ A snippet class for sampled variables """ + tag = "variable" + + def use_grid(self, kind: str, construction: str, values: list[float], steps: int | None = None) -> None: + """ + Use a grid of values to sample the variable + @ In, kind, str, the type of sampling to do + @ In, construction, str, how to construct the values to sample + @ In, values, list[float], values used by the constructor + @ In, steps, int, optional, the number of steps to make along the interval defined by 'values' + @ Out, None + """ + attrib = {"type": kind, "construction": construction} + if steps: + attrib["steps"] = steps + ET.SubElement(self, "grid", attrib).text = " ".join([str(v) for v in values]) + + @property + def initial(self) -> float | None: + """ + Getter for initial value + @ In, None + @ Out, initial, float | None, the initial value + """ + node = self.find("initial") + return None if node is None else node.text + + @initial.setter + def initial(self, value: float) -> None: + """ + Setter for initial value + @ In, value, float, the initial value + @ Out, None + """ + find_node(self, "initial").text = value + + @property + def distribution(self) -> str | None: + """ + Getter for distribution name + @ In, None + @ Out, distribution, str | None, the distribution name + """ + node = self.find("distribution") + return None if node is None else node.text + + @distribution.setter + def distribution(self, value: Distribution | str) -> None: + """ + Setter for distribution name + @ In, value, Distribution | str, the distribution object or its name + @ Out, None + """ + find_node(self, "distribution").text = str(value) + + +class Sampler(RavenSnippet): + """ Sampler snippet base class """ + snippet_class = "Samplers" + + @property + def num_sampled_vars(self) -> int: + """ + Getter for the number of variables sampled by the sampler + @ In, None + @ Out, num_sampled_vars, int, the number of sampled variables + """ + return len(self.findall("variable")) + + @property + def denoises(self) -> int | None: + """ + Getter for denoises + @ In, None + @ Out, denoises, int | None, the number of denoises + """ + node = self.find("constant[@name='denoises']") + return None if node is None else node.text + + @denoises.setter + def denoises(self, value: int) -> None: + """ + Setter for denoises + @ In, value, int, the number of denoises + @ Out, None + """ + find_node(self, "constant[@name='denoises']").text = value + + @property + def init_seed(self) -> int | None: + """ + Getter for sampler seed + @ In, None + @ Out, init_seed, int | None, the seed value + """ + node = self.find("samplerInit/initialSeed") + return None if node is None else node.text + + @init_seed.setter + def init_seed(self, value: int) -> None: + """ + Setter for sampler seed + @ In, value, int, the seed value + @ Out, None + """ + find_node(self, "samplerInit/initialSeed").text = value + + @property + def init_limit(self) -> int | None: + """ + Getter for sampling limit + @ In, None + @ Out, init_limit, int | None, the sampling limit + """ + node = self.find("samplerInit/limit") + return None if node is None else node.text + + @init_limit.setter + def init_limit(self, value: int) -> None: + """ + Setter for sampling limit + @ In, value, int, the sampling limit + @ Out, None + """ + find_node(self, "samplerInit/limit").text = value + + def add_variable(self, variable: SampledVariable) -> None: + """ + Add a variable to sample to the sampler + @ In, variable, SampledVariable, the variable to sample + @ Out, None + """ + self.append(variable) + + def add_constant(self, name: str, value: Any) -> None: + """ + Add a constant to the sampler + @ In, name, str, the name of the constant + @ In, value, Any, the value of the constant + @ Out, None + """ + ET.SubElement(self, "constant", attrib={"name": name}).text = value + + def has_variable(self, variable: str | SampledVariable) -> bool: + """ + Does the sampler sample a given variable? + @ In, variable, str | SampledVariable, the variable to check for + @ Out, var_found, bool, if the variable is in the sampler + """ + var_name = variable if isinstance(variable, str) else variable.name + var_found = self.find(f"variable[@name='{var_name}']") is not None + return var_found + +class Grid(Sampler): + """ Grid sampler snippet class """ + tag = "Grid" + +class MonteCarlo(Sampler): + """ Monte Carlo sampler snippet class """ + tag = "MonteCarlo" + +class Stratified(Sampler): + """ Stratified sampler snippet class """ + tag = "Stratified" + + def __init__(self, name: str | None = None) -> None: + """ + Constructor + @ In, name, str, optional, entity name + @ Out, None + """ + super().__init__(name) + # Must have samplerInit node + ET.SubElement(self, "samplerInit") + +class CustomSampler(Sampler): + """ Custom sampler snippet class """ + tag = "CustomSampler" + +class EnsembleForward(Sampler): + """ Ensemble sampler snippet class """ + tag = "EnsembleForward" + sampler_types = {samp_type.tag: samp_type for samp_type in [Grid, MonteCarlo, Stratified, CustomSampler]} + + @classmethod + def from_xml(cls, node: ET.Element) -> "EnsembleForward": + """ + Alternative constructor for creating an EnsembleForward instance from an XML node + @ In, node, ET.Element, the XML node + @ Out, sampler, EnsembleForward, the ensemble sampler snippet object + """ + # The EnsembleForward sampler takes other samplers as subelements, so here we take care to create the + # appropriate Sampler objects for these subnodes if they're of an implemented type. + sampler = cls() + sampler.attrib |= node.attrib + for sub in node: + if sub.tag in cls.sampler_types: + sampler.append(cls.sampler_types[sub.tag].from_xml(sub)) + else: + sampler.append(sub) + return sampler diff --git a/templates/snippets/steps.py b/templates/snippets/steps.py new file mode 100644 index 00000000..6ad9edee --- /dev/null +++ b/templates/snippets/steps.py @@ -0,0 +1,134 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Step snippets + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import xml.etree.ElementTree as ET +from .base import RavenSnippet + + +class Step(RavenSnippet): + """ Step snippet base class """ + snippet_class = "Steps" + _allowed_subs = ["Function", "Input", "Model", "Sampler", "Optimizer", "SolutionExport", "Output"] + + @classmethod + def from_xml(cls, node: ET.Element) -> "Step": + """ + Alternative constructor from XML node + @ In, node, ET.Element, the XML node + @ Out, step, Step, the snippet class object + """ + # Using the match_text requires the text of the nodes to match (in addition to tag and attributes). This is to make + # sure similar assembler nodes (e.g. outputting two PointSets) + return super().from_xml(node, match_text=True) + + def _add_item(self, tag: str, entity: RavenSnippet) -> None: + """ + Add an item to the step + @ In, tag, str, the node tag; defines the role of the entity in the step + @ In, entity, RavenSnippet, the item to add + @ Out, None + """ + if tag not in self._allowed_subs: + raise ValueError(f"Step type {self.tag} does not accept subelements with tag {tag}. " + f"Allowed: {self._allowed_subs}.") + + # Create an Assembler node from the entity + # NOTE: The entity snippet must have defined "class" and "name" attributes + node = entity.to_assembler_node(tag) + + # Is the entity already serving this role in the step? Check so no duplicates are added. + for sub in self.findall(tag): + if sub.attrib == node.attrib and sub.text == node.text: + return + + # Where to insert the node? Let's do it in order to keep things pretty. + for i, sub in enumerate(self): # linear search over subelements since there won't be that many + if self._allowed_subs.index(sub.tag) > self._allowed_subs.index(node.tag): + self.insert(i, node) + break + else: + self.append(node) + + ############################ + # Add subelement utilities # + ############################ + def add_function(self, entity: RavenSnippet) -> None: + """ + Add an entity as a Function + @ In, entity, RavenSnippet, the entity to add to the step + @ Out, None + """ + self._add_item("Function", entity) + + def add_input(self, entity: RavenSnippet) -> None: + """ + Add an entity as an Input + @ In, entity, RavenSnippet, the entity to add to the step + @ Out, None + """ + self._add_item("Input", entity) + + def add_model(self, entity: RavenSnippet) -> None: + """ + Add an entity as a Model + @ In, entity, RavenSnippet, the entity to add to the step + @ Out, None + """ + self._add_item("Model", entity) + + def add_sampler(self, entity: RavenSnippet) -> None: + """ + Add an entity as a Sampler + @ In, entity, RavenSnippet, the entity to add to the step + @ Out, None + """ + self._add_item("Sampler", entity) + + def add_optimizer(self, entity: RavenSnippet) -> None: + """ + Add an entity as an Optimizer + @ In, entity, RavenSnippet, the entity to add to the step + @ Out, None + """ + self._add_item("Optimizer", entity) + + def add_solution_export(self, entity: RavenSnippet) -> None: + """ + Add an entity as a Solution Export + @ In, entity, RavenSnippet, the entity to add to the step + @ Out, None + """ + self._add_item("SolutionExport", entity) + + def add_output(self, entity: RavenSnippet) -> None: + """ + Add an entity as an Output + @ In, entity, RavenSnippet, the entity to add to the step + @ Out, None + """ + self._add_item("Output", entity) + + +class IOStep(Step): + """ An IOStep step snippet """ + tag = "IOStep" + _allowed_subs = ["Input", "Output"] + + +class MultiRun(Step): + """ A MultiRun step snippet """ + tag = "MultiRun" + + +class PostProcess(Step): + """ A PostProcess step snippet """ + tag = "PostProcess" + _allowed_subs = ["Input", "Model", "Output"] + + +# NOTE: RAVEN step types not currently used by HERON: SingleRun, RomTrainer diff --git a/templates/snippets/variablegroups.py b/templates/snippets/variablegroups.py new file mode 100644 index 00000000..0c15d134 --- /dev/null +++ b/templates/snippets/variablegroups.py @@ -0,0 +1,48 @@ +# Copyright 2020, Battelle Energy Alliance, LLC +# ALL RIGHTS RESERVED +""" + Variable group snippet class + + @author: Jacob Bryan (@j-bryan) + @date: 2024-11-08 +""" +import xml.etree.ElementTree as ET + +from .base import RavenSnippet +from ..decorators import listproperty + + +class VariableGroup(RavenSnippet): + """ A group of variable names """ + snippet_class = "VariableGroups" + tag = "Group" + + @classmethod + def from_xml(cls, node: ET.Element) -> "VariableGroup": + """ + Alternate construction from XML node + @ In, node, ET.Element, the XML node + @ Out, vargroup, VariableGroup, the corresponding variable group object + """ + vargroup = cls(node.get("name")) + if node.text: + var_names = [varname.strip() for varname in node.text.split(",")] + vargroup.variables = var_names + return vargroup + + @listproperty + def variables(self) -> list[str]: + """ + Getter for list of variables in group + @ In, None + @ Out, variables, list of variables in group + """ + return self.text + + @variables.setter + def variables(self, value: list[str]) -> None: + """ + Setter for variables list + @ In, value, list[str], list of variable names + @ Out, None """ + self.text = value diff --git a/templates/template_driver.py b/templates/template_driver.py index 6f67d7c5..8eac2903 100644 --- a/templates/template_driver.py +++ b/templates/template_driver.py @@ -1,1979 +1,139 @@ - # Copyright 2020, Battelle Energy Alliance, LLC # ALL RIGHTS RESERVED """ - Holds the template information for creating LCOE SWEEP OPT input files. -""" -import os -import sys -import copy -import shutil -import xml.etree.ElementTree as ET -import itertools as it -import socket -import glob -import re + Driver for selecting the appropriate template, then creating and writing the workflow(s) -import numpy as np + @author: Jacob Bryan (@j-bryan) + @date: 2024-12-23 +""" +from pathlib import Path import dill as pk -# load utils -sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..')) -from HERON.src.base import Base -import HERON.src._utils as hutils -sys.path.pop() - -# get raven location -RAVEN_LOC = os.path.abspath(os.path.join(hutils.get_raven_loc(), "ravenframework")) -try: - import TEAL.src -except ModuleNotFoundError: - CF_LOC = hutils.get_cashflow_loc(raven_path=RAVEN_LOC) - if CF_LOC is None: - raise RuntimeError('TEAL has not been found!\n' + - f'Check TEAL installation for the RAVEN at "{RAVEN_LOC}"') - - sys.path.append(os.path.join(CF_LOC, '..')) -from TEAL.src.main import getProjectLength -from TEAL.src import CashFlows - -sys.path.append(os.path.join(RAVEN_LOC, '..')) -from ravenframework.utils import xmlUtils -from ravenframework.InputTemplates.TemplateBaseClass import Template as TemplateBase -sys.path.pop() - -# default stats abbreviations -DEFAULT_STATS_NAMES = ['expectedValue', 'sigma', 'median'] -SWEEP_DEFAULT_STATS_NAMES = ['maximum', 'minimum', 'percentile', 'samples', 'variance'] - -# prefixes for financial metrics only -FINANCIAL_PREFIXES = ["sharpe", "sortino", "es", "VaR", "glr"] - -class Template(TemplateBase, Base): - """ - Template for lcoe sweep opt class - This templates the workflow split into sweeping over unit capacities - in an OUTER run while optimizing unit dispatch in a INNER run. - - As designed, the ARMA stochastic noise happens entirely on the INNER, - for easier parallelization. - """ - - # dynamic naming templates - TemplateBase.addNamingTemplates({'jobname' : '{case}_{io}', - 'stepname' : '{action}_{subject}', - 'variable' : '{unit}_{feature}', - 'dispatch' : 'Dispatch__{component}__{tracker}__{resource}', - 'tot_activity' :'{stats}_TotalActivity__{component}__{tracker}__{resource}', - 'data object' : '{source}_{contents}', - 'distribution' : '{unit}_{feature}_dist', - 'ARMA sampler' : '{rom}_sampler', - 'lib file' : 'heron.lib', # TODO use case name? - 'cashfname' : '_{component}{cashname}', - 're_cash' : '_rec_{period}_{driverType}{driverName}', - 'cluster_index' : '_ROM_Cluster', - 'metric_name' : '{stats}_{econ}', - }) - - # template nodes - dist_template = xmlUtils.newNode('Uniform') - dist_template.append(xmlUtils.newNode('lowerBound')) - dist_template.append(xmlUtils.newNode('upperBound')) - - var_template = xmlUtils.newNode('variable') - var_template.append(xmlUtils.newNode('distribution')) - - ############ - # API # - ############ - def __repr__(self): - """ - String representation of this Handler and its VP - @ In, None - @ Out, repr, str, string representation - """ - msg = f'