v3.0 (#6)

* WIP: ref prob form docs

* restore missing args

* FIX: filter disabled buses

* FIX: missing switches from events

* UPD: add bounds/start to radiality vars

* ADD: n=0 support for settings

* FIX: stats for networks with no storage

* FIX: failing cl args validation

* FIX: missing field from args test

* FIX: relax opf test tol

* REF: objective equation formatting

* FIX: block-line-losses bigger than cumulative weight

If the block line losses are bigger than the culumlative weight, the objective could become negative

* ADD: quad precision and numerical focus to gurobi settings

* REF: opf problem formatting

* UPD: prepare_dispatch_data

* FIX: constraint_mc_power_balance_shed_block

 wrong call to `PMD.diag`, should have been `LinearAlgebra.diag`

* ADD: inverters

Adds grid-forming/following inverters to the onm problem

* ADD: grid-following storage phase unbalance constraint

* ADD: theta ref inverter constraints and variables

* UPD: radial constraint

* UPD: warning message in _find_switch_id_from_source_id

* ADD: disable-presolver user option

* UPD: JuMP v0.23, Cbc to HiGHS

* FIX: transformer control infeasibility in opf

* REF: Int64->Int

* ADD: get_timestep_fault_currents filter output option

* FIX: typo in _prepare_fault_study_multinetwork_data

* ADD: inverter states to output

* ADD: disable_presolver to settings functions

* UPD: mld objective term balances

* ADD: helper function for correcting switch directions

* UPD: unit tests

* UPD: PMD version for bugfixes

* ADD: args to CLI

* FIX: missing args from input schema

* FIX: missing args from settings parsing

* FIX: HiGHS solver settings

* ADD: acr theta ref constraint

* UPD: unit tests

* ADD: disable-networking argument

* FIX: disable_switch_penalty had opposite of intended effect

* UPD: objective_mc_min_storage_utilization

Add scaled cost functions for gen

* FIX: radiality constraint

* FIX: switch scores

* ADD: constraint_mc_storage_phase_unbalance_grid_following for active power forms

* FIX: constraint_mc_storage_losses_block_on_off for no bounds versions

* FIX: bus_type missing for disabled buses in get_timestep_bus_types

* FIX: _prepare_fault_study_multinetwork_data

* FIX: build_block_mld, wrong storage balance constraint

* ADD: constraint_mc_power_balance_shed_traditional for NFA form

* REF: constraint_grid_forming_inverter_per_cc

Don't need traditional and block forms of this constraint

* ADD: BIM versions of build_traditional_mld

* UPD: unit tests

* UPD: unit tests

* ADD: build_settings tests

* UPD: fault unit tests

* UPD: events schema, to allow string values

* FIX: missing inverter outputs from output schema

* ADD: disable-networking constraint

* FIX: Events schema validation

* UPD: unit tests to attempt to lift degeneracy

* UPD: find better unit tests

* UPD: build_events_file documentation, multiple dispatch

* ADD: inverter field to schemas to control z_inverter variable

* UPD: filtered logger creation

* ADD: prepare_data!

* UPD: output validation warning

* ADD: KNITRO option to CLI

* REF: recursive_merge_no_vecs to recursive_merge

* RM: disable-networking from ref

* FIX: angmin,angmax init in acp form

* ADD: correct_json_import!

* ADD: autogen output from schema

* UPD: output schema defaults

* UDP: sys metadata schema defaults

* ADD: JuMP 1.0 support

* ADD: ref_add_options!

* REF: User Options

* FIX: get_timestep_inverter_states!

* FIX: get_timestep_microgrid_networks

* ADD: status to protection info output

* REF: _convert_event_data_types!

* UPD: build_events

* RM: deprecated arguments

* ADD: autogen runtime args parser from schema

* UPD: unit tests

* UPD: events schema

* UPD: Additional statistics schema

* UPD: device action timeline schema

* UPD: powerflow output schema

* UPD: protection info output schema

* UPD: beginners guide

* ADD: DSS settings schema

Enabled easier addition of new properties by dss source ids, e.g. vsource.source, etc.

* FIX: fixed inverter variable

* ADD: instantiate_onm_model

* ADD: transform_data_model for ONM

* FIX: variable name typo global_keys_default

* ADD: default switch_close_actions_ub

* DOC: add block mld example

* UPD: Docs

* FIX: bug in `correct_json_import!`

passing `v` instead of `data[k]` to `_fix_symbols!` was overwriting previously changed values.

* FIX: parse_dss_settings

* FIX: parse_dss_settings (Base.parse)

* FIX: writing settings Symbols to JSON

Symbols were getting written as plain strings to JSON, with no way to differentiate them between Symbols ans Strings. Added JSON.lower to fix this.

* RM: time-elapsed-scale

broke native time-elapsed derived from network data structure

* FIX: Gurobi Presolve default

should be -1, not 1

* FIX: constraint_mc_storage_losses_block_on_off

ps constraint should have been outside of if-else statement

* FIX: unbound-generation-power option

Was missing constraint_mc_generator_power_block_on_off

* FIX: unbound-storage-power

was missing constraint_mc_storage_thermal_limit

* UPD: disable-load-block-weight-cost

sets weights to 1.0 if option is true

* ADD: solution_blocks!

Adds solution processor to put block id information in solution dictionary

* FIX: unit tests

* ADD: disable-microgrid-expansion option

* FIX: nlp_solver_tol deprecation

* FIX: objective options

* FIX: build_settings defaults

* ADD: info for when fix-small-numbers is applied

* UPD: operations algorithm name update

* RM: defaults from deprecated fields

* FIX: operations-algorithm value conversion

* FIX: overwriting of properties with defaults

* ADD: set_option! helper functions

* ADD: example - JuMP Model by hand for MLD-Block problem

* DOC: update JuMP Model by Hand documentation

* UPD: beginners guide pluto notebook

* Explicit export of power models

* ADD: use case data

* DOC: Schema descriptions

* DOC: missing docstring

* ADD: strictly increasing constraint

* DOC: Use case examples notebook

* DOC: update documentation

* ADD: graphml export

* UPD: import pattern

* ADD: stability, faults options, settings helpers

* Remove ProgressMeter

* UPD: Use case examples

* FIX: parse(AbstractUnbalancePowerModel, ...)

* UPD: default stability run, only if inverters file

* DOC: Changelog

* FIX: documentation build: schema

* DOC: disable use-case pluto notebook build

* FIX: knitro default presolve setting

* FIX: knitro deprecated tol path

* FIX: parse_events!

* FIX: fault event_type

* FIX: _fix_nulls!

* FIX: PMP dss2eng functions

* FIX: settings setters

* FIX: build_settings_new

* FIX: build_mc_fault_study => sparse

* ADD: unit tests

* DOC: remove PowerModelsONM from docs project.toml

* ADD: unit test for analyze_results!

* DOC: Add infrastructure models ecosystem graph

* ADD: debugging messages for documentation

* ADD: data unit tests

* FIX: log-level option, skip option

* UPD: beginners guide

* DOC: re-enabled use case examples notebook

* FIX: log-level

* FIX: output_data schema validation

* FIX: log-level check in solver instances

* FIX: ACP/ACR functions

* ADD: parse_file

* ADD: graphml tests

* UPD: Juniper solver settings

* ADD: explicit exports

* ADD: multinetwork kwarg to apply_settings

* FIX: deprecated to juniper mapping

* FIX: import issue

* ADD: switch settings

* UPD: graphml tests

* ADD: nlp mld tests

* FIX: broken unit test

* FIX: Adjust nlp test solver parameters

* UPD: solver settings for nlp tests

.github/workflows/documentation.yml

@@ -13,12 +13,8 @@ jobs:
     steps:
       - uses: actions/checkout@v2
       - uses: julia-actions/julia-buildpkg@latest
-      - uses: actions/setup-node@v1
-        with:
-          node-version: '16.x'
-      - name: Install @adobe/jsonschema2md
-        run: npm install -g @adobe/[email protected]
       - uses: julia-actions/julia-docdeploy@latest
         env:
+          PYTHON: ""
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}

CHANGELOG.md

@@ -4,6 +4,74 @@
 
 - none
 
+## v3.0.0
+
+- Added documentation for GraphML export
+- Updated process flow diagram for ONM
+- Added helper functions to set options and get options from the different data structures used by ONM
+- Explicitly exported a number of AbstractUnbalancedPowerModels from PowerModelsDistribution, for better user experience
+- Switched to `import LongName as LN` pattern
+- Updated `"iterative"` to `"rolling-horizon"` and `"global"` to `"full-lookahead"` (**breaking**)
+- Deprecated many runtime arguments in favor of settings schema
+- Updated default logger settings
+- Added `build_settings_new` functions to match updated schema
+- Refactored to use schemas directly to build Julia data structures, to make API maintanence easier
+- Added `prepare_data!` function to quickly build the multinetwork `network` data from `network`, `settings` and `events` files
+- Refactored settings functions to apply settings to base_network and then rebuild the multinetwork structure (**breaking**)
+- Refactored settings schemas to allow for more options for user control of different parts of the entrypoint function (**breaking**)
+- Added more documentation for new users to the examples folder, including use cases, basic usage of the Block-MLD problem, and how to build a JuMP model by hand
+- Added support for exporting network data as a graph in the GraphML format
+- Added EzXML as a dependency to support GraphML export
+- Removed ProgressMeter dependency
+- Added support for JuMP v1
+- Added `transform_data_model` specific to ONM
+- Added `instantiate_onm_model`, an ONM-specific version of `instantiate_mc_model` from PowerModelsDistribution
+- Added `dss` settings schema for easier adding of inverter property by source id, e.g., "vsource.source", etc.
+- Added `constraint_disable_networking` based on coloring model to enabled microgrids to expand but not network
+- Updated events schema to allow for typical string values for certain switch fields
+- Added disable-networking option to CLI for future implementation of feature
+- Changed objective function term balances to ensure that restoring load is always the most critical term
+- Fixed bug in `_prepare_fault_study_multinetwork_data` where `va` was not being used
+- Added option to `get_timestep_fault_currents` to filter out switches from outputs that have no associated protection devices (i.e., relay, recloser, fuse)
+- Changed instances of `Int64` to `Int`
+- Fixed issue with transformer control constraints in dispatch optimization
+- Added support for [JuMP](https://jump.dev) v0.23
+- Changed built-in mip solver from [Cbc](https://github.com/jump-dev/Cbc.jl) to [HiGHS](https://github.com/jump-dev/HiGHS.jl)
+- Added user option to disable presolvers in the built-in solvers `disable_presolver`
+- Changed `@warn` to `@info` in `_find_switch_id_from_source_id`
+- Updated radial topology constraint in `constraint_radial_topology` to be switch-direction-agnostic (previously required a strongly connected directed graph)
+- Added constraint for reference buses that uses `inverter` state to set theta constraints, `constraint_mc_inverter_theta_ref`
+- Added phase unbalance constraint for grid-following storage `constraint_mc_storage_phase_unbalance_grid_following`
+- Added inverters to `_prepare_fault_study_multinetwork_data` and `_prepare_dispatch_data`
+- Added user option to disable inverter constraint: `disable_inverter_constraint`
+- Added constraint for identifying a single grid-forming inverter per connected component `constraint_grid_forming_inverter_per_cc_{block|traditional}`
+- Added `get_timestep_inverter_states!` which adds inverter states to the `"Powerflow output"`
+- Added `solution_inverter!`, which converts `inverter` variable value to `Inverter` enum
+- Added `Inverter`, with `GRID_FOLLOWING` and `GRID_FORMING` enums to indicate what generation object is acting as grid-forming or following
+- Fixed `constraint_mc_power_balance_shed_block`, wrong call to `PMD.diag`, should have been `LinearAlgebra.diag`
+- Added `cost_pg_parameters` and `cost_pg_model` to settings schemas for generators, voltage sources, and storage and solar devices
+- Added `opt-switch-problem` flag to runtime input to enable section of `block` or `traditional` optimal switching problems
+- Removed `SwitchModel` types to realign software design with InfrastructureModels (**breaking**)
+- Refactored problems to better delineate mld code from PMD (**breaking**)
+- Added `traditional` mld problem
+- Renamed problems, objective functions, and constraint functions to be more simple for users (**breaking**)
+- Added solution processor function `solution_statuses!` to assist in converting solution statuses to `Status` enums
+- Fixed `_prepare_dispatch_data` to account for new `traditional` mld problem type
+- Disabled _indicator_ constraints (**breaking**)
+- Fixed bug in `get_timestep_microgrid_networks`
+- Introduced `block` and `traditional` versions of constraints to account for different `z` indicator variables (**breaking**)
+- Renamed `constraint_switch_state_max_actions` to `constraint_switch_close_action_limit` to better reflect the nature of the constraint (**breaking**)
+- Renamed `variable_mc_block_indicator` to `variable_block_indicator`, since it was not a multiconductor variable (**breaking**)
+- Renamed `variable_mc_switch_state` to `variable_switch_state`, since it was not a multiconductor variable (**breaking**)
+- Refactored `variable_mc_switch_fixed` to be called from inside `variable_switch_state` directly (**breaking**)
+- Added `variable_mc_storage_power_mi_on_off`, which will not attempt to make its own `z_storage` indicator variable as in PowerModelsDistribution
+- Added "traditional" indicator variable functions, `variable_bus_voltage_indicator`, `variable_generator_indicator`, `variable_storage_indicator`, and `variable_load_indicator`
+- Fixed bug in `solution_reference_buses!`
+- Added `solve_onm_model`
+- Updated README
+- Updated documentation
+- Updated examples
+
 ## v2.1.2
 
 - Fixed documentation build process
@@ -88,7 +156,7 @@
 - Added `mn_opf_oltc_capc` problem for dispatch step
 - Added `variable_mc_storage_indicator` due to overlap of `z_storage` with `z_block`
 - Added LinearAlgebra (stdlib) dependency
-- Fixed order of parsing in `entrypoint` (events should go *after* settings)
+- Fixed order of parsing in `entrypoint` (events should go _after_ settings)
 - Add support for `null` values in settings schema, and new objects / fields
 - Updated power variables to be `bounded=false` in the switching problem, and use ampacity constraints only instead
 - Updated `sbase_default` to `1e3` to avoid convergence issues

Project.toml

@@ -1,14 +1,15 @@
 name = "PowerModelsONM"
 uuid = "25264005-a304-4053-a338-565045d392ac"
 authors = ["David M Fobes <[email protected]>"]
-version = "2.1.2"
+version = "3.0.0"
 
 [deps]
 ArgParse = "c7e460c6-2fb9-53a9-8c5b-16f535851c63"
-Cbc = "9961bab8-2fa3-5c5a-9d89-47fab24efd76"
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
 Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
+EzXML = "8f5d6c58-4d21-5cfd-889c-e3ad7ee6a615"
 Graphs = "86223c79-3864-5bf0-83f7-82e725a168b6"
+HiGHS = "87dc4568-4c63-4d18-b0c0-bb2238e4078b"
 Hwloc = "0e44f5e4-bd66-52a0-8798-143a42290a1d"
 InfrastructureModels = "2030c09a-7f63-5d83-885d-db604e0e9cc0"
 Ipopt = "b6b21f68-93f8-5de0-b562-5493be1d77c9"
@@ -19,41 +20,41 @@ Juniper = "2ddba703-00a4-53a7-87a5-e8b9971dde84"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
 Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
 LoggingExtras = "e6f89c97-d47a-5376-807f-9c37f3926c36"
-PackageCompiler = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d"
 Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 PolyhedralRelaxations = "2e741578-48fa-11ea-2d62-b52c946f73a0"
 PowerModelsDistribution = "d7431456-977f-11e9-2de3-97ff7677985e"
 PowerModelsProtection = "719c1aef-945b-435a-a240-4c2992e5e0df"
 PowerModelsStability = "f9e4c324-c3b6-4bca-9c3d-419775f0bd17"
-ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
 Requires = "ae029012-a4dd-5104-9daa-d747884805df"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 
 [compat]
-ArgParse = "~1.1"
-Cbc = "~0.9"
-Graphs = "~1.4.1"
-Hwloc = "~2.0"
-InfrastructureModels = "~0.7"
-Ipopt = "~0.9"
-JSON = "~0.21"
-JSONSchema = "~0.3.3"
-JuMP = "~0.22"
-Juniper = "~0.8"
-LoggingExtras = "~0.4.7"
-PackageCompiler = "~1.2.6"
-PolyhedralRelaxations = "~0.3.2"
-PowerModelsDistribution = "~0.14.1"
-PowerModelsProtection = "~0.5"
-PowerModelsStability = "~0.3.0"
-ProgressMeter = "~1.7.1"
-Requires = "~1.1.3"
-julia = "^1.6"
+ArgParse = "1.1"
+EzXML = "1.1.0"
+Graphs = "1.4.1, 1.6, 1.7"
+HiGHS = "1.1.3"
+Hwloc = "2"
+InfrastructureModels = "0.7"
+Ipopt = "0.9, 1.0.2"
+JSON = "0.21"
+JSONSchema = "0.3.3, 1"
+JuMP = "0.22, 0.23, 1"
+Juniper = "0.8, 0.9"
+LoggingExtras = "0.4.7, 0.4.9"
+PolyhedralRelaxations = "0.3.3, 0.3.4"
+PowerModelsDistribution = "0.14.4"
+PowerModelsProtection = "0.5.2"
+PowerModelsStability = "0.3.2"
+Requires = "1.1.3, 1.3"
+julia = "1.6"
 
 [extras]
+HiGHS = "87dc4568-4c63-4d18-b0c0-bb2238e4078b"
+Ipopt = "b6b21f68-93f8-5de0-b562-5493be1d77c9"
 JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
+Juniper = "2ddba703-00a4-53a7-87a5-e8b9971dde84"
 PowerModelsDistribution = "d7431456-977f-11e9-2de3-97ff7677985e"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Test", "JSON", "PowerModelsDistribution"]
+test = ["Test", "HiGHS", "Ipopt", "JSON", "Juniper", "PowerModelsDistribution"]

README.md

@@ -1,10 +1,10 @@
 # PowerModelsONM
 
-_An Optmization library for the operation and restoration of electric power distribution feeders featuring networked microgrids_
+An Optmization library for the operation and restoration of electric power distribution feeders featuring networked microgrids
 
-|                             __Documentation__                             |                               __Build Status__                                |
-| :-----------------------------------------------------------------------: | :---------------------------------------------------------------------------: |
-| [![][docs-stable-img]][docs-stable-url] [![][docs-dev-img]][docs-dev-url] | [![][github-actions-img]][github-actions-url] [![][codecov-img]][codecov-url] |
+|                                      **Documentation**                                       |                                          **Build Status**                                          |
+| :------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------: |
+| [![docs-stable][docs-stable-img]][docs-stable-url] [![docs-dev][docs-dev-img]][docs-dev-url] | [![github-actions][github-actions-img]][github-actions-url] [![codecov][codecov-img]][codecov-url] |
 
 This package combines various packages in the [InfrastructureModels.jl](https://github.com/lanl-ansi/InfrastructureModels.jl) optimization library ecosystem, particularly those related to electric power distribution.
 
@@ -14,7 +14,7 @@ PowerModelsONM focuses on optimizing the operations and restoration of phase unb
 
 To install PowerModelsONM, use the built-in Julia package manager
 
-```
+```julia
 pkg> add PowerModelsONM
 ```
 
@@ -44,16 +44,12 @@ This code is provided under a BSD license as part of the Multi-Infrastructure Co
 
 [docs-dev-img]: https://github.com/lanl-ansi/PowerModelsONM.jl/workflows/Documentation/badge.svg
 [docs-dev-url]: https://lanl-ansi.github.io/PowerModelsONM.jl/dev
-
 [docs-stable-img]: https://github.com/lanl-ansi/PowerModelsONM.jl/workflows/Documentation/badge.svg
 [docs-stable-url]: https://lanl-ansi.github.io/PowerModelsONM.jl/stable
-
 [github-actions-img]: https://github.com/lanl-ansi/PowerModelsONM.jl/workflows/CI/badge.svg
 [github-actions-url]: https://github.com/lanl-ansi/PowerModelsONM.jl/actions/workflows/ci.yml
-
 [codecov-img]: https://codecov.io/gh/lanl-ansi/PowerModelsONM.jl/branch/main/graph/badge.svg
 [codecov-url]: https://codecov.io/gh/lanl-ansi/PowerModelsONM.jl
-
 [contrib-url]: https://lanl-ansi.github.io/PowerModelsONM.jl/stable/developer/contributing.html
 [discussions-url]: https://github.com/lanl-ansi/PowerModelsONM.jl/discussions
 [issues-url]: https://github.com/lanl-ansi/PowerModelsONM.jl/issues

docs/.gitignore

@@ -1,3 +1,2 @@
 build/
 site/
-src/schemas

docs/Project.toml

@@ -1,11 +1,13 @@
 [deps]
+Conda = "8f4d0f93-b110-5947-807f-2305c1781a2d"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 Gumbo = "708ec375-b3d6-5a57-a7ce-8257bf98657a"
-NodeJS = "2bd173c7-0d6d-553b-b6af-13a54713934c"
 Pluto = "c3e4b0f8-55cb-11ea-2926-15256bba5781"
+PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
 
 [compat]
+Conda = "1.7"
 Documenter = "0.27"
 Gumbo = "0.8"
-NodeJS = "1.3"
 Pluto = "0.19"
+PyCall = "1.93.1"