Features/restructure time series (#236)

* changed code to match flake8/pep8 conformity

* geopandas is now part of the standard installation and therefore the import was moved to the standard import

* small bug fixes

* removed unused classes in components.py

* moved shapely from dev imports to standard imports

* moved aggregation functionality from edisgo.py to tools.py #226

* removed some unnecessary nesting within the aggregation functionality

* made the aggregation functionality more readable

* set flake8 line length to 88

* Added functionality to export geo referenced components from Grid objects as geopandas.GeoDataFrames
* added new container class for GeoDataFrames
* added a new geopandas property in the Grids class
* added a new method in the Topology class
* added docstrings
* added simple tests

* Move reactive power control to flex_opt

* Introduce TimeSeriesRaw class

* Drop old functions

* Change function to not use edisgo object

* Add functions for manual mode

* Add function for worst case time series of loads

* Delete old code

* Delete old code

* Add attribute to TimeSeriesRaw

* Rewrite helper functions

* Add function to get reactive power default configs per component

* Rename feedin to feed-in

* Change ding0 import temporarily to set type of loads

* Add function to generate feed-in worst case time series

* Add storage units worst case

* Add config values for charging points and heat pumps

* Remove charging points from pypsa_io

* Change charging_points_df property

* Add worst case for charging points

* Add information of parallel lines

* Add function to reset all time series

* Finalise worst case functions

* Add component check when time series are added manually

* Add wrapper in EDisGo for worst case and manual time series setting

* Allow setting of timeindex

* Remove old function for setting time series

* Do not use list as default value

* Return only existing components

* Adapt TimeSeriesRaw class

* Remove old code

* Add function to set conventional load time series by sector

* Delete old code

* Add functions for predefined profiles

* Move function

* Expand docstring

* Remove old docstring content

* Add predefined charging time series per use case

* Delete old imports

* Delete old docstrings

* Update docstring

* Add function to set timeindex

* Add warning in case timeindex is not set

* Add EDisGo wrapper for predefined profiles

* Split function into function to retrieve power factor and power mode

* Expand docstring

* Change import

* Add fixed cosphi function

* Update docstring

* Add wrapper

* Use time index instead of worst cases as index

* Adapt test

* Change naming

* Fix tests

* Remove old function

* Add option to not raise error in case power flow analysis did not converge for all time steps

* Rewrite aggregate_components function

* Minor docstring changes

* Adapt add_component function

* Adapt remove_component function

* Move functions up

* Move function

* Adapt EDisGo tests

* Make function public

* Add ToDo

* Delete setting time series in generator import

* Bug fix

* Raise warning instead of error

* Change type naming to be more consistent

* Adapt tests

* Remove old function

* Minor docstring changes

* Expand worst case time series tests

* Add test ToDos

* Fix tests

* Bug fix

* Change version of black

* Add test

* Fix test

* test

* Fix q control tests

* change timeseries getters such that they raise a warning when index of timeseries and timeindex are not aligned, add respective test

* Merge

* Fix example test

* Fix tests

* simplified if statements in timeseries.py

* f string formatting

* Return not converged time steps

* specifying werkzeug version

* minor fix

* minor fix

* Add to docstring

* Raise warning in case of not converged time steps

* renamed p_nom to p_set for loads (#251)

* Bug fix

* Add test

* adapt import from demandlib such that only the chosen timeindex is returned, adapt test respectively

* add tests for predefined_conventional_loads_by_sector and predefined_conventional_loads_by_sector, slightly adapt methods to raise warnings

* Minor docstring fix

* Bugfix usage of old function

* move consistency check (_validate_ding0_import) from ding0_import.py to topology.py, rename to check_integrity. Change tests respectively.

* Rename function

* simplify topology consistency check and adapt test accordingly

* change ValueErrors in topology consistency to warnings and adapt test accordingly

* Fix minimum working examples

* Fix storage unit integration documentation

* Minor changes

* Fix power flow documentation

* add check_integrity method to timeseries and add respective test

* add check for timeindex in TimeSeries.check_integrity method and adapt respective test

* add check_integrity method to edisgo.py and add respective test

* adapt check_integrity method in edisgo.py such that it only checks active power instead of appearant power and adapt respective test

* adapt check_integrity method in edisgo.py such that it only checks active power instead of appearant power and adapt respective test

* Feature/reinforcement worst cases (#252)

* Changed reinforcement for worst-case calculations.

Reinforcement for worst-case calculations now distinguishes between different
simultaneousness assumptions at the mv and lv levels. An lv reinforcement mode was
therefore added. The TimeSeries Class now has a new property which reflects if the
time series is a worst-case time series.

* using timeseries.timeindex_worst_cases to determine lv and mv worst-case timesteps

* * Renamed time_series_mode to is_worst_case and changed data type to bool
* Only reinforce mv crit nodes if specified
* added simple tests for reinforce_grid modes

* minor test fix

* rewritten is_worst_case to be determined by the timeindex and made it an optional keyword in edisgo.reinforce()

Co-authored-by: birgits <[email protected]>

* Add tests

* Feature/update pandas (#256)

* update check_tech_constraints

* update costs

* update curtailment

* update reinforce_grid

* update reinforce_measures

* update reinforce_measures

* update pypsa_io

* update timeseries

* update topology

* update preprocess_pypsa_opf_structure

* update tests

* updated pandas version and updated code accordingly

* upgrade pip before installing dependencies with github actions

* bug fixes

* Set dtype when initialising empty dataframe

Co-authored-by: Birgit Schachler <[email protected]>

* Simplify test

* Use logger warning instead of raising warning

* Simplify code

* Only overwrite duplicate entries

* Bug fix

* add possibility to check integrity in to_pypsa

* add check of very small impedance values in _check_integrity_of_pypsa

* Update pypsa_io.py

bugfix: check if z is empty first

* minor bug fix

* minor bug fix

* Bug fix

* Add option to get default reactive power when adding component

* Make code shorter

* Bug fix - do not overwrite other reactive power time series

* Add example

* Add test

* Add test

* Expand docstring

* Add test

* Add test residual load

* Add test time steps load and feed-in case

* Add test for reduce_memory

* Add test to_csv

* Bug fix

* Add test from_csv

* Add test reduce_memory

* Use fixture

* Remove yield_fixture as it is deprecated

* Add tests to_ and from_csv

* Add tests for time series helper functions

* replaced format with f-strings

* removed faulty test

* Add test edisgo set_time_series_manual

* Bug fix also reset reactive power time series

* Add option to only set worst case time series for certain components

* Add test for set_worst_case

* Bug fix for empty component lists

* Add return statement to avoid errors

* Add test for set_time_series_worst_case_analysis

* Add test for set_time_series_active_power_predefined

* Bugfix use selected dataframe

* Adapt simultaneity factors

* Add documentation for various time series options

* Add oedb link

* Bug fix grid expansion costs plot without storage

* Mark test as slow

* Adapt tests to changed simultaneity factors

* Update docstrings

* Use greek letter for phi

* Update rtd requirements

* Update modules in API doc

* Only import shapely if environment is not readthedocs

* Add werkzeug to rtd requirements

* Add jupyterdash to rtd requirements

* Change default reactive power behavior of heat pumps

* Minor docstring changes

* Remove pypsa_io check integrity function

* Remove check from time series integrity check

* Add checks to topology check integrity

* Raise warning instead of error

* Fix check integrity tests

Co-authored-by: Kilian Helfenbein <[email protected]>
Co-authored-by: Kilian Helfenbein <[email protected]>
Co-authored-by: AnyaHe <[email protected]>
Co-authored-by: Anya Heider <[email protected]>

.github/workflows/tests-coverage.yml

@@ -19,6 +19,7 @@ jobs:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies
       run: |
+        pip3 install --upgrade pip
         pip3 install -e $GITHUB_WORKSPACE[full]
         pip3 install coveralls
     - name: Run coverage

.pre-commit-config.yaml

@@ -6,13 +6,13 @@ repos:
   -   id: end-of-file-fixer
   -   id: trailing-whitespace
 - repo: https://github.com/psf/black
-  rev: 21.12b0
+  rev: 22.3.0
   hooks:
     - id: black
-#- repo: https://github.com/pycqa/flake8
-#  rev: 4.0.1
-#  hooks:
-#    - id: flake8
+- repo: https://github.com/pycqa/flake8
+  rev: 4.0.1
+  hooks:
+    - id: flake8
 - repo: https://github.com/pycqa/isort
   rev: 5.10.1
   hooks:

doc/api/edisgo.flex_opt.rst

@@ -20,6 +20,14 @@ edisgo.flex\_opt.costs module
 edisgo.flex\_opt.exceptions module
 ----------------------------------
 
+.. automodule:: edisgo.flex_opt.exceptions
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+edisgo.flex\_opt.q\_control module
+----------------------------------
+
 .. automodule:: edisgo.flex_opt.exceptions
     :members:
     :undoc-members:

doc/api/edisgo.tools.rst

@@ -20,6 +20,22 @@ edisgo.tools.edisgo\_run module
 edisgo.tools.geo module
 -----------------------
 
+.. automodule:: edisgo.tools.geo
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+edisgo.tools.geopandas\_helper module
+----------------------------------------
+
+.. automodule:: edisgo.tools.geo
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+edisgo.tools.networkx\_helper module
+----------------------------------------
+
 .. automodule:: edisgo.tools.geo
     :members:
     :undoc-members:
@@ -57,7 +73,6 @@ edisgo.tools.tools module
     :undoc-members:
     :show-inheritance:
 
-
 Module contents
 ---------------
 

doc/conf.py

@@ -28,7 +28,6 @@
 # serve to show the default.
 
 import os
-import shlex
 import sys
 
 from unittest.mock import MagicMock
@@ -55,6 +54,7 @@
     "sphinx.ext.viewcode",
     "sphinx.ext.napoleon",  # enable Napoleon Sphinx v>1.3
     "sphinx.ext.extlinks",  # enables external links with a key
+    "sphinx_autodoc_typehints",
 ]
 
 # Napoleon settings
@@ -76,8 +76,13 @@
         "http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.%s.html",
         "pandas.",
     ),
+    "geopandas": (
+        "https://geopandas.org/en/stable/docs/reference/api/geopandas.%s.html",
+        "geopandas.",
+    ),
     "networkx": (
-        "https://networkx.github.io/documentation/stable/reference/classes/graph.html#%s",
+        "https://networkx.github.io/documentation/stable/reference/classes/"
+        "graph.html#%s",
         "networkx.",
     ),
     "sqlalchemy": (
@@ -86,7 +91,7 @@
     ),
     "shapely": ("https://shapely.readthedocs.io/en/latest/manual.html#%s", "shapely."),
     "ding0": ("https://dingo.readthedocs.io/en/dev/api/ding0.html#%s", "Ding0"),
-    "pypsa": ("https://pypsa.org/doc/components.html#%s", "pypsa"),
+    "pypsa": ("https://pypsa.readthedocs.io/en/latest/components.html#%s", "pypsa"),
     "plotly": ("https://plotly.com/python-api-reference/generated/#%s.html", "plotly"),
 }
 # Add any paths that contain templates here, relative to this directory.
@@ -104,9 +109,9 @@
 master_doc = "index"
 
 # General information about the project.
-project = u"eDisGo"
-copyright = u"2017, open_eGo-Team"
-author = u"open_eGo-Team"
+project = "eDisGo"
+copyright = "2017, open_eGo-Team"
+author = "open_eGo-Team"
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -162,8 +167,9 @@
 todo_include_todos = True
 
 
-# Fix import error of modules which depend on C modules (mock out the imports for these modules)
-# see http://read-the-docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
+# Fix import error of modules which depend on C modules (mock out the imports for these
+# modules) see http://read-the-docs.readthedocs.io/en/latest/
+#  faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
 if "READTHEDOCS" in os.environ:
 
     class Mock(MagicMock):
@@ -181,7 +187,7 @@ def __getattr__(cls, name):
 # a list of builtin themes.
 # html_theme = 'alabaster'
 
-import sphinx_rtd_theme
+import sphinx_rtd_theme  # noqa: E402
 
 html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 html_theme = "sphinx_rtd_theme"
@@ -282,20 +288,20 @@ def __getattr__(cls, name):
 
 latex_elements = {
     # The paper size ('letterpaper' or 'a4paper').
-    #'papersize': 'letterpaper',
+    # 'papersize': 'letterpaper',
     # The font size ('10pt', '11pt' or '12pt').
-    #'pointsize': '10pt',
+    # 'pointsize': '10pt',
     # Additional stuff for the LaTeX preamble.
-    #'preamble': '',
+    # 'preamble': '',
     # Latex figure (float) alignment
-    #'figure_align': 'htbp',
+    # 'figure_align': 'htbp',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, "edisgo.tex", u"eDisGo Documentation", u"open_eGo-Team", "manual"),
+    (master_doc, "edisgo.tex", "eDisGo Documentation", "open_eGo-Team", "manual"),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -323,7 +329,7 @@ def __getattr__(cls, name):
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [(master_doc, "eDisGo", u"eDisGo Documentation", [author], 1)]
+man_pages = [(master_doc, "eDisGo", "eDisGo Documentation", [author], 1)]
 
 # If true, show URL addresses after external links.
 # man_show_urls = False
@@ -338,7 +344,7 @@ def __getattr__(cls, name):
     (
         master_doc,
         "eDisGo",
-        u"eDisGo Documentation",
+        "eDisGo Documentation",
         author,
         "eDisGo",
         "One line description of project.",

doc/features_in_detail.rst

@@ -7,8 +7,7 @@ Power flow analysis
 -------------------
 
 In order to analyse voltages and line loadings a non-linear power flow analysis (PF) using pypsa is conducted.
-All loads and generators are modelled as PQ nodes; the slack is modelled as a PV node with a set voltage of 1\,p.u.
-and positioned at the substation's secondary side.
+All loads and generators are modelled as PQ nodes. The slack is positioned at the substation's secondary side.
 
 Multi period optimal power flow
 ---------------------------------
@@ -267,10 +266,14 @@ Storage integration
 
 .. warning:: The storage integration methods described below are not yet adapted to the refactored code and therefore currently do not work.
 
-Besides the possibility to connect a storage with a given operation to any node in the grid, eDisGo provides a methodology that takes
-a given storage capacity and allocates it to multiple smaller storages such that it reduces line overloading and voltage deviations.
-The methodology is implemented in :py:func:`~edisgo.flex_opt.storage_positioning.one_storage_per_feeder`. As the above described
-curtailment allocation methodologies it is intended to be used in combination with `eTraGo <https://github.com/openego/eTraGo>`_ where
+Besides the possibility to connect a storage with a given operation to any node in the
+grid, eDisGo provides a methodology that takes
+a given storage capacity and allocates it to multiple smaller storage units such that it
+reduces line overloading and voltage deviations.
+The methodology is implemented in :py:func:`~edisgo.flex_opt.storage_positioning.one_storage_per_feeder`.
+As the above described
+curtailment allocation methodologies it is intended to be used in combination
+with `eTraGo <https://github.com/openego/eTraGo>`_ where
 storage capacity and operation is optimized.
 
 For each feeder with load or voltage issues it is checked if integrating a