epytflow_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Usage of EPyT-Flow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[EPyT-Flow](https://github.com/WaterFutures/EPyT-Flow) is a Python package building on top of [EPyT](https://github.com/OpenWaterAnalytics/EPyT) \n",
    "for providing easy access to water distribution network simulations.\n",
    "It aims to provide a high-level interface for the easy generation of hydraulic and water quality scenario data.\n",
    "However, it also provides access to low-level functions by [EPANET](https://github.com/USEPA/EPANET2.2) \n",
    "and [EPANET-MSX](https://github.com/USEPA/EPANETMSX/).\n",
    "\n",
    "EPyT-Flow provides easy access to popular benchmark data sets for event detection and localization.\n",
    "Furthermore, it also provides an environment for developing and testing control algorithms.\n",
    "\n",
    "Unique features of EPyT-Flow that make it superior to other (Python) toolboxes are the following:\n",
    "\n",
    "- High-performance hydraulic and (advanced) water quality simulation\n",
    "- High- and low-level interface\n",
    "- Object-orientated design that is easy to extend and customize\n",
    "- Sensor configurations\n",
    "- Wide variety of pre-defined events (e.g. leakages, sensor faults, actuator events, cyber-attacks, etc.)\n",
    "- Wide variety of pre-defined types of uncertainties (e.g. model uncertainties)\n",
    "- Step-wise simulation and environment for training and evaluating control strategies\n",
    "- Serialization module for easy exchange of data and (scenario) configurations\n",
    "- REST API to make EPyT-Flow accessible in other applications\n",
    "- Access to many WDNs and popular benchmarks (incl. their evaluation)\n",
    "\n",
    "### Outline\n",
    "\n",
    "This notebook demonstrates the basic usage of EPyT-Flow:\n",
    "\n",
    "1. Loading an existing (benchmark) scenario.\n",
    "2. Working with the WDN topology:\n",
    "    - Visualizing the network topology\n",
    "    - Retrieving information about the network and solving common problems such as finding the shortest path between two nodes\n",
    "4. Running a complete scenario simulation by:\n",
    "    - Specifying general parameters such as the simulation duration and the hydraulic units.\n",
    "    - Specifying a sensor placement\n",
    "5. Post-processing the simulation results\n",
    "    - Retrieving the final sensor readings\n",
    "    - Plotting time series data\n",
    "\n",
    "More details can be found in the [documentation](https://epyt-flow.readthedocs.io/en/stable) of EPyT-Flow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[EPyT-Flow](https://github.com/WaterFutures/EPyT-Flow) is available on [PyPI](https://pypi.org/project/epyt-flow/) and can be installed via `pip install epyt-flow`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install epyt-flow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=ImportWarning)\n",
    "\n",
    "import os\n",
    "\n",
    "from epyt.epanet import ToolkitConstants\n",
    "from epyt_flow.topology import unitscategoryid_to_str\n",
    "from epyt_flow.simulation import ScenarioSimulator, ScenarioVisualizer\n",
    "from epyt_flow.simulation.scada import ScadaDataXlsxExport\n",
    "from epyt_flow.data.benchmarks import load_leakdb_scenarios\n",
    "from epyt_flow.utils import to_seconds, plot_timeseries_data, get_temp_folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Loading an Existing (Benchmark) Scenario"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load an existing benchmark scenario -- here we load the first Hanoi scenario of [LeakDB](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.data.benchmarks.html#module-epyt_flow.data.benchmarks.leakdb) by calling [`load_leakdb_scenarios()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.data.benchmarks.html#epyt_flow.data.benchmarks.leakdb.load_scenarios):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scenario_config, = load_leakdb_scenarios(scenarios_id=[\"1\"], use_net1=False, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(scenario_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the scenario into a new instance of [`ScenarioSimulator`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sim = ScenarioSimulator(scenario_config=scenario_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, one could load an arbitrary .inp file by setting the `f_inp` argument when creating a new [`ScenarioSimulator`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator) instance -- e.g. for loading the \"Hanoi.inp\" file:\n",
    "\n",
    "```\n",
    "ScenarioSimulator(f_inp=\"Hanoi.inp\")\n",
    "```\n",
    "\n",
    "More details can be found in the [documentation](https://epyt-flow.readthedocs.io/en/stable/tut.scenarios.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with the WDN Topology\n",
    "\n",
    "EPyT-Flow provides easy access to the topology of the WDN, with functions for plotting the network topology, and functions for getting detailed information about the elements in the WDN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the network topology of the loaded scenario utilizing the [`ScenarioVisualizer`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_visualizer.ScenarioVisualizer):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ScenarioVisualizer(sim).plot_topology()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get more information about the loaded WDN by calling [`get_topology()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator.get_topology) of the simulator class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get network topology\n",
    "topo = sim.get_topology()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned [`NetworkTopology`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.html#epyt_flow.topology.NetworkTopology) instance allows us to get more information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print all nodes and edges\n",
    "print(f\"Nodes: {topo.nodes}\")\n",
    "print(f\"Edges: {topo.edges}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the shortest path between nodes \"2\" and \"22\" by using the [`get_shortest_path()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.html#epyt_flow.topology.NetworkTopology.get_shortest_path) function -- either the numer of links or the length of the links are used as weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Shortest path between '2' and '22': {topo.get_shortest_path('2', '22')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get detailed information about a node or link by calling [`get_node_info()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.html#epyt_flow.topology.NetworkTopology.get_node_info)/[`get_link_info()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.html#epyt_flow.topology.NetworkTopology.get_link_info):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Node '2': {topo.get_node_info('2')}\")\n",
    "print(f\"Link '10': {topo.get_link_info('10')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The topology can also be converted to [`GeoDataFrame`](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.html) instances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Nodes as geopandas.GeoDataFrame\n",
    "topo.to_gis()[\"nodes\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the used units by querying the attribute [`units`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.html#epyt_flow.topology.NetworkTopology.units) and convert it to a human-readable string by calling [`unitscategoryid_to_str`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.html#epyt_flow.topology.unitscategoryid_to_str):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(unitscategoryid_to_str(topo.units))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Running a complete Scenario Simulation\n",
    "\n",
    "Running a simulation usually involves at least the following three steps:\n",
    "1. Specifying general parameters such as simulation duration and units. \n",
    "2. Specifying a sensor placement.\n",
    "3. Run the simulation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. We can set some general parameters by calling [`set_general_parameters()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator.set_general_parameters).\n",
    "Here, we set the simulation duration to two days and the flow units to *cubic meter per hour*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sim.set_general_parameters(simulation_duration=to_seconds(days=2),\n",
    "                           flow_units_id=ToolkitConstants.EN_CMH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. The sensor placement can be specified by manually specifying all sensors of each type.\n",
    "Here, we create a couple of pressure sensors and a single flow sensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Place pressure sensors at nodes \"13\", \"16\", \"22\", and \"30\"\n",
    "sim.set_pressure_sensors(sensor_locations=[\"13\", \"16\", \"22\", \"30\"])\n",
    "\n",
    "# Place a flow sensor at link/pipe \"1\"\n",
    "sim.set_flow_sensors(sensor_locations=[\"1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. We run the complete simulation by calling [`run_simulation()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator.run_simulation). The function returns the results as an [`ScadaData`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.scada.html#epyt_flow.simulation.scada.scada_data.ScadaData) instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scada_data = sim.run_simulation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Post-processing the Simulation Results\n",
    "\n",
    "Once the simulation is finished, we can access the final sensor readings and visualize the obtained time series.\n",
    "More advanced post-processings such as changing the measurement units, sensor configuration, etc. are also possible -- see the [documentation](https://epyt-flow.readthedocs.io/en/stable/tut.scada.html) for details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to retrieve the final sensor readings is by calling the corresponding functions separately. E.g. here we retrieve the final readings at all pressure sensors by calling the [`get_data_pressures()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.scada.html#epyt_flow.simulation.scada.scada_data.ScadaData.get_data_pressures) function which returns a [NumPy array](https://numpy.org/doc/stable/reference/generated/numpy.array.html) (first dimension is time):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(scada_data.get_data_pressures())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The retrieved sensor readings (i.e. time series) can be plotted utilizing the [`plot_timeseries_data()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.html#epyt_flow.utils.plot_timeseries_data) function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot all pressure readings over time\n",
    "plot_timeseries_data(scada_data.get_data_pressures().T,\n",
    "                     x_axis_label=\"Time (30min steps)\", y_axis_label=\"Pressure in $m$\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot flow readings over time\n",
    "plot_timeseries_data(scada_data.get_data_flows().T,\n",
    "                     x_axis_label=\"Time (30min steps)\", y_axis_label=\"Flow rate in $m^3/h$\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides accessing and post-processing the data in Python, EPyT-Flow also supports the export of [`ScadaData`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.scada.html#epyt_flow.simulation.scada.scada_data.ScadaData) instances to different file formats:\n",
    "- [NumPy](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.scada.html#epyt_flow.simulation.scada.scada_data_export.ScadaDataNumpyExport)\n",
    "- [MS Excel](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.scada.html#epyt_flow.simulation.scada.scada_data_export.ScadaDataXlsxExport)\n",
    "- [MATLAB](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.scada.html#epyt_flow.simulation.scada.scada_data_export.ScadaDataMatlabExport)\n",
    "\n",
    "Here, we demonstrate how to export the entire simulated data to MS Excel (i.e. .xlsx file):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export all simulated data to MS Excel\n",
    "ScadaDataXlsxExport(os.path.join(get_temp_folder(), \"my_data.xlsx\")).export(scada_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Close the Simulation\n",
    "\n",
    "Do not forget to close the simulation by calling the [`close()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator.close) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sim.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You can avoid calling [`close()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator.close) by opening [`ScenarioSimulator`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator) in a `with` statement -- i.e. [`close()`](https://epyt-flow.readthedocs.io/en/stable/epyt_flow.simulation.html#epyt_flow.simulation.scenario_simulator.ScenarioSimulator.close) is called automatically in the background:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open 'ScenarioSimulator' in a 'with' statement to avoid calling 'close'\n",
    "with ScenarioSimulator(scenario_config=scenario_config) as sim:\n",
    "    # Set some general parameters\n",
    "    sim.set_general_parameters(simulation_duration=to_seconds(days=2),\n",
    "                               flow_units_id=ToolkitConstants.EN_CMH)\n",
    "    \n",
    "    # ....\n",
    "\n",
    "    # Run simulation\n",
    "    sim.run_simulation()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "epytflow2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}