Skip to content

Commit

Permalink
Merge pull request #163 from Dewberry/release/0.5.x
Browse files Browse the repository at this point in the history
Release/0.5.0
  • Loading branch information
mdeshotel authored Sep 9, 2024
2 parents 584d7b4 + 11ea61e commit d865189
Show file tree
Hide file tree
Showing 36 changed files with 21,443 additions and 1,761 deletions.
9 changes: 8 additions & 1 deletion docs/reference.rst → docs/api_reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,14 @@ API Reference / Postman Collection

For reference and documentation of the API, please open the postman collection for the version of ripple1d

`v0.4.1: <https://github.com/Dewberry/ripple1d/blob/v0.4.1/ripple1d/api/postman_collection.json> `_ This beta version contains the endpoints included in the first production testing release. Note that the following variables should be set in the postman environment

`v0.5.0: <https://github.com/Dewberry/ripple1d/blob/3c90acc3fa212fde9c9b361dc3b907beaca17919/ripple1d/api/postman_collection.json>`_ This beta version contains new endpoints:
- `geom_to_gpkg`: Extract the data from a model source dirctory to a gepoackage.
- `conflate`: Conflate all reaches from the NWM network corresponding to the source model.
- `conflation_metrics`: Apply conflation metrics for a conflated source model.


`v0.4.1-v0.4.2: <https://github.com/Dewberry/ripple1d/blob/666190451620e033e8783241c020d2cde21660c9/ripple1d/api/postman_collection.json>`_ This beta version contains the endpoints included in the first production testing release. Note that the following variables should be set in the postman environment


.. code-block:: YAML
Expand Down
3 changes: 2 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ on the NOAA National Water Model network.
:caption: For Users

for_users.rst
reference.rst
api_reference.rst
tech_reference.rst
users_change_log.rst


Expand Down
417 changes: 417 additions & 0 deletions docs/tech_reference.rst

Large diffs are not rendered by default.

76 changes: 69 additions & 7 deletions docs/users_change_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,72 @@ Change Log for Users

Go to the `Releases <https://github.com/Dewberry/ripple1d/releases.html>`_ page for a list of all releases.

Bugfix Release 0.4.1

Feature Release 0.5.0
~~~~~~~~~~~~~~~~~~~~~
Users Changelog
----------------

This release of `ripple1d` incorporates geometry extraction, conflation, and conflation metrics into the API, and fixes several bugs.


Features Added
----------------
**Conflation improvements**

- The source HEC-RAS river centerline is now clipped to the most upstream and most downstream cross sections prior to starting conflation. This helps prevent identifying network reaches that are far away from the cross sections and improves the accuracy of the conflation.
- Overlapped reaches are now tracked and documented in the conflation json file.
- A bbox column has been added to the network parquet file for faster reading. This was especially needed for the new conflation endpoint since each request needs to load the parquet file. Load times without the bbox column were between 5-20 seconds; this is reduced to 1-2 seconds with the bbox column.
- The conflation function now reads locally instead of from s3.
- The conflation function no longer creates a STAC item.
- RAS metadata is now added to the conflation json.
- The source network's metadata is now added to the conflation json.
- Length and flow units are now documented in the conflation json file.

**Conflation Metrics**

Three metrics are computed to asses the qualitiy of the conflation:

- `Coverage`: The the start and end location of the reach coverage is computed as a ratio of the length of the network reach.
- `Lengths`: The lengths between the most upstream cross section and most downstream cross section along the network reach and source HEC-RAS Model's centerline is computed. The ratio of the two lengths is also provided.
- `XS`: The distance between where the network reach and HEC-RAS Model's centerline intersects the cross sections is computed. A similar comparison is performed using the cross section's thalweg location and the network reaches intersection location with the cross sections. The mean, min, max, std, and quartiles are provided as a summary for both comparisons.


**Geometry Extraction improvements**

- A new function to verify .prj file is a HEC-RAS file has been added.
- The extracted geopackage now contians a non-spatial metadata table for the souce HEC-RAS model.
- Tests have been added for extracting geopackage from HEC-RAS model.
- Additional attributes are added to the source model gpkg for downstream use.
- Units are extracted from the source RAS model and added to metadata.

**API**

- An endpoint was added for extracting geometry and relevant metdata for the soure HEC-RAS models and storing it in in a geopackage.
- An endpoint to compute conflation metric was added.
- An endpoint for conflation (which includes metrics calculations) was added.
- Tests were added for the conflation, conflation metrics, and geopackage endpoints.


Bug Fixes
----------

- Reaches whose conflation results indicate upstream and downstream cross sections are the same are now considered a failed conflation.
- The function to create a concave hull for the cross sections has been improved when junctions are present.
- Eclipsed reaches are now better identified and are documented in the conflation json with a boolean.
- A check is now performed to ensure cross sections intersect the source HEC-RAS model's river centerline. If cross sections do not intersect the centerline they are dropped.
- A conflation json is no longer written for source HEC-RAS models that fail to conflate.
- Handling has been added to subset gpkg endpoint for river stationings of interpolated. These river stations contain an "*" to indicate interpolated cross section.
- Several issues with the automated API tests were identified and fixed.
- API tests no longer re-run gpkg_from_ras and conflate_model for every reach; just once per source test model.
- When API tests pass the resulting files are now removed automatically. Resulting files for tests that fail are not removed so that the tester can better trouble shoot.



Bugfix Release 0.4.1-0.4.2
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Users Changelog
----------------
This release of `ripple1d` fixes several bugs identified during testing.

Features Added
Expand All @@ -21,9 +83,7 @@ Bug Fixes
- A bug due to a hard coded terrain path name causing an error on `create_fim_lib` has been resolved.
- A bug associated with the `ripple_version` parameter has been resolved by removing the parameter from the body of requests (see note in API above).
- An issue with including lateral structures (not yet implemented) in the ras geometry files causing hang ups has been resolved. This fix resolved another issue where stationing was mis-applied in the newly created ras geometry files.


-----------
- A bug which caused a failure when calling subset_gpkg in cases where the model geometries are simple (no structures / no junctions).


Feature Release 0.4.0
Expand Down Expand Up @@ -59,11 +119,13 @@ Features Added


**API**
- `ripple_version` is no longer a required argument for any endpoint.

- `ripple_version` is no longer a required argument for any endpoint.


Bug Fixes
----------
Numerous small bug fixes were made to enable the support of hydraulic structures. Other notable bugs include:
- HEC-RAS stations with length > 8 characters are now supported.
- Mangled profile names resulting from negative elevations producing FIM libraries has been fixed.

- HEC-RAS stations with length > 8 characters are now supported.
- Mangled profile names resulting from negative elevations producing FIM libraries has been fixed.
4 changes: 2 additions & 2 deletions production/step_1_extract_geometry.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from ripple1d.errors import (
NotAPrjFile,
)
from ripple1d.ras_to_gpkg import geom_to_gpkg_s3
from ripple1d.ras_to_gpkg import gpkg_from_ras_s3
from ripple1d.ripple1d_logger import configure_logging
from ripple1d.utils.s3_utils import init_s3_resources, list_keys

Expand All @@ -30,7 +30,7 @@ def process_one_geom(

# read the geometry and write the geopackage
if bucket:
geom_to_gpkg_s3(key, crs, gpkg_path, bucket)
gpkg_from_ras_s3(key, crs, gpkg_path, bucket)
return f"s3://{bucket}/{gpkg_path}"


Expand Down
10 changes: 10 additions & 0 deletions production/step_3_5_metrics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
from ripple1d.ops.metrics import compute_conflation_metrics
from ripple1d.ripple1d_logger import configure_logging

if __name__ == "__main__":
TEST_DIR = os.path.dirname(__file__).replace("production", "tests")

src_gpkg_path = os.path.join(TEST_DIR, "ras-data\\Baxter\\Baxter.gpkg")
conflation_json = os.path.join(TEST_DIR, "ras-data\\Baxter\\Baxter.conflation.json")
nwm_pq_path = os.path.join(TEST_DIR, "nwm-data\\flows.parquet")
conflation_parameters = compute_conflation_metrics(src_gpkg_path, nwm_pq_path, conflation_json)
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "ripple1d"
version = "0.4.2"
version = "0.5.0"
description = "HEC-RAS model automation"
readme = "README.md"
maintainers = [
Expand Down
23 changes: 22 additions & 1 deletion ripple1d/api/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,41 @@
from ripple1d.api import tasks
from ripple1d.api.utils import get_unexpected_and_missing_args
from ripple1d.ops.fim_lib import create_fim_lib, fim_lib_stac, nwm_reach_model_stac
from ripple1d.ops.metrics import compute_conflation_metrics
from ripple1d.ops.ras_conflate import conflate_model
from ripple1d.ops.ras_run import (
create_model_run_normal_depth,
run_incremental_normal_depth,
run_known_wse,
)
from ripple1d.ops.ras_terrain import create_ras_terrain
from ripple1d.ops.subset_gpkg import extract_submodel
from ripple1d.ras_to_gpkg import gpkg_from_ras

app = Flask(__name__)


@app.route("/processes/conflate_model/execution", methods=["POST"])
def process__conflate_model():
"""Enqueue a task to conflate a source model."""
return enqueue_async_task(conflate_model)


@app.route("/processes/compute_conflation_metrics/execution", methods=["POST"])
def process__compute_conflation_metrics():
"""Enqueue a task to compute conflation metrics."""
return enqueue_async_task(compute_conflation_metrics)


@app.route("/processes/gpkg_from_ras/execution", methods=["POST"])
def process__gpkg_from_ras():
"""Enqueue a task to create a new GeoPackage from a source model."""
return enqueue_async_task(gpkg_from_ras)


@app.route("/processes/extract_submodel/execution", methods=["POST"])
def process__extract_submodel():
"""Enqueue a task to create a new GeoPackage."""
"""Enqueue a task to create a new GeoPackage for a NWM reach."""
return enqueue_async_task(extract_submodel)


Expand Down
106 changes: 100 additions & 6 deletions ripple1d/api/postman_collection.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"info": {
"_postman_id": "9c07b074-5f35-4309-bbaf-3ed3f1aec045",
"_postman_id": "a4de6034-22c7-4a2b-b91c-12494de19040",
"name": "ripple1d",
"description": "Collection for processing existing HEC-RAS models for use in the production of Flood Inundation Maps (FIMs) and rating curves for use in near-real time flood forecasting on the NOAA National Water Model",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json",
"_exporter_id": "11115435"
"_exporter_id": "29128857"
},
"item": [
{
Expand Down Expand Up @@ -68,6 +68,64 @@
},
"response": []
},
{
"name": "compute_conflation_metrics",
"request": {
"method": "POST",
"header": [],
"body": {
"mode": "raw",
"raw": "{\r\n \"source_model_directory\": \"{{source_model_directory}}\",\r\n \"source_network\": {\"file_name\":\"{{nwm_data_directory}}\\\\flows.parquet\",\r\n \"version\":\"2.1\", // optional\r\n \"file_hash\": \"a6786ba02073f0233b9a69afc49d92bf-178\", // optional\r\n \"type\":\"nwm_hydrofabric\"} \r\n}",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "{{url}}/processes/compute_conflation_metrics/execution",
"host": [
"{{url}}"
],
"path": [
"processes",
"compute_conflation_metrics",
"execution"
]
},
"description": "Compute conflation metrics for a given gpkg/conflation json."
},
"response": []
},
{
"name": "conflate_model",
"request": {
"method": "POST",
"header": [],
"body": {
"mode": "raw",
"raw": "{\r\n \"source_model_directory\": \"{{source_model_directory}}\",\r\n \"source_network\": {\"file_name\":\"{{nwm_data_directory}}\\\\flows.parquet\",\r\n \"version\":\"2.1\", // optional\r\n \"type\":\"nwm_hydrofabric\"}\r\n \r\n}",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "{{url}}/processes/conflate_model/execution",
"host": [
"{{url}}"
],
"path": [
"processes",
"conflate_model",
"execution"
]
},
"description": "Conflate a source HEC-RAS model with the NWM network."
},
"response": []
},
{
"name": "extract_submodel",
"request": {
Expand Down Expand Up @@ -97,6 +155,35 @@
},
"response": []
},
{
"name": "gpkg_from_ras",
"request": {
"method": "POST",
"header": [],
"body": {
"mode": "raw",
"raw": "{\r\n \"source_model_directory\": \"{{source_model_directory}}\",\r\n \"crs\": 2227,\r\n \"metadata\": {\"stac_api\":\"https://stac2.dewberryanalytics.com\", // optional\r\n \"stac_collection_id\":\"ebfe-12090301_LowerColoradoCummins\", // optional\r\n \"stac_item_id\":\"137a9667-e5cf-4cea-b6ec-2e882a42fdc8\"} // optional\r\n}\r\n",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "{{url}}/processes/gpkg_from_ras/execution",
"host": [
"{{url}}"
],
"path": [
"processes",
"gpkg_from_ras",
"execution"
]
},
"description": "Create a new model for a given nwm_id (NWM reach) from an existing HEC-RAS model."
},
"response": []
},
{
"name": "create_ras_terrain",
"request": {
Expand Down Expand Up @@ -151,7 +238,7 @@
"execution"
]
},
"description": "Run a range of flows for Normal Depth on a HEC-RAS model. The results will be used to inform flows required to develop rating curves at a provided increment."
"description": "Run a range of flows for dormal depth on a HEC-RAS model. The results will be used to inform flows required to develop rating curves at a provided increment."
},
"response": []
},
Expand Down Expand Up @@ -209,7 +296,7 @@
"execution"
]
},
"description": "Run a series of HEC-RAS simulations using Known Water Surface Elevations."
"description": "Run a HEC-RAS simulation using Known Water Surface Elevations."
},
"response": []
},
Expand Down Expand Up @@ -266,7 +353,8 @@
"nwm_reach_model_stac",
"execution"
]
}
},
"description": "Create a STAC item for a HEC-RAS model representing a NWM reach."
},
"response": []
},
Expand Down Expand Up @@ -294,7 +382,8 @@
"fim_lib_stac",
"execution"
]
}
},
"description": "Create a STAC item for a FIM library for a NWM reah."
},
"response": []
}
Expand Down Expand Up @@ -505,6 +594,11 @@
"key": "source_model_directory",
"value": "~\\\\repos\\\\ripple1d\\\\tests\\\\ras-data\\\\Baxter",
"type": "string"
},
{
"key": "nwm_data_directory",
"value": "~\\\\repos\\\\ripple1d\\\\tests\\\\nwm-data",
"type": "string"
}
]
}
4 changes: 2 additions & 2 deletions ripple1d/conflate/plotter.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ def plot_conflation_results(
buf.seek(0)
s3_client.put_object(Bucket=bucket, Key=key, Body=buf, ContentType="image/png")
else:
with open(Path(key).name, "w") as f:
plt.savefig(f, format="png")
output_png = str(Path(rfc.source_model_directory) / Path(key).name)
plt.savefig(output_png, format="png")

plt.close()
Loading

0 comments on commit d865189

Please sign in to comment.