Subregional planning support

[twelch]

Resolves

• Decouple precalc from data:import #196 create prealc command with first-class support for geographies. Migrates all the good work from azores reports supporting subregions - Add subregional reporting to Azores nearshore reports azores-nearshore-reports#11
• template-ocean-eez hard-coded to use fsm eez #191 Support stats for external datasources
• Totals can be 0 in subregion #203 0 value for subregions
• Solve pain point in using template-ocean-eez #168

Requirements

• Create multi-geog tests for precalcVectorDatasource
• Create multi-geog tests for precalcRasterDatasource

Geographies
Create first-class support for project geographies, which can be the entire world, a planning region and/or one or more subregions within it.

• Geography has one defined datasource (internal or external)

Geography pointing at internal datasource:

• Expected to be available in geojson format containing a feature collection of Polygon or MultiPolygon features. Should error if not. Flatgeobuf ran into an issue and support can be added in the future.
• Geography can be configured with a propertyFilter to identify a specific feature and geographic area within the datasource where that features is located. Otherwise defaults to using all features in the collection as the geography.
• should be able to filter to a single geography within a multiple polygon datasource
• should be able to filter to multiple geography polygons within a multiple polygon datasource

Geography pointing at external datasource:

• External datasource geography is expected to be available in flatgeobuf or subdivided format containing an array/collection of Polygon or MultiPolygon features. Throw error if not.

• Geography must be configured with a propertyFilter and bboxFilter to identify a specific feature and geographic area within the external datasource to quickly fetch the features.

• should be able to filter to a single geography within a multiple polygon datasource

• should be able to filter to multiple geography polygons within a multiple polygon datasource

• A groups property should allow the geography to be associated with one or more geography groups. Default to project boundary group identifying the top-level geography. Report client can use groups to retrieve precalculated metrics for the appropriate set of geographies, even if the the geographies change.

• Should be able to exclude geography from precalculations (precalc: false), so that the geography can stay present in config, without being used.

• Should be able to exclude datasource from precalculations (precalc: false) but still used for a geography.

Global datasources

• Add precalc: false to all global datasources by default. Developer can remove it if they want to start using it as a datasource in reports.

How to use global datasources going forward in projects:

OSM land polygons have been traditionally used by SeaSketch as a baseline (coastline) because it matches basemaps that use OSM (like Mapbox used by SeaSketch). Its accuracy wasn't questioned until it was seen that for small island nations it can be missing small landmasses. This matters when defining for example a nearshore boundary for planning, if you are missing small islands, the nearshore area around them will not be included in the planning area or in drawn sketches. The % of area relative to the whole EEZ is small, but it could have localized impacts particularly regarding enforcement.

• EEZ project template: preprocessors will default to using global-eez-with-land-mr-v3 and then clip to global-land-osm.
• Simple project template: default geography for basic projects will be the world.json polygon
• Template eez projects will have a default geography that uses global-eez-mr-v11 (eez with mr land baseline cut out), because it's available right now in global form. This will create a small misalignment with the osm land baseline used by the preprocessor.

After creating a new report project, developers can then decide which baseline is best:

• if the Marine Regions baseline is more accurate and should be used, then switch to a preprocessor that clips to global-eez-mr-v11. Keep the geography using global-eez-mr-v11.
• if the OSM baseline is accurate enough and fit of clipped shapes to the map is more important, then preprocessor can stay with global-eez-with-land-mr-v3 and a local geography can be created that takes the MR EEZ boundary and subtracts out OSM land, as we have traditionally done for all projects.

Project init

• basic project should have a default geography for the entire world. Datasource should be in data/default/world.json which gets copied in. default geographies.json pointing at world geography should also be used.
• User should optionally select EEZ planning unit type, and select specific EEZ using new global-eez-mr-v11 datasource
• A geographies.json file will be created with eez datasource pointing at global-eez-mr-v11 datasource and user-selected EEZ ID and display value. It should include both a propertyFilter and bboxFilter since this is a global datasource.
• Geography client should be provided for reading, writing, and validating of geographies on disk.

Precalc command

• create precalc:data command
• refactoring project client (getPrecalcMetrics). figure out if breaking change
• Separate precalc from import, drop keyStats
• Default init project with empty geographies.json file for basic-project, or eez geography for ocean-eez project
• Default init project with empty precalc.json file for basic-project, or eez precalc for ocean-eez project
• Support one or many geographies
• Should be self-cleaning and remove precalc metrics that don't match an available datasourceId or geographyId
• Support external geography (loading of geometry)
• Geography display names should be translated, plug them into extraExtraTerms.ts

External datasource precalc

• using new mr-eez external datasource, precalculate bbox and turf_area and provide mr-eez.ts file for reading. Actual eez boundaries will be published at a later date in global-reports project.

Report Client

• refactor SizeCard.tsx to read the precalc area instead of using hardcoded Micronesia value

Geoprocessing Functions

• demonstrate use of default geography in template functions. project.getGeographyById should allows for a caller-defined fallback geography specified with group name project-boundary. This will be default group name for world or eez geography installed on init.

ToDo:

• refactor start-data to use in and out directory. Drop intermediate start-data-testing command
• Add more raster precalc tests
• Drop server start from import and reimport of data since precalc is not being done (used for raster load)
• geojson should not be required for precalc
• geojson should be an optional additional format, flatgeobuf should be required.
• Make sure precalc is set properly for world to false

Future

• basic project should default to world and world geography. User can then create and switch to their own geography when they want to.
• precalc metrics could/should be published, so that external projects can utilize (e.g. publishing global datasources). Building towards offering an endpoint, published via manifest, that serves up precalc metrics. (see Publish precalculated metrics for each datasource global-datasources#4)
• Drop the notion of internal datasource and external datasource. Distinguish between datasource import config and published datasources (see Datasource implementation is complicated #238)

Docs:

• add migration path to CHANGELOG
• What to do if more than one EEZ
• datasource filters are powerful (property) but know that propertyFilters are applied client-side. You want to make sure a bboxFilter is in place that narrows the dataset enough so that it's not sending too much data over the wire. Subdivision can be used on top of that to further reduce the size of datasource polygons.
• Document port use 8001, 8080
• datasources do not get modified, they get clipped at runtime in precalc and in gp function
• When does NaN value occur. This use case
  happens when no features for a data class overlap with a given
  geography. In this scenario a NaN metric value is produced by
  toPercentMetric(), because the denominator value is zero. ClassTable
  picks up on this and displays more information to the user()
• zero geography
• smoke tests run for the default geography. If you add a geographyswitcher to your report client, you will be able to switch geography, and that may show strange behavior because it loads the precalc metrics for the right geography but the sketch metrics are only for the default geography.

Post merge doc updates:

• [] After "What countries EEZ is this for?" question, give link to list of all EEZ's