Process pointinspection and Documentation #271

nilshempelmann · 2018-12-06T19:03:17Z

@cehbrecht I need help with the workdir settings. If I get it right it needs to be updated in all the eggshell functions as well, since there are many files produced there.

@huard @Zeitsperre If you have time check the Subset Process Documentation.

nilshempelmann · 2018-12-07T10:22:22Z

missed to commit changes in eggshell

flyingpigeon/processes/wps_pointinspection.py

cehbrecht · 2018-12-07T16:35:52Z

travis and codacy are not happy yet. You could also add a test.

cehbrecht · 2018-12-07T16:41:08Z

I need help with the workdir settings. If I get it right it needs to be updated in all the eggshell functions as well, since there are many files produced there.

I haven't checked the eggshell ... but it should provide supporting functions not knowing about self.workdir handling ... that is pywps only. So, you need to find the right set of function parameters.

huard

My suggestion would be for all processes to return netCDF files. This way we're able to use process outputs as inputs to other processes.

We can then have a process to convert netCDF to CSV.

huard · 2018-12-07T19:50:25Z

docs/source/descriptions/subset.rst

+**Polygons**
+  Abbreviation of the appropriate polygon.
+
+**Mosaic**


Confusing, since PointInspection has no mosaic argument.

nilshempelmann · 2018-12-08T16:46:13Z

@huard agree with netcdf outputs. How to configure the output if the number of outputs can vary? Can you point me to an example?

huard · 2018-12-10T13:23:57Z

@nilshempelmann Do you mean if the number of geometries is larger than one, or if the number of input files is larger than one ? I'll assume it's the former, and then OCGIS supports it out of the box, you just pass a list of geometries to OCGIS and make sure you specify aggregate=True. The idea is to create another dimension for the output where the index is the gemetry index. This allows you to store the lat, lon as well along this dimension.
NCPP/ocgis#459
https://ocgis.readthedocs.io/en/latest/appendix.html?highlight=geom_dim#netcdf-output

nilshempelmann · 2018-12-10T17:26:41Z

@huard The number of netCDF output files vary if input files are model results from different GCMs-RCMs. Input files can be belonging to different modelruns, the process are sorting them and process all different models separatly eggshell.nc.nc_utils.sort_by_filename(resource, historical_concatination=False)
(mulitiple files of the same Modelrun are currently aggregated, including also aggregation of historical and suitable rcp run of the same Model output). Its not recommendet to store multiple models into one netcdf file.

And yes multiple geometries / optional unify results in variation of the number of outputs. So the current solution was to archive all of them and output one singe archive.

We also should keep the DRS-nameing logig and respect the CMIP/Cordex archive standarts. Here they separate the geometries/domains in seperate files. eggshell.nc.nc_utils.drs_filename()

A following process (e.g. visualisation) starts with archive_extract, in case of input=tarfile.

If you have a solution to avoid the archive-tar would be nice.

huard · 2018-12-10T17:46:57Z

I understand now. You are saying the the current output is a tar file, which allows us to bundle multiple output files together in one single ComplexOutput.

I think now is the time to implement a generic approach for multiple file output using the MetaLink standard.

See geopython/pywps#298
bird-house/emu#64 (comment)

I suggest we first test and discuss it in Emu then move the solution to this process.

nilshempelmann · 2018-12-10T17:51:34Z

And here comes the issue #273 :-)

nilshempelmann · 2018-12-10T20:15:12Z

Following processes are sucessfully running, but test are failing (related bird-house/emu#70):
Subset_countries
subset_continets
pointinspection

@huard need your help to get the oranus subset processes in place. #274

nilshempelmann · 2018-12-12T12:27:40Z

@cehbrecht Finally :-)
But subset countries and continents output path needs to be fixed in the tests (disabled for the moment)

cehbrecht

short review ... not tested yet. Added comments to the code.

tests/test_eggshell.py

cehbrecht · 2018-12-12T17:28:27Z

tests/test_wps_subset.py

+        while execution.status == 'ProcessAccepted':
+            execution.checkStatus(sleepSecs=1)
+        output_json = execution.processOutputs[0].reference
+        if output_json[:7] == 'file://':


why is if needed? In a test you have a fixed expectation.

@huard That's an Ouranos process. I leave it to you to change it correctly

cehbrecht · 2018-12-12T17:29:58Z

tests/test_wps_subset.py

+        self.assertEqual(nc.subset_featureid, 'montreal_circles.43')
+        nc.close()
+
+    def test_subset_wfs_opendap_01_default_geoserver(self):


This test (and others) look quite complicated. Can it be changed? Test code should be simple, otherwise it is to likely to break as well.

@huard That's an Ouranos process. I leave it to you to change it correctly

cehbrecht · 2018-12-12T17:31:28Z

tests/test_wps_subset_WFS.py

+
+try:
+    from tests import test_wps_utils
+except ImportError:


why is the try needed?

@huard That's an Ouranos process. I leave it to you to change it correctly

docs/source/descriptions/subset.rst

docs/source/descriptions/shapefilepreparation.rst

tests/test_eggshell.py

cehbrecht · 2018-12-12T17:42:18Z

tests/test_wps_subset.py

+        self.config = configparser.RawConfigParser()
+        if os.path.isfile('configtests.cfg'):
+            self.config.read('configtests.cfg')
+        else:


@huard That's an Ouranos process. I leave it to you to change it correctly

flyingpigeon/processes/wps_pointinspection.py

nilshempelmann · 2018-12-14T15:59:03Z

@cehbrecht : commented your change request.
Feel free to update the Ouranos processes

nilshempelmann added 6 commits December 5, 2018 23:04

switch on pointinspection

6d3d9d3

test passed

c9fe100

disable python 2.7 in travis.yml

f00e624

update documentation

056527c

include user guide template

bfca0dd

include subset description

fa1666b

nilshempelmann requested a review from cehbrecht December 6, 2018 19:03

nilshempelmann added the help wanted label Dec 7, 2018

nilshempelmann mentioned this pull request Dec 7, 2018

only support Python 3.6? #270

Closed

cehbrecht reviewed Dec 7, 2018

View reviewed changes

flyingpigeon/processes/wps_pointinspection.py Outdated Show resolved Hide resolved

nilshempelmann and others added 2 commits December 7, 2018 19:16

custom.cgf

20a2ffe

aesthetic changes

bc93564

huard reviewed Dec 7, 2018

View reviewed changes

nilshempelmann added 4 commits December 8, 2018 11:45

add tests and testdata

e204cf5

update common.py

b400046

enable and update tests

4a4e7ce

eggshell tests passed

5d6af2b

nilshempelmann added 2 commits December 8, 2018 19:35

tests subset WFS enabled

7345bb8

updated subset processes

c5e4fef

nilshempelmann added 2 commits December 10, 2018 19:10

change indentification abstract

5c1f1ae

change default.cfg

387574c

nilshempelmann added 5 commits December 11, 2018 18:14

update test_eggshell

b8fbc4d

guess_main_variables to get_variable

9fd5f41

adopt tests

2927fa4

switch off country continent tests

d4a91f3

pep8

a923a72

update subset tests

df6fa6d

cehbrecht requested changes Dec 12, 2018

View reviewed changes

nilshempelmann added 3 commits December 13, 2018 13:31

move eggshell tests

3d4407a

update documentation

5173999

workdir as outdir

450ad03

update README

1abe124

cehbrecht approved these changes Dec 14, 2018

View reviewed changes

nilshempelmann merged commit 7395296 into master Dec 15, 2018

cehbrecht added this to the 1.5 milestone May 17, 2019

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Process pointinspection and Documentation #271

Process pointinspection and Documentation #271

nilshempelmann commented Dec 6, 2018

nilshempelmann commented Dec 7, 2018

cehbrecht commented Dec 7, 2018

cehbrecht commented Dec 7, 2018

huard left a comment

huard Dec 7, 2018

nilshempelmann commented Dec 8, 2018

huard commented Dec 10, 2018

nilshempelmann commented Dec 10, 2018 •

edited

Loading

huard commented Dec 10, 2018

nilshempelmann commented Dec 10, 2018

nilshempelmann commented Dec 10, 2018 •

edited

Loading

nilshempelmann commented Dec 12, 2018

cehbrecht left a comment

cehbrecht Dec 12, 2018

nilshempelmann Dec 13, 2018 •

edited

Loading

cehbrecht Dec 12, 2018

nilshempelmann Dec 13, 2018

cehbrecht Dec 12, 2018

nilshempelmann Dec 13, 2018

cehbrecht Dec 12, 2018

nilshempelmann Dec 14, 2018

nilshempelmann commented Dec 14, 2018

Process pointinspection and Documentation #271

Process pointinspection and Documentation #271

Conversation

nilshempelmann commented Dec 6, 2018

nilshempelmann commented Dec 7, 2018

cehbrecht commented Dec 7, 2018

cehbrecht commented Dec 7, 2018

huard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilshempelmann commented Dec 8, 2018

huard commented Dec 10, 2018

nilshempelmann commented Dec 10, 2018 • edited Loading

huard commented Dec 10, 2018

nilshempelmann commented Dec 10, 2018

nilshempelmann commented Dec 10, 2018 • edited Loading

nilshempelmann commented Dec 12, 2018

cehbrecht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilshempelmann Dec 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilshempelmann commented Dec 14, 2018

nilshempelmann commented Dec 10, 2018 •

edited

Loading

nilshempelmann commented Dec 10, 2018 •

edited

Loading

nilshempelmann Dec 13, 2018 •

edited

Loading