Complete Python3 update and integrate THREDDS+ORCA #282

eyvorchuk · 2023-03-28T17:37:14Z

This PR completes the migration of the Python code from Python2 to Python3, and it directs requests made in the raster and hydro station data portals to the OPeNDAP Request Compiler Application (orca), which redirects these requests to the THREDDS server. This means we no longer have to use PyDAP for these portals. This PR makes use of code from the generate-metadata branch in pdp_util and the metadata-requests branch in orca.

Resolves #212.

…pose.yaml

…appropriate

…es from ORCA urls

…docker)

corviday

Looks good to me! The last thing I want to try before Eric leaves is getting it up and running on my workstation, so he can help me out if I have trouble.

corviday · 2024-06-27T01:25:10Z

docker/ci/Dockerfile

@@ -1,7 +1,7 @@
-FROM pcic/pdp-base-minimal-unsafe:1.0.0
+FROM pcic/pdp-base-minimal-unsafe:pdp-python3


what does "unsafe" mean here?

See the pdp-docker project for documentation on that

corviday · 2024-06-27T01:27:55Z

tests/test_functional.py

 ])
 def test_climo_index(
        pcic_data_portal, url, title, body_strings):
    req = Request.blank(url)
    resp = req.get_response(pcic_data_portal)
    assert resp.status == '200 OK'
    assert resp.content_type == 'text/html'
-    assert resp.content_length < 0
+    #assert resp.content_length < 0


what's the deal with the previous version of this line? why would content length have been less than zero at any point? Is this leftover from an intensive debugging process (possibly before you took over)?

I have no idea what this is. I can get rid of it.

corviday · 2024-06-27T01:34:07Z

doc/source/org.rst

@@ -24,18 +24,26 @@ All raster overlay layers are rendered and served by a PCIC-modificiation of the
 Pydap
 -----

-Using Pydap for our OPeNDAP backend server has presented us with a variety of opportunities and challenges. On one hand, development of pydap is very modular, dynamic, and open. This has allowed us to easily write custom code to accomplish things that would be otherwise impossible, such as streaming large data responses, having a near-zero memory footprint, and write are own custom data handlers and responses. On the other hand, pydap can be a moving target. Pydap's development repository has lived in three different locations since we started, most of the code base is not rigorously tested (until lately), and API changes have been common. Few of our contributions have been upstreamed, which means that we live in a perpertual state of fear of upgrade. Pydap is mostly a one man show, which mean works-for-me syndrome is common.
+In the past, we used Pydap as our OPeNDAP backend server for all of our data portals, but it is now solely used for the (now deprecated) PCDS portal. Using Pydap has presented us with a variety of opportunities and challenges. On one hand, development of pydap is very modular, dynamic, and open. This has allowed us to easily write custom code to accomplish things that would be otherwise impossible, such as streaming large data responses, having a near-zero memory footprint, and write our own custom data handlers and responses. On the other hand, pydap can be a moving target. Pydap's development repository has lived in three different locations since we started, most of the code base is not rigorously tested (until lately), and API changes have been common. Few of our contributions have been upstreamed, which means that we live in a perpertual state of fear of upgrade. Pydap is mostly a one man show, which mean works-for-me syndrome is common.


Could probably shorten this paragraph to just the first sentence or so, since we're not using pydap any more.

corviday · 2024-06-27T01:34:21Z

doc/source/org.rst

+https://www.unidata.ucar.edu/software/tds/current/
+
+Using THREDDS has allowed us to mitigate the challenges associated with maintaining the codebase while using Pydap. Despite this, it comes with its own challenges. Most notably, OPenDAP requests have a size limit of 500 MB. To allow users to request larger datasets, we developed an OPeNDAP Request Compiler Application (ORCA), which recursively bisects initial requests larger than 500 MB, sends those smaller requests to THREDDS, and concatenates the returned data before returning that to the user. More information about this application can be found here:
+https://github.com/pacificclimate/orca


good to have this in here.

corviday · 2024-06-27T01:39:55Z

doc/source/raster.rst

@@ -206,7 +208,7 @@ At present, there are eight pages for which one can retrieve catalogs: ``bc_pris

 Metadata and Data
 ^^^^^^^^^^^^^^^^^
-All of our multidimensional raster data is made available via `Open-source Project for a Network Data Access Protocol (OPeNDAP) <http://opendap.org/>`_, the specification of which can be found `here <http://www.opendap.org/pdf/ESE-RFC-004v1.2.pdf>`_. Requests are serviced by our deployment of the `Pydap server <http://www.pydap.org/>`_ which PCIC has heavily modified and rewritten to be able to stream large data requests.
+All of our multidimensional raster data is made available via `Open-source Project for a Network Data Access Protocol (OPeNDAP) <http://opendap.org/>`_, the specification of which can be found `here <http://www.opendap.org/pdf/ESE-RFC-004v1.2.pdf>`_. Requests are serviced by our deployment of the `THREDDS server <https://www.unidata.ucar.edu/software/tds/current/>`_ which, when used in conjunction with our OPeNDAP Request Compiler Application (ORCA), allows PCIC to be able to stream large data requests.


Does a user downloading data with a script need to combine multiple requests themself (and it needs to be documented here?), or does ORCA do it for them? I think ORCA does it for them, but wanted to check...

ORCA will do it for them.

Assuming said script amounts to something like wget https://data.pacificclimate.org/data/<portal>/<filename>?<varname>[time_bounds][lat_bounds][lon_bounds].

corviday · 2024-06-27T01:41:57Z

pdp/error.py

+                # * strings - sometimes sent by backend API calls
+                # * Response objects - not yet clear who is sending these
+                # TODO: get everyone to send expected generators,
+                # remove this failsafe code.  


Hopefully this comment isn't true now that everything has been fixed. Update?

corviday · 2024-06-27T01:42:36Z

pdp/error.py

+                        response_headers = [
+                            ("content-type", "text/plain"),
+                            ("Location", response_iter.location),
+                            ("Access-Control-Allow-Origin", "*")


(looks like this has been fixed)

corviday · 2024-06-27T01:58:56Z

Played with downloading things and looking at DAS and DDS calls from the demo, that seemed to go well.

rod-glover

Just a few small suggestions and a couple of questions.

rod-glover · 2024-06-28T19:03:11Z

doc/source/org.rst

@@ -24,18 +24,26 @@ All raster overlay layers are rendered and served by a PCIC-modificiation of the
 Pydap
 -----

-Using Pydap for our OPeNDAP backend server has presented us with a variety of opportunities and challenges. On one hand, development of pydap is very modular, dynamic, and open. This has allowed us to easily write custom code to accomplish things that would be otherwise impossible, such as streaming large data responses, having a near-zero memory footprint, and write are own custom data handlers and responses. On the other hand, pydap can be a moving target. Pydap's development repository has lived in three different locations since we started, most of the code base is not rigorously tested (until lately), and API changes have been common. Few of our contributions have been upstreamed, which means that we live in a perpertual state of fear of upgrade. Pydap is mostly a one man show, which mean works-for-me syndrome is common.
+In the past, we used Pydap as our OPeNDAP backend server for all of our data portals, but it is now solely used for the (now deprecated) PCDS portal. Using Pydap has presented us with a variety of opportunities and challenges. On one hand, development of pydap is very modular, dynamic, and open. This has allowed us to easily write custom code to accomplish things that would be otherwise impossible, such as streaming large data responses, having a near-zero memory footprint, and write our own custom data handlers and responses. On the other hand, pydap can be a moving target. Pydap's development repository has lived in three different locations since we started, most of the code base is not rigorously tested (until lately), and API changes have been common. Few of our contributions have been upstreamed, which means that we live in a perpertual state of fear of upgrade. Pydap is mostly a one man show, which mean works-for-me syndrome is common.


rod-glover · 2024-06-28T19:12:48Z

docker/ci/Dockerfile

@@ -1,7 +1,7 @@
-FROM pcic/pdp-base-minimal-unsafe:1.0.0
+FROM pcic/pdp-base-minimal-unsafe:pdp-python3


Add a comment that this should be replaced by a release tag when that branch has been merged.

docker/local-run/Dockerfile

docker/production/Dockerfile

rod-glover · 2024-06-28T19:21:23Z

pdp/static/js/hydro_stn_controls.js

@@ -87,7 +87,7 @@ function addToSidebar(idx, dataArray) {
    });

    link = document.createElement('a');
-    link.href = pdp.data_root + "/hydro_stn_cmip5/" + dataArray[idx].FileName + '.ascii';
+    link.href = `${pdp.data_root}/hydro_stn_cmip5/${dataArray[idx].FileName}`;


Why did we lose .ascii here?

It's because CSV files from THREDDS don't support the .ascii extension. So, they're just downloaded as filename.csv instead of filename.csv.csv or filename.csv.ascii.

…e it's created

corviday · 2024-09-05T19:49:31Z

I have deployed this branch to beehive, backended by ORCA and THREDDS, and everything seems to be working. Maybe needs a bit more testing, then merge and release the new version.

corviday and others added 30 commits September 15, 2021 17:10

Start update to python 3

5dfbe9b

Update dockers to run python3

ab3ca52

Upgrade gunicorn to version 19.9.0

a32d9bd

Remove uses of Google Analytics client

a175295

Proxy Pass THREDDS URL and set ORCA URL configs

80c521c

Merge branch 'master' of https://github.com/pacificclimate/pdp into py3

5fbe711

Proxy pass orca URL in nginx.conf

51f9084

Handle .ascii requests from THREDDS

b96ee56

Merge branch 'master' of https://github.com/pacificclimate/pdp into py3

dfc5b24

Decrement end bounds for lon/lat in each raster portal app

727499a

Remove reference to use_analytics in conftest.py

98d9ddf

Set THREDDS_ROOT and ORCA_ROOT as environment variables in docker-com…

2663e70

…pose.yaml

Use new database DSNs

a1a14d9

Remove crmpdb mark from incorrectly marked tests

cdf110f

Only target climatology_bnds in climatological averages datasets

6fe24a1

Merge branch 'master' of https://github.com/pacificclimate/pdp into py3

2c0017f

Merge branch 'master' of https://github.com/pacificclimate/pdp into py3

cb8d42d

Use pdp.portals in import statements in main.py

5eb84ee

Merge branch 'master' of https://github.com/pacificclimate/pdp into py3

0c024b3

Remove THREDDS url from nginx.conf and docker-compose.yaml

7a3eaa9

Use python3 image and add ORCA_ROOT to local test build

b490a98

Remove end slashes from default THREDDS and ORCA urls

0e199df

Remove /catalog/ from metadata_url

0929d4b

Decrement upper lon/lat bounds in canesm5 app

18caba6

Set climatology bounds in url template for BC prism portal only when …

877027c

…appropriate

Replace PyDAP server for Hydro Station portals with redirects to ORCA

634f472

Update automated tests to handle ORCA redirects and obtaining respons…

10049bc

…es from ORCA urls

Update package-lock.json after having fun npm install

f42b02c

Update files in docker/production

92d5a47

Update Dockerfile for CI to use Python3

ddc3b97

eyvorchuk added 17 commits June 24, 2024 13:19

Pin numpy version (normally gets upgraded even when pre-installed in …

44d43b6

…docker)

Fix tests

ce23940

Fix typo in doc

ddb5b9d

Get storage root of hydro station data from yaml

922c427

Remove print statements from error.py

d856af2

Remove commented prints from test_functional.py

b0ee0cb

Fix typo and remove sentence in doc

c243954

Clean up js code

98dabe6

Rename is_valid_orca functions to assert_is_valid_orca

9c25441

Remove print in test

6557fad

Yield strings throughout error.py

0670787

Replace default THREDDS host

175e5ab

Add THREDDS_ROOT to production docker-compose files

e74f36a

Remove config property from hydro_stn_cmip5

68a0045

Import yaml in test_functional.py

5acf408

Yield bytes instead of strings

c2f0c37

Use let instead of const for looping variable

e84dfe9

eyvorchuk requested review from corviday and rod-glover June 26, 2024 19:08

corviday approved these changes Jun 27, 2024

View reviewed changes

rod-glover approved these changes Jun 28, 2024

View reviewed changes

eyvorchuk added 4 commits June 28, 2024 12:48

Remove comment about non-iterable errors

df55cce

Remove most of Pydap paragraph

9d72ffe

Add comment about replacing pdp-base-minimal tag with new release onc…

56ea346

…e it's created

Move Dockerfile comments to own line

b818f9e

corviday mentioned this pull request Aug 24, 2024

Update local test setup to python 3 #258

Closed

corviday added 2 commits August 23, 2024 20:08

update to new pdp_util release

4821f09

Merge branch 'master' into py3

1f8dd16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Complete Python3 update and integrate THREDDS+ORCA #282

Complete Python3 update and integrate THREDDS+ORCA #282

eyvorchuk commented Mar 28, 2023 •

edited

Loading

corviday left a comment

corviday Jun 27, 2024

rod-glover Jun 28, 2024

corviday Jun 27, 2024

eyvorchuk Jun 28, 2024

corviday Jun 27, 2024

rod-glover Jun 28, 2024

corviday Jun 27, 2024

corviday Jun 27, 2024

eyvorchuk Jun 28, 2024

eyvorchuk Jun 28, 2024

corviday Jun 27, 2024

corviday Jun 27, 2024

corviday commented Jun 27, 2024

rod-glover left a comment

rod-glover Jun 28, 2024

rod-glover Jun 28, 2024

rod-glover Jun 28, 2024

eyvorchuk Jun 28, 2024

corviday commented Sep 5, 2024 •

edited

Loading

		@@ -1,7 +1,7 @@
		FROM pcic/pdp-base-minimal-unsafe:1.0.0
		FROM pcic/pdp-base-minimal-unsafe:pdp-python3

Complete Python3 update and integrate THREDDS+ORCA #282

Are you sure you want to change the base?

Complete Python3 update and integrate THREDDS+ORCA #282

Conversation

eyvorchuk commented Mar 28, 2023 • edited Loading

corviday left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corviday commented Jun 27, 2024

rod-glover left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corviday commented Sep 5, 2024 • edited Loading

eyvorchuk commented Mar 28, 2023 •

edited

Loading

corviday commented Sep 5, 2024 •

edited

Loading