Complete Python3 update and integrate THREDDS+ORCA

CONTRIBUTING.rst

@@ -134,12 +134,6 @@ pdp/config.env
 |
 | ``NCWMS_URL``
 | Raster portal ncWMS URL of the form ``<docker_host>:<port>/ncWMS/``. The host/port must match ``APP_ROOT``.
-|
-| ``USE_ANALYTICS``
-| Enable or disable Google Analytics reporting (default is ``true``).
-|
-| ``ANALYTICS``
-| Google Analytics ID.
 
 
 docker basics

deploy_requirements.txt

@@ -3,14 +3,14 @@ Babel==2.9.1
 docutils==0.17.1
 gevent==21.1.2
 greenlet==1.1.0
-gunicorn==19.8.1
+gunicorn==19.9.0
 imagesize==1.2.0
 packaging==20.9
 Pygments==2.5.2
 pyparsing==2.4.7
 snowballstemmer==2.1.0
 Sphinx==1.8.5
 sphinxcontrib-websupport==1.1.2
-typing==3.10.0.0
+typing==3.7.4.3
 zope.event==4.5.0
 zope.interface==5.4.0

doc/source/org.rst

@@ -24,18 +24,26 @@ All raster overlay layers are rendered and served by a PCIC-modificiation of the
 Pydap
 -----
 
-Using Pydap for our OPeNDAP backend server has presented us with a variety of opportunities and challenges. On one hand, development of pydap is very modular, dynamic, and open. This has allowed us to easily write custom code to accomplish things that would be otherwise impossible, such as streaming large data responses, having a near-zero memory footprint, and write are own custom data handlers and responses. On the other hand, pydap can be a moving target. Pydap's development repository has lived in three different locations since we started, most of the code base is not rigorously tested (until lately), and API changes have been common. Few of our contributions have been upstreamed, which means that we live in a perpertual state of fear of upgrade. Pydap is mostly a one man show, which mean works-for-me syndrome is common.
+In the past, we used Pydap as our OPeNDAP backend server for all of our data portals, but it is now solely used for the (now deprecated) PCDS portal.
 
-Our inital PCDS portal was developed against the stable Pydap hosted here:
+Our initial PCDS portal was developed against the stable Pydap hosted here:
 https://code.google.com/p/pydap/
 
-Our inital raster portal was developed against the development version of Pydap hosted here:
+Our initial raster portal was developed against the development version of Pydap hosted here:
 https://bitbucket.org/robertodealmeida/pydap
 
 But now he's developing on github with a branch that looks pretty similar to the inital stable version:
 https://github.com/robertodealmeida/pydap
 
-Where to go? Nobody knows. I fear that we may need to maintain our own fork in perpetuity.
+
+THREDDS
+-------
+
+In the latest version of the data portal, we have transitioned from serving our raster data and hydro station data via Pydap to our deployment of the THREDDS Data Server (TDS), which is developed and supported by Unidata, a division of the University Corporation for Atmospheric Research (UCAR). More information about this server can be found here:
+https://www.unidata.ucar.edu/software/tds/current/
+
+Using THREDDS has allowed us to mitigate the challenges associated with maintaining the codebase while using Pydap. Despite this, it comes with its own challenges. Most notably, OPenDAP requests have a size limit of 500 MB. To allow users to request larger datasets, we developed an OPeNDAP Request Compiler Application (ORCA), which recursively bisects initial requests larger than 500 MB, sends those smaller requests to THREDDS, and concatenates the returned data before returning that to the user. More information about this application can be found here:
+https://github.com/pacificclimate/orca
 
 Data Interfaces
 ---------------

doc/source/raster.rst

@@ -153,6 +153,8 @@ Some of the larger datasets have been packed in accordance with the `netCDF stan
 
 The `scale_factor` and `add_offset` values are documented in the metadata of a packed variable.
 
+Please note that in the past, we have offered an additional "ArcInfo/ASCII Grid" format, which consisted of a Zip archive containing one .asc file and one .prj (projection) file representing a map at each timestamp; however, this format is no longer offered as of the latest version of the data portal.
+
 .. _power-user:
 
 Power user HOWTO
@@ -206,7 +208,7 @@ At present, there are eight pages for which one can retrieve catalogs: ``bc_pris
 
 Metadata and Data
 ^^^^^^^^^^^^^^^^^
-All of our multidimensional raster data is made available via `Open-source Project for a Network Data Access Protocol (OPeNDAP) <http://opendap.org/>`_, the specification of which can be found `here <http://www.opendap.org/pdf/ESE-RFC-004v1.2.pdf>`_. Requests are serviced by our deployment of the `Pydap server <http://www.pydap.org/>`_ which PCIC has heavily modified and rewritten to be able to stream large data requests.
+All of our multidimensional raster data is made available via `Open-source Project for a Network Data Access Protocol (OPeNDAP) <http://opendap.org/>`_, the specification of which can be found `here <http://www.opendap.org/pdf/ESE-RFC-004v1.2.pdf>`_. Requests are serviced by our deployment of the `THREDDS server <https://www.unidata.ucar.edu/software/tds/current/>`_ which, when used in conjunction with our OPeNDAP Request Compiler Application (ORCA), allows PCIC to be able to stream large data requests.
 
 The *structure* and *attributes* of a dataset can be retrieved using OPeNDAP by making a `DDS or DAS <http://www.opendap.org/api/pguide-html/pguide_6.html>`_ request respectively. For example, to determine how many timesteps are available from one of the BCSD datasets, one can make a DDS request against that dataset as such: ::
 
@@ -363,61 +365,4 @@ To construct a proper DAP selection, please refer to the `DAP specification <htt
 
 Note that for this example the temperature values are all packed integer values and to obtain the proper value you may need to apply a floating point offset and/or scale factor which are available in the DAS response and the netcdf data response.
 
-Download multiple variables
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-For users that are interested in downloading multiple variables for a single dataset, this *is* possible for certain datasets. The web user interface does not expose this functionality, but if you are willing to do some scripting or URL hacking, you'll be rewarded with a faster download.
-
-To determine whether your dataset of interest contains multiple variables, check by reading the `Dataset Descriptor Structure (DDS) <http://docs.opendap.org/index.php/UserGuideOPeNDAPMessages>`_. You can get this by making a request to the dataset of interest with the ".dds" suffix appended to the end. E.g. the following DDS request shows that the dataset in question contains 3 independent variables (pr, tasmax, tasmin) and 3 axis variables (lon ,lat, time). All of those are requestable in a single request. ::
-
-  james@basalt:~$ curl 'https://data.pacificclimate.org/data/downscaled_gcms_archive/pr+tasmax+tasmin_day_BCCAQ+ANUSPLIN300+MPI-ESM-LR_historical+rcp26_r3i1p1_19500101-21001231.nc.dds'
-  Dataset {
-  Float64 lon[lon = 1068];
-    Float64 lat[lat = 510];
-    Float64 time[time = 55152];
-    Grid {
-        Array:
-            Float32 pr[time = 55152][lat = 510][lon = 1068];
-        Maps:
-            Float64 time[time = 55152];
-            Float64 lat[lat = 510];
-            Float64 lon[lon = 1068];
-    } pr;
-    Grid {
-        Array:
-            Float32 tasmax[time = 55152][lat = 510][lon = 1068];
-        Maps:
-            Float64 time[time = 55152];
-            Float64 lat[lat = 510];
-            Float64 lon[lon = 1068];
-    } tasmax;
-    Grid {
-        Array:
-            Float32 tasmin[time = 55152][lat = 510][lon = 1068];
-        Maps:
-            Float64 time[time = 55152];
-            Float64 lat[lat = 510];
-            Float64 lon[lon = 1068];
-    } tasmin;
-    } pr%2Btasmax%2Btasmin_day_BCCAQ%2BANUSPLIN300%2BMPI-ESM-LR_historical%2Brcp26_r3i1p1_19500101-21001231%2Enc;
-
-To request multiple variables in a single request, you need to use multiple comma separated variable requests in
-the query params. That format looks like this: ::
-
-  [dataset_url].[response_extension]?[variable_name_0][subset_spec],[variable_name_1][subset_spec],...
-
-So if the base dataset that you want to download is
-https://data.pacificclimate.org/data/downscaled_gcms_archive/pr+tasmax+tasmin_day_BCCAQ+ANUSPLIN300+MPI-ESM-LR_historical+rcp26_r3i1p1_19500101-21001231.nc,
-and you want to download the NetCDF response, so your extension will
-be '.nc'.
-
-Assume you just want the first 100 timesteps ([0:99]) and a 50x50
-square somewhere in the middle ([250:299][500:549]).
-
-Putting that all together, it will look something like this: ::
-
-  https://data.pacificclimate.org/data/downscaled_gcms_archive/pr+tasmax+tasmin_day_BCCAQ+ANUSPLIN300+MPI-ESM-LR_historical+rcp26_r3i1p1_19500101-21001231.nc.nc?tasmax[0:99][250:299][500:549],tasmin[0:99][250:299][500:549],pr[0:99][250:299][500:549]
-
-It's not quite as easy as clicking a few buttons on the web page, but
-depending on your use case, you can evaluate whether it's worth your
-effort to script together these multi-variable requests.
+Please note that in the past, we have allowed users to download multiple variables for a single dataset using a single request; however, this functionality is no longer supported as of the latest version of the data portal.

docker/ci/Dockerfile

@@ -1,7 +1,8 @@
-FROM pcic/pdp-base-minimal-unsafe:1.0.0
+# TODO: Replace pdp-base-minimal tag with new release tag when this branch of pdp-docker has been merged
+FROM pcic/pdp-base-minimal-unsafe:pdp-python3
 
 COPY ./ ${USER_DIR}
 
-RUN python -m pip install -r requirements.txt -r test_requirements.txt
-RUN python -m pip install sphinx
-RUN python -m pip install .
+RUN python3 -m pip install -r requirements.txt -r test_requirements.txt
+RUN python3 -m pip install sphinx==1.8.5
+RUN python3 -m pip install .

docker/local-run/Dockerfile

@@ -1,4 +1,5 @@
-FROM pcic/pdp-base-minimal:1.0.0
+# TODO: Replace pdp-base-minimal tag with new release tag when this branch of pdp-docker has been merged
+FROM pcic/pdp-base-minimal:pdp-python3
 LABEL Maintainer="Rod Glover <[email protected]>"
 
 USER root
@@ -16,6 +17,6 @@ USER ${USERNAME}
 WORKDIR /codebase
 ADD *requirements.txt /codebase/
 
-RUN pip install -r requirements.txt -r test_requirements.txt -r deploy_requirements.txt
+RUN pip3 install -r requirements.txt -r test_requirements.txt -r deploy_requirements.txt
 
-ENTRYPOINT ./docker/local-run/entrypoint.sh
+ENTRYPOINT ./docker/local-run/entrypoint.sh

docker/local-run/common.env

@@ -1,6 +1,6 @@
 # environment variables common to fe and be
 
-DSN=postgresql://httpd_meta:XXXXXX@pgbouncer-dev:5432/pcic_meta_test
+DSN=postgresql://httpd_meta:XXXXXX@pgbouncer-dev:5432/pcic_meta
 PCDS_DSN=postgresql://httpd:XXXXXX@pgbouncer-dev:5432/crmp
 
 GUNICORN_BIND=0.0.0.0:8000
@@ -11,5 +11,3 @@ GUNICORN_TIMEOUT=86400
 USE_AUTH=False
 SESSION_DIR=default
 CLEAN_SESSION_DIR=True
-USE_ANALYTICS=True
-ANALYTICS=UA-20166041-2

docker/local-run/docker-compose.yaml

@@ -54,6 +54,8 @@ services:
       - common-with-passwords.env
     environment:
       # Specific to backend
+      - ORCA_ROOT=http://pdp.localhost:5000/orca
+      - THREDDS_ROOT=https://marble-dev01.pcic.uvic.ca/twitcher/ows/proxy/thredds/dodsC/datasets
       - APP_MODULE=pdp.wsgi:backend
       - GUNICORN_WORKER_CLASS=gevent
     entrypoint: /codebase/docker/local-run/entrypoint-be.sh

docker/local-run/entrypoint-be.sh

@@ -12,6 +12,6 @@
 #pip install -e /home/rglover/code/pdp_util
 
 # *Always* do this. It's the whole point of this Docker setup.
-pip install -e .
+pip3 install -e .
 
 gunicorn --config docker/local-run/gunicorn.conf --log-config docker/local-run/logging.conf pdp.wsgi:backend

docker/local-run/entrypoint-fe.sh

@@ -12,6 +12,6 @@
 #pip install -e /home/rglover/code/pdp_util
 
 # *Always* do this. It's the whole point of this Docker setup.
-pip install -e .
+pip3 install -e .
 
 gunicorn --config docker/local-run/gunicorn.conf --log-config docker/local-run/logging.conf pdp.wsgi:frontend

docker/local-run/entrypoint.sh

@@ -11,6 +11,6 @@
 #pip install -e /home/rglover/code/pdp_util
 
 # *Always* do this. It's the whole point of this Docker setup.
-pip install -e .
+pip3 install -e .
 
 /bin/bash

docker/local-run/nginx.conf

@@ -30,5 +30,13 @@ http {
             proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header   X-Forwarded-Host $server_name;
         }
+
+        location /orca/ {
+            proxy_pass https://services.pacificclimate.org/dev/orca/;
+            proxy_set_header   Host $host;
+            proxy_set_header   X-Real-IP $remote_addr;
+            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header   X-Forwarded-Host $server_name;
+        }
     }
-}
+}

docker/local-test/Dockerfile

@@ -1,4 +1,5 @@
-FROM pcic/pdp-base-minimal:1.0.0
+# TODO: Replace pdp-base-minimal tag with new release tag when this branch of pdp-docker has been merged
+FROM pcic/pdp-base-minimal:pdp-python3
 
 USER root
 # TODO: Move into pdp-base-minimal?
@@ -8,4 +9,4 @@ USER ${USERNAME}
 # You must mount the local codebase to /codebase
 WORKDIR /codebase
 
-ENTRYPOINT ./docker/local-test/entrypoint.sh
+ENTRYPOINT ./docker/local-test/entrypoint.sh

docker/local-test/env.env

@@ -1,2 +1,4 @@
 DSN=postgresql://httpd_meta:[email protected]:5432/pcic_meta
 PCDS_DSN=postgresql://httpd:[email protected]:5432/crmp
+ORCA_ROOT=https://services.pacificclimate.org/dev/orca
+THREDDS_ROOT=https://marble-dev01.pcic.uvic.ca/twitcher/ows/proxy/thredds/dodsC/datasets

docker/production/Dockerfile

@@ -2,7 +2,8 @@
 # Dockerfile to run the PCIC data portal   #
 ############################################
 
-FROM pcic/pdp-base-minimal:1.0.0
+# TODO: Replace pdp-base-minimal tag with new release tag when this branch of pdp-docker has been merged
+FROM pcic/pdp-base-minimal:pdp-python3
 LABEL Maintainer="James Hiebert <[email protected]>"
 
 USER root
@@ -21,14 +22,14 @@ ADD --chown=${USERNAME}:${GROUPNAME} . ${USER_DIR}/
 
 # Install dependencies. Note: Base image already contains several of the
 # heaviest ones.
-RUN pip install -r requirements.txt -r deploy_requirements.txt
+RUN pip3 install -r requirements.txt -r deploy_requirements.txt
 
 # Install and build the docs
 # Must pre-install to provide dependencies and version number
 # for build_spinx
-RUN pip install .
-RUN python setup.py build_sphinx
-RUN pip install .
+RUN pip3 install .
+RUN python3 setup.py build_sphinx
+RUN pip3 install .
 
 # gunicorn.conf is set up so that one can tune gunicorn settings when
 # running the container by setting environment an variable

docker/production/common.env

@@ -11,5 +11,3 @@ GUNICORN_TIMEOUT=86400
 USE_AUTH=False
 SESSION_DIR=default
 CLEAN_SESSION_DIR=True
-USE_ANALYTICS=True
-ANALYTICS=UA-20166041-2

docker/production/docker-compose-local.yaml

@@ -51,6 +51,8 @@ services:
       - common-with-passwords.env
     environment:
       # Specific to backend
+      - ORCA_ROOT=http://pdp.localhost:5000/orca
+      - THREDDS_ROOT=https://marble-dev01.pcic.uvic.ca/twitcher/ows/proxy/thredds/dodsC/datasets
       - APP_MODULE=pdp.wsgi:backend
       - GUNICORN_WORKER_CLASS=gevent
     volumes:

docker/production/docker-compose.yaml

@@ -32,6 +32,8 @@ services:
       - common-with-passwords.env
     environment:
       # Specific to backend
+      - ORCA_ROOT=https://services.pacificclimate.org/orca
+      - THREDDS_ROOT=https://marble-dev01.pcic.uvic.ca/twitcher/ows/proxy/thredds/dodsC/datasets
       - APP_MODULE=pdp.wsgi:backend
       - GUNICORN_WORKER_CLASS=gevent
     ports:

docs/deployment.md

@@ -65,8 +65,6 @@ They are loaded from the environment variables of the same name, upper cased.
 | `old_ncwms_url` | Raster portal pure ncWMS 1.x URL. Used to fill in missing services from ncWMS 2.x. |
 | `na_tiles_url` | MapProxy URL for serving North America base maps |
 | `bc_basemap_url` | Tile server URLs  (space separated list) for BC base maps
-| `use_analytics` | Enable or disable Google Analytics reporting |
-| `analytics` | Google Analytics ID |
 
 ### Gunicorn configuration
 
@@ -432,4 +430,4 @@ When upgrading, it's easiest to simply copy the existing config and update the p
 
 Using `supervisorctl`, you should then be able to `reread` the new config, `update` the old version config (so it stops, picks up new autostart/autorestart=false), and `update` the new version.
 
-If there are any errors, they can be found in the `supervisord_logfile`. Errors starting gunicorn can be found in the `error_logfile`.
+If there are any errors, they can be found in the `supervisord_logfile`. Errors starting gunicorn can be found in the `error_logfile`.

docs/installation.md

@@ -39,15 +39,15 @@ procedures in our Docker infrastructure mentioned just above.
 
 ***These instructions have not been tested on a workstation.*** YMMV.
 
-1. We're assuming you have Python 2.7 installed. If not, install it.
+1. We're assuming you have Python 3.8 installed. If not, install it.
 
 1. Install system-level dependencies.
 
    ```
    apt-get install libhdf5-dev libgdal-dev libnetcdf-dev
    ```
 
-1. Create a Python 2.7 virtual environment and activate it.
+1. Create a Python 3.8 virtual environment and activate it.
 
 1. Install Python build packages.
 
@@ -66,8 +66,8 @@ procedures in our Docker infrastructure mentioned just above.
 1. Install Python dependencies (separate install for GDAL is required).
 
    ```
-   pip install --no-binary :all: numpy==1.16.6 Cython==0.22 gdal==2.2
-   pip install --no-binary :all: h5py==2.7.1
+   pip install --no-binary :all: numpy==1.16.6
+   pip install --no-binary :all: h5py==2.7.1 gdal==3.0.4
    pip install -r requirements.txt -r test_requirements.txt -r deploy_requirements.txt
    pip install -e .
    ```