Merge pull request #419 from crim-ca/output-ref

.dockerignore

@@ -18,6 +18,11 @@ env
 package-lock.json
 node_modules
 
+## Celery
+celeryconfig*
+celery-config*
+celerybeat-schedule.*
+
 ## Python / Extensions etc.
 *~
 *.mo

.gitignore

@@ -16,6 +16,11 @@ node_modules
 ## Docker
 #Dockerfile
 
+## Celery
+celeryconfig*
+celery-config*
+celerybeat-schedule.*
+
 ## Python / Extensions etc.
 *~
 *.mo

CHANGES.rst

@@ -10,8 +10,41 @@ Changes
 `Unreleased <https://github.com/crim-ca/weaver/tree/master>`_ (latest)
 ========================================================================
 
+Important:
+----------
+- In order to support *synchronous* execution, setting ``RESULT_BACKEND`` **MUST** be specified in
+  the ``weaver.ini`` configuration file.
+  See `Weaver INI Configuration Example <https://github.com/crim-ca/weaver/blob/master/config/weaver.ini.example>`_
+  in section ``[celery]`` for more details.
+- With resolution and added support of ``transmissionMode`` handling according to `OGC API - Processes` specification,
+  requests that where submitted with ``reference`` outputs will produce results in a different format than previously
+  since this parameter was ignored and always returned ``value`` representation.
+
 Changes:
 --------
+- Support ``Prefer`` header with ``wait`` or ``respond-async`` directives to select ``Job`` execution mode either
+  as *synchronous* or *asynchronous* task, according to supported ``jobControlOptions`` of the relevant ``Process``
+  being executed (resolves `#247 <https://github.com/crim-ca/weaver/issues/247>`_).
+- Increase minor version of all ``builtin`` processes that will now be executable in wither (a)synchronous modes.
+- Add ``weaver.exec_sync_max_wait`` and ``weaver.quote_sync_max_wait`` settings allowing custom definition for the
+  maximum duration that can be specified to wait for a `synchronous` response from task workers.
+- Add ``-B`` (``celery beat``) option to Docker command of ``weaver-worker`` to run scheduled task in parallel
+  to ``celery worker`` in order to periodically cleanup task results introduced by *synchronous* execution.
+- Add support of ``transmissionMode`` handling as ``reference`` to generate HTTP ``Link`` references for results
+  requested this way (resolves `#377 <https://github.com/crim-ca/weaver/issues/377>`_).
+- Updated every ``Process`` to report that they support ``outputTransmission`` both as ``reference`` and ``value``,
+  since handling of results is accomplished by `Weaver` itself, regardless of the application being executed.
+- Add partial support of ``response=raw`` parameter for execution request submission in order to handle results to
+  be returned accordingly to specified ``outputTransmission`` by ``reference`` or ``value``.
+  Multipart contents for multi-output results are not yet supported
+  (relates to `#376 <https://github.com/crim-ca/weaver/issues/376>`_).
+- Add `CLI` option ``-R/--ref/--reference`` for ``execute`` operation allowing to request corresponding ``outputs``
+  by ID to be returned using the ``transmissionMode: reference`` method, producing HTTP ``Link`` headers for those
+  entries rather than inserting values in the response content body.
+- Add requested ``outputs`` into response of ``GET /jobs/{jobId}/inputs`` to obtain submitted ``Job`` definitions.
+- Add query parameter ``schema`` for ``GET /jobs/{jobId}/inputs`` (and corresponding endpoints under ``/processes``
+  and ``/providers``) allowing to retrieve submitted input values and requested outputs with either ``OGC``/``OLD``
+  formats.
 - Improve conformance for returned status codes and error messages when requesting results for an unfinished,
   failed, or dismissed ``Job``.
 - Adjust conformance item references to correspond with `OGC API - Processes: Part 2` renamed from `Transactions` to
@@ -22,6 +55,13 @@ Changes:
 
 Fixes:
 ------
+- Fix ``outputs`` permitted to be completely omitted from the execution request
+  (resolves `#375 <https://github.com/crim-ca/weaver/issues/375>`_).
+- Fix ``outputs`` permitted as explicit empty mapping or list as equivalent to omitting them, defining by default
+  that all ``outputs`` should be returned with ``transmissionMode: value`` for ``Job`` execution.
+- Fix all instances of ``outputTransmission`` reported as ``reference`` in ``Process`` descriptions, although `Weaver`
+  behaved with the ``value`` method, which is to return values and file references in content body, instead of
+  HTTP ``Link`` header references.
 - Fix `WPS 1/2` endpoint not reporting the appropriate instance URL
   (fixes `#83 <https://github.com/crim-ca/weaver/issues/83>`_).
 - Fix `CLI` ``deploy`` operation headers incorrectly passed down to the deployment request.
@@ -325,7 +365,7 @@ Fixes:
 - Fix parsing of inputs for `OpenSearch` parameters lookup that was assuming inputs were always provided as
   listing definition, not considering possible mapping definition.
 - Fix incorrect documentation section ``Package as External Execution Unit Reference`` where content was omitted
-  and incorrectly anchored as following ``process-esgf-cwt`` section.
+  and incorrectly anchored as following ``ESGF-CWT`` section.
 
 .. _changes_4.4.0:
 

config/weaver.ini.example

@@ -67,6 +67,13 @@ weaver.ssl_verify = true
 # see 'requests_options.yml.example'
 weaver.request_options =
 
+# --- Weaver Execution settings ---
+
+# maximum wait time allowed for Prefer header to run Job/Quote synchronously
+# over this limit, they will automatically fallback to asynchronous execution/estimation
+weaver.exec_sync_max_wait = 20
+weaver.quote_sync_max_wait = 20
+
 # --- Weaver CWL settings ---
 # NOTE: [experimental]
 # enforce provided effective user/group identifiers for Application Package execution
@@ -147,6 +154,15 @@ weaver.vault_dir = /tmp/vault
 [celery]
 #USE_CELERYCONFIG = True
 BROKER_URL = mongodb://mongodb:27017/celery
+# Result backend is required for SYNC execution.
+#   Using only the backend type matching the broker URL will automatically resolve to use it database location.
+#   For an alternative result location, provide the full backend directly rather than using "mongodb_backend_settings".
+#   This setting is not correctly parsed (dict) by "pyramid_celery", and separate [celery:mongodb_backend_settings] is
+#   also not found. If more configuration are required, consider using a "celeryconfig" (and "USE_CELERYCONFIG = True").
+#   That configuration should be placed at the root of weaver since "pyramid_celery" doesn't support custom locations.
+#       https://github.com/sontek/pyramid_celery/pull/89
+RESULT_BACKEND = mongodb
+# RESULT_BACKEND = mongodb://mongodb:27017/celery
 
 ###
 # wsgi server configuration

docker/Dockerfile-worker

@@ -13,9 +13,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     && apt update \
     # NOTE:
     #   Only install CLI package, 'docker-ce' and 'containerd.io' not required as they should be provided by host.
-    #   Docker sibliing execution is expected. See 'docker/docker-compose.yml.example' for details.
+    #   Docker sibling execution is expected. See 'docker/docker-compose.yml.example' for details.
     && apt install --no-install-recommends docker-ce-cli \
 	&& rm -rf /var/lib/apt/lists/*
 
 # run app
-CMD celery worker -E -A pyramid_celery.celery_app --ini "${APP_CONFIG_DIR}/weaver.ini"
+CMD celery worker -B -E -A pyramid_celery.celery_app --ini "${APP_CONFIG_DIR}/weaver.ini"

docs/_static/custom.css

@@ -1,4 +1,3 @@
-
 /* override readthedocs theme to enforce using full-screen width for content */
 .wy-nav-content {
     max-width: none;

docs/source/configuration.rst

@@ -213,14 +213,36 @@ they are optional and which default value or operation is applied in each situat
     completion if an email was provided in the :ref:`Execute <proc_op_execute>` request body
     (see also: :ref:`Email Notification`).
 
+.. versionadded:: 4.15.0
+
+- | ``weaver.exec_sync_max_wait``
+  | (default: ``20``, :class:`int`, seconds)
+  |
+  | Defines the maximum duration allowed for running a :term:`Job` execution in `synchronous` mode.
+  |
+  | See :ref:`proc_exec_mode` for more details on the feature and how to employ it.
+  | Ensure `Celery`_ worker is configured as specified below.
+
+.. versionadded:: 4.15.0
+
+- | ``weaver.quote_sync_max_wait``
+  | (default: ``20``, :class:`int`, seconds)
+  |
+  | Defines the maximum duration allowed for running a :term:`Quote` estimation in `synchronous` mode.
+  |
+  | See :ref:`proc_exec_mode` for more details on the feature and how to employ it.
+  | Ensure `Celery`_ worker is configured as specified below.
 
 .. note::
 
     Since `Weaver` employs `Celery`_ as task queue manager and `MongoDB`_ as backend, relevant settings for the
-    |celery-config|_ and the |celery-mongo|_ should be referred to. Processing of task jobs and results reporting
+    |celery-config|_ and the |celery-mongo|_ should be employed. Processing of task jobs and results reporting
     is accomplished according to the specific implementation of these services. Therefore, all applicable settings
     and extensions should be available for custom server configuration and scaling as needed.
 
+.. warning::
+    In order to support `synchronous` execution, the ``RESULT_BACKEND`` setting **MUST** be defined.
+
 .. |celery-config| replace:: configuration of Celery
 .. _celery-config: https://docs.celeryproject.org/en/latest/userguide/configuration.html#configuration
 .. |celery-mongo| replace:: configuration of MongoDB Backend
@@ -344,16 +366,42 @@ simply set setting ``weaver.wps_processes_file`` as *undefined* (i.e.: nothing a
 Configuration of Request Options
 =======================================
 
-.. todo:: complete docs
+.. versionadded:: 1.8.0
 
-:term:`Request Options`
+It is possible to define :term:`Request Options` that consist of additional arguments that will be passed down to
+:func:`weaver.utils.request_extra`, which essentially call a traditional request using :mod:`requests` module, but
+with extended handling capabilities such as caching, retrying, and file reference support. The specific parameters
+that are passed down for individual requests depend whether a match based on URL (optionally with regex rules) and
+method definitions can be found in the :term:`Request Options` file. This file should be provided using
+the ``weaver.request_options`` configuration setting. Using this definition, it is possible to provide specific
+requests handling options, such as extended timeout, authentication arguments, SSL certification verification setting,
+etc. on a per-request basis, leave other requests unaffected and generally more secure.
 
-``weaver.ssl_verify``
+.. seealso::
+    File `request_options.yml.example`_ provides more details and sample :term:`YAML` format of the expected contents
+    for :term:`Request Options` feature.
 
+.. seealso::
+    Please refer to :func:`weaver.utils.request_extra` documentation directly for supported parameters and capabilities.
 
-.. versionadded:: 1.8.0
 
-`request_options.yml.example`_
+- | ``weaver.request_options = <file-path>``
+  | (default: ``None``)
+  |
+  | Path of the :term:`Request Options` definitions to employ.
+
+
+- | ``weaver.ssl_verify = true|false``
+  | (default: ``true``)
+  |
+  | Toggle the SSL certificate verification across all requests.
+
+.. warning::
+    It is **NOT** recommended to disable SSL verification across all requests for security reasons
+    (avoid man-in-the-middle attacks). This is crucial for requests that involve any form of authentication, secured
+    access or personal user data references. This should be employed only for quickly resolving issues during
+    development. Consider fixing SSL certificates on problematic servers, or disable the verification on a per-request
+    basis using :term:`Request Options` for acceptable cases.
 
 
 Starting the Application

docs/source/faq.rst

@@ -46,7 +46,7 @@ Please refer to below references for more details.
 
 .. seealso::
 
-    - Supported :term:`Application Package` definitions in :ref:`process-wps-rest` deployment.
+    - Supported :term:`Application Package` definitions in :ref:`proc_wps_rest` deployment.
     - :ref:`Deploy <proc_op_deploy>` request.