Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first implementation prefer header for execute job #415

Merged
merged 24 commits into from
Mar 31, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
72ba589
first implementation prefer header for execute job
fmigneault Mar 15, 2022
6de71bc
support Prefer header properly (resolves #247) + allow omitting outpu…
fmigneault Mar 15, 2022
161cf9c
adjust conformance prefer return none (relates to #414)
fmigneault Mar 15, 2022
bf02f08
Merge branch 'master' into job-prefer-header
fmigneault Mar 15, 2022
5520872
remove references to fix #247 in CLI, always using async mode with mo…
fmigneault Mar 15, 2022
bed33da
remove reference to fix #375 in CLI, outputs can be omitted but not f…
fmigneault Mar 15, 2022
42640cd
fix test to consider job exec params now optional to allow alternatives
fmigneault Mar 16, 2022
637adca
fix lint
fmigneault Mar 16, 2022
7b938fc
celery beat to worker docker command for task result cleaup
fmigneault Mar 16, 2022
b59949f
ignore linkcheck celery docs down (relates to https://github.com/cele…
fmigneault Mar 16, 2022
ff2f94c
fix lint
fmigneault Mar 16, 2022
9d56a4e
ensure jobControlOptions and outputTransmission are reported sorted
fmigneault Mar 16, 2022
83adf03
Merge branch 'master' into job-prefer-header
fmigneault Mar 16, 2022
070a9e7
Merge branch 'master' into job-prefer-header
fmigneault Mar 17, 2022
b8de26d
Merge branch 'master' into job-prefer-header
fmigneault Mar 17, 2022
895d25d
add notes regarding order of reported jobControlOptions
fmigneault Mar 17, 2022
896a4fa
Merge branch 'job-prefer-header' of https://github.com/crim-ca/weaver…
fmigneault Mar 17, 2022
34fc119
adjust test for new handling of jobControlOptions
fmigneault Mar 17, 2022
b70cb9c
revert weaver default jobControlOption async only - if sync needed, m…
fmigneault Mar 17, 2022
0e9119a
revert test for supported jobControlOptions
fmigneault Mar 17, 2022
170c6fd
ignore except too broad for int check
fmigneault Mar 21, 2022
2ba0679
fix doc indents
fmigneault Mar 22, 2022
3ac2830
quote requests typing
fmigneault Mar 22, 2022
bf99d10
add ref execute schema not required inputs
fmigneault Mar 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ env
package-lock.json
node_modules

## Celery
celeryconfig*
celery-config*
celerybeat-schedule.*

## Python / Extensions etc.
*~
*.mo
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ node_modules
## Docker
#Dockerfile

## Celery
celeryconfig*
celery-config*
celerybeat-schedule.*

## Python / Extensions etc.
*~
*.mo
Expand Down
19 changes: 18 additions & 1 deletion CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@ Changes

Changes:
--------
- Support ``Prefer`` header with ``wait`` or ``respond-async`` directives to select ``Job`` execution mode either
as *synchronous* or *asynchronous* task, according to supported ``jobControlOptions`` of the relevant ``Process``
being executed (resolves `#247 <https://github.com/crim-ca/weaver/issues/247>`_).
- Increase minor version of all ``builtin`` processes that will now be executable in wither (a)synchronous modes.
- Add ``weaver.exec_sync_max_wait`` and ``weaver.quote_sync_max_wait`` settings allowing custom definition for the
maximum duration that can be specified to wait for a `synchronous` response from task workers.
- Add ``-B`` (``celery beat``) option to Docker command of ``weaver-worker`` to run scheduled task in parallel
to ``celery worker`` in order to periodically cleanup task results introduced by *synchronous* execution.
- Improve conformance for returned status codes and error messages when requesting results for an unfinished,
failed, or dismissed ``Job``.
- Adjust conformance item references to correspond with `OGC API - Processes: Part 2` renamed from `Transactions` to
Expand All @@ -20,8 +28,17 @@ Changes:
(resolves `#180 <https://github.com/crim-ca/weaver/issues/180>`_).
- Improve ``Process`` undeployment to consider running ``Job`` to block its removal while in use.

Important Note
--------------
- In order to support *synchronous* execution, setting ``RESULT_BACKEND`` **MUST** be specified in
the ``weaver.ini`` configuration file.
See `Weaver INI Configuration Example <https://github.com/crim-ca/weaver/blob/master/config/weaver.ini.example>`_
in section ``[celery]`` for more details.

Fixes:
------
- Fix ``outputs`` permitted to be completely omitted from the execution request
(resolves `#375 <https://github.com/crim-ca/weaver/issues/375>`_).
- Fix `WPS 1/2` endpoint not reporting the appropriate instance URL
(fixes `#83 <https://github.com/crim-ca/weaver/issues/83>`_).
- Fix `CLI` ``deploy`` operation headers incorrectly passed down to the deployment request.
Expand Down Expand Up @@ -325,7 +342,7 @@ Fixes:
- Fix parsing of inputs for `OpenSearch` parameters lookup that was assuming inputs were always provided as
listing definition, not considering possible mapping definition.
- Fix incorrect documentation section ``Package as External Execution Unit Reference`` where content was omitted
and incorrectly anchored as following ``process-esgf-cwt`` section.
and incorrectly anchored as following ``ESGF-CWT`` section.

.. _changes_4.4.0:

Expand Down
16 changes: 16 additions & 0 deletions config/weaver.ini.example
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,13 @@ weaver.ssl_verify = true
# see 'requests_options.yml.example'
weaver.request_options =

# --- Weaver Execution settings ---

# maximum wait time allowed for Prefer header to run Job/Quote synchronously
# over this limit, they will automatically fallback to asynchronous execution/estimation
weaver.exec_sync_max_wait = 20
weaver.quote_sync_max_wait = 20

# --- Weaver CWL settings ---
# NOTE: [experimental]
# enforce provided effective user/group identifiers for Application Package execution
Expand Down Expand Up @@ -147,6 +154,15 @@ weaver.vault_dir = /tmp/vault
[celery]
#USE_CELERYCONFIG = True
BROKER_URL = mongodb://mongodb:27017/celery
# Result backend is required for SYNC execution.
# Using only the backend type matching the broker URL will automatically resolve to use it database location.
# For an alternative result location, provide the full backend directly rather than using "mongodb_backend_settings".
# This setting is not correctly parsed (dict) by "pyramid_celery", and separate [celery:mongodb_backend_settings] is
# also not found. If more configuration are required, consider using a "celeryconfig" (and "USE_CELERYCONFIG = True").
# That configuration should be placed at the root of weaver since "pyramid_celery" doesn't support custom locations.
# https://github.com/sontek/pyramid_celery/pull/89
RESULT_BACKEND = mongodb
# RESULT_BACKEND = mongodb://mongodb:27017/celery

###
# wsgi server configuration
Expand Down
4 changes: 2 additions & 2 deletions docker/Dockerfile-worker
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& apt update \
# NOTE:
# Only install CLI package, 'docker-ce' and 'containerd.io' not required as they should be provided by host.
# Docker sibliing execution is expected. See 'docker/docker-compose.yml.example' for details.
# Docker sibling execution is expected. See 'docker/docker-compose.yml.example' for details.
&& apt install --no-install-recommends docker-ce-cli \
&& rm -rf /var/lib/apt/lists/*

# run app
CMD celery worker -E -A pyramid_celery.celery_app --ini "${APP_CONFIG_DIR}/weaver.ini"
CMD celery worker -B -E -A pyramid_celery.celery_app --ini "${APP_CONFIG_DIR}/weaver.ini"
1 change: 0 additions & 1 deletion docs/_static/custom.css
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

/* override readthedocs theme to enforce using full-screen width for content */
.wy-nav-content {
max-width: none;
Expand Down
60 changes: 54 additions & 6 deletions docs/source/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -213,14 +213,36 @@ they are optional and which default value or operation is applied in each situat
completion if an email was provided in the :ref:`Execute <proc_op_execute>` request body
(see also: :ref:`Email Notification`).

.. versionadded:: 4.15.0

- | ``weaver.exec_sync_max_wait``
| (default: ``20``, :class:`int`, seconds)
|
| Defines the maximum duration allowed for running a :term:`Job` execution in `synchronous` mode.
|
| See :ref:`proc_exec_mode` for more details on the feature and how to employ it.
| Ensure `Celery`_ worker is configured as specified below.

.. versionadded:: 4.15.0

- | ``weaver.quote_sync_max_wait``
| (default: ``20``, :class:`int`, seconds)
|
| Defines the maximum duration allowed for running a :term:`Quote` estimation in `synchronous` mode.
|
| See :ref:`proc_exec_mode` for more details on the feature and how to employ it.
| Ensure `Celery`_ worker is configured as specified below.

.. note::

Since `Weaver` employs `Celery`_ as task queue manager and `MongoDB`_ as backend, relevant settings for the
|celery-config|_ and the |celery-mongo|_ should be referred to. Processing of task jobs and results reporting
|celery-config|_ and the |celery-mongo|_ should be employed. Processing of task jobs and results reporting
is accomplished according to the specific implementation of these services. Therefore, all applicable settings
and extensions should be available for custom server configuration and scaling as needed.

.. warning::
In order to support `synchronous` execution, the ``RESULT_BACKEND`` setting **MUST** be defined.

.. |celery-config| replace:: configuration of Celery
.. _celery-config: https://docs.celeryproject.org/en/latest/userguide/configuration.html#configuration
.. |celery-mongo| replace:: configuration of MongoDB Backend
Expand Down Expand Up @@ -344,16 +366,42 @@ simply set setting ``weaver.wps_processes_file`` as *undefined* (i.e.: nothing a
Configuration of Request Options
=======================================

.. todo:: complete docs
.. versionadded:: 1.8.0

:term:`Request Options`
It is possible to define :term:`Request Options` that consist of additional arguments that will be passed down to
:func:`weaver.utils.request_extra`, which essentially call a traditional request using :mod:`requests` module, but
with extended handling capabilities such as caching, retrying, and file reference support. The specific parameters
that are passed down for individual requests depend whether a match based on URL (optionally with regex rules) and
method definitions can be found in the :term:`Request Options` file. This file should be provided using
the ``weaver.request_options`` configuration setting. Using this definition, it is possible to provide specific
requests handling options, such as extended timeout, authentication arguments, SSL certification verification setting,
etc. on a per-request basis, leave other requests unaffected and generally more secure.

``weaver.ssl_verify``
.. seealso::
File `request_options.yml.example`_ provides more details and sample :term:`YAML` format of the expected contents
for :term:`Request Options` feature.

.. seealso::
Please refer to :func:`weaver.utils.request_extra` documentation directly for supported parameters and capabilities.

.. versionadded:: 1.8.0

`request_options.yml.example`_
- | ``weaver.request_options = <file-path>``
| (default: ``None``)
|
| Path of the :term:`Request Options` definitions to employ.


- | ``weaver.ssl_verify = true|false``
| (default: ``true``)
|
| Toggle the SSL certificate verification across all requests.

.. warning::
It is **NOT** recommended to disable SSL verification across all requests for security reasons
(avoid man-in-the-middle attacks). This is crucial for requests that involve any form of authentication, secured
access or personal user data references. This should be employed only for quickly resolving issues during
development. Consider fixing SSL certificates on problematic servers, or disable the verification on a per-request
basis using :term:`Request Options` for acceptable cases.


Starting the Application
Expand Down
2 changes: 1 addition & 1 deletion docs/source/faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Please refer to below references for more details.

.. seealso::

- Supported :term:`Application Package` definitions in :ref:`process-wps-rest` deployment.
- Supported :term:`Application Package` definitions in :ref:`proc_wps_rest` deployment.
- :ref:`Deploy <proc_op_deploy>` request.


Expand Down
Loading