From 68ba237d57e1bfddd0f96419d745728f68a6a7a3 Mon Sep 17 00:00:00 2001 From: Francis Charette-Migneault Date: Wed, 23 Mar 2022 00:40:26 -0400 Subject: [PATCH] update docs with new response=raw, mode=sync and transmissionMode=value/reference supports (relates to #376, #377) --- docs/source/processes.rst | 85 +++++++++++++++++++++++++------------- docs/source/references.rst | 12 +++++- 2 files changed, 66 insertions(+), 31 deletions(-) diff --git a/docs/source/processes.rst b/docs/source/processes.rst index 8d85a0428..5253b51d9 100644 --- a/docs/source/processes.rst +++ b/docs/source/processes.rst @@ -477,9 +477,12 @@ and parametrization of various input/output combinations. Let's employ the follo - `OGC API - Processes, Execution Mode `_ for more general details on the execution negotiation (formerly with ``mode`` parameter) and more recently with ``Prefer`` header. - - `OGC API - Processes, Response `_ + - |ogc-exec-sync-responses|_ and |ogc-exec-async-responses|_ for a complete listing of available ``response`` formats considering all other parameters. +.. |exec-api| replace:: OpenAPI Execute +.. _exec-api: `exec-req`_ + .. _proc_exec_body: Execution Body @@ -512,23 +515,11 @@ This could be added later if ``Multipart`` raw data representation is required. Please |submit-issue|_ to request this feature if it is relevant for your use-cases. .. fixme:: - Filtering not implemented (everything always available). + Filtering of ``outputs`` not implemented (everything always available). https://github.com/crim-ca/weaver/issues/380 -.. fixme:: - Transmission mode ``value`` not implemented. Only ``reference`` is supported. - https://github.com/crim-ca/weaver/issues/377 - -.. fixme:: - The ``response`` field is only supported with the ``document`` value. This parameter is present only for - compatibility with other :term:`ADES` implementation, but does not actually affects `Weaver`'s response. - - Response representation mode ``raw`` to be implemented. - https://github.com/crim-ca/weaver/issues/376 - -.. |exec-api| replace:: OpenAPI Execute -.. _exec-api: `exec-req`_ - +Other parameters presented in the above examples, namely ``mode`` and ``response`` are further detailed in +the following :ref:`proc_exec_mode` section. .. _proc_exec_mode: @@ -584,17 +575,36 @@ to wait on resources for too long), the :term:`Job` execution will automatically The distinction between an `asynchronous` or `synchronous` response when executing a :term:`Job` can be observed in multiple ways. The easiest is with the HTTP status code of the response, 200 being for a :term:`Job` *entirely completed* synchronously, and 201 for a created :term:`Job` that should be -:ref:`monitored ` asynchronously. Another method is to observe the ``"status"`` value. If the -status is ``accepted`` or ``running``, it means the operation is guaranteed to be `asynchronous`, since `synchronous` -always return a final status (``succeeded`` or ``failed``). Note that a final status is possible in both modes, so -seing one of those values does not *guarantee* it was executed `synchronously`, but the complete :term:`Job` status -can be immediately requested with the :ref:`GetStatus ` request. In general, a `synchronous` response -will be much more verbose than an `asynchronous` one, since details are not yet all available. Finally, it is possible -to extract the ``Preference-Applied`` response header which will clearly indicate if the submitted ``Prefer`` header -was respected (because it could be with available worker resources) or not. In general, this means that if -the :term:`Job` submission request was not provided with ``Prefer: wait=X`` **AND** replied with the -same ``Preference-Applied`` value, it is safe to assume `Weaver` decided to queue the :term:`Job` for `asynchronous` -execution. That :term:`Job` could be executed immediately, or at a later time, according to worker availability. +:ref:`monitored ` asynchronously. Another method is to observe the ``"status"`` value. +Effectively, a :term:`Job` that is executed `asynchronously` will return status information contents, while +a `synchronous` :term:`Job` will return the results directly, along a ``Location`` header referring to the +equivalent contents returned by :ref:`GetStatus ` as in the case of `asynchronous` :term:`Job`. +It is also possible to extract the ``Preference-Applied`` response header which will clearly indicate if the +submitted ``Prefer`` header was respected (because it could be with available worker resources) or not. +In general, this means that if the :term:`Job` submission request was not provided with ``Prefer: wait=X`` **AND** +replied with the same ``Preference-Applied`` value, it is safe to assume `Weaver` decided to queue the :term:`Job` +for `asynchronous` execution. That :term:`Job` could be executed immediately, or at a later time, according to +worker availability. + +It is also possible that a ``failed`` :term:`Job`, even when `synchronous`, will respond with equivalent contents +to the status location instead of results. This is because it is impossible for `Weaver` to return +the result(s) as outputs would not be generated by the incomplete :term:`Job`. + +Finally, the ``response`` parameter defines how to return the results produced by the :term:`Process`. +When ``response=document``, regardless of ``mode=async`` or ``mode=sync``, and regardless of requested +outputs ``transmissionMode=value`` or ``transmissionMode=reference``, the results will be returned in +a :term:`JSON` format containing either literal values or URL references to produced files. If ``mode=async``, +this results *document* is obtained with |result-req|_ request, while ``mode=sync`` returns it directly. +When ``response=raw``, the specific contents (type and quantity), HTTP ``Link`` headers or a mix of those components +depends both on the number of available :term:`Process` outputs, which ones were requested, and how they were +requested (i.e.: ``transmissionMode``). It is also possible that further content negotiation gets involved +accordingly to the ``Accept`` header and available ``Content-Type`` of the outputs if multiple formats are supported +by the :term:`Process`. For more details regarding those combination, the official +|ogc-exec-sync-responses|_ and |ogc-exec-async-responses|_ should be employed as reference. + +For any of the previous combinations, it is always possible to obtain :term:`Job` outputs, along with logs, exceptions +and other details using the :ref:`proc_op_result` endpoints. + .. _proc_exec_steps: @@ -1233,10 +1243,10 @@ format is employed according to the chosen location. .. _proc_op_result: -Obtaining output results, logs or errors +Obtaining results, outputs, logs or errors --------------------------------------------------------------------- -In the case of successful :term:`Job` execution, the outputs can be retrieved with |result-req|_ request to list +In the case of successful :term:`Job` execution, the *outputs* can be retrieved with |output-req|_ request to list each corresponding output ``id`` with the generated file reference URL. Keep in mind that the purpose of those URLs are only to fetch the results (not persistent storage), and could therefore be purged after some reasonable amount of time. The format should be similar to the following example, with minor variations according to :ref:`Configuration` @@ -1253,6 +1263,15 @@ parameters for the base :term:`WPS` output location: ] } +For the :term:`OGC` compliant endpoint, the |result-req| request can be employed instead. +In the event of a :term:`Job` executed with ``response=document``, the contents will be very similar. +On the other hand, a :term:`Job` submitted with ``response=raw`` can produce many alternative variations according +to :term:`OGC` requirements. For this reason, the *outputs* endpoint will always provide all data and file references +in the response body as :term:`Job`, no matter the original ``response`` format. The *outputs* endpoint can also +receive additional query parameters, such as ``schema``, to return contents formatted similarly to *results*, but +enforcing a :term:`JSON` body as if ``response=document`` was specified during submission of the :term:`Process` +execution. + In situations where the :term:`Job` resulted into ``failed`` status, the |except-req|_ can be use to retrieve the potential cause of failure, by capturing any raised exception. Below is an example of such exception details. @@ -1288,6 +1307,14 @@ Note again that the more the :term:`Process` is verbose, the more tracking will .. literalinclude:: ../../weaver/wps_restapi/examples/job_logs.json :language: json + +.. note:: + All endpoints to retrieve any of the above information about a :term:`Job` can either be requested directly + (i.e.: ``/jobs/{jobID}/...``) or with equivalent :term:`Provider` and/or :term:`Process` prefixed endpoints, + if the requested :term:`Job` did refer to those :term:`Provider` and/or :term:`Process`. + A *local* :term:`Process` would have its :term:`Job` references as ``/processes/{processId}/jobs/{jobID}/...`` + while a :ref:`proc_remote_provider` will use ``/provider/{providerName}/processes/{processId}/jobs/{jobID}/...``. + .. _vault: Uploading File to the Vault diff --git a/docs/source/references.rst b/docs/source/references.rst index 418d0557c..bb473deb5 100644 --- a/docs/source/references.rst +++ b/docs/source/references.rst @@ -56,6 +56,10 @@ .. _ogc-home: `ogc`_ .. |ogc-proc-api| replace:: OGC API - Processes .. _ogc-proc-api: https://github.com/opengeospatial/ogcapi-processes +.. |ogc-exec-sync-responses| replace:: OGC API - Processes, Responses (sync) +.. _ogc-exec-sync-responses: https://docs.ogc.org/is/18-062r2/18-062r2.html#sc_execute_response +.. |ogc-exec-async-responses| replace:: OGC API - Processes, Responses (async) +.. _ogc-exec-async-responses: https://docs.ogc.org/is/18-062r2/18-062r2.html#response_7 .. |pywps| replace:: PyWPS .. _pywps: https://github.com/geopython/pywps/ .. |pywps-status| replace:: Progress and Status Report @@ -135,8 +139,12 @@ .. _status-req-name: `status-req`_ .. |status-req| replace:: ``GET {WEAVER_URL}/processes/{id}/jobs/{id}`` (GetStatus) .. _status-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Status%2Fpaths%2F~1processes~1{process_id}~1jobs~1{job_id}%2Fget -.. |result-req| replace:: ``GET {WEAVER_URL}/processes/{id}/jobs/{id}/result`` (GetResult) -.. _result-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Results%2Fpaths%2F~1processes~1%7Bprocess_id%7D~1jobs~1%7Bjob_id%7D~1result%2Fget +.. |inputs-req| replace:: ``GET {WEAVER_URL}/jobs/{id}/inputs`` (Inputs) +.. _input-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/inputs/paths/~1jobs~1{job_id}~1inputs/get +.. |output-req| replace:: ``GET {WEAVER_URL}/jobs/{id}/outputs`` (Outputs) +.. _output-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/outputs/paths/~1jobs~1{job_id}~1outputs/get +.. |result-req| replace:: ``GET {WEAVER_URL}/jobs/{id}/results`` (Results) +.. _result-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/Results/paths/~1jobs~1{job_id}~1results/get .. |update-token-req| replace:: Update Token .. _update-token-req: https://pavics-weaver.readthedocs.io/en/latest/api.html#tag/UpdateToken/paths/~1processes~1{process_id}/put .. |vault-upload-req| replace:: Vault File Upload (POST)