Skip to content

Commit

Permalink
fix tests and add docs for error_output implementation
Browse files Browse the repository at this point in the history
  • Loading branch information
ekneg54 committed Sep 25, 2024
1 parent 5fc1ea2 commit 5905de4
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 6 deletions.
13 changes: 13 additions & 0 deletions doc/source/development/connector_how_to.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,19 @@ Analogous to the input, exceptions that require a restart of Logprep should inhe
Exceptions that inherit from `OutputWarning` will be logged, but they do not require any error handling.


.. _error_output:

Error Output
------------

Error output is setup in :py:class:`~logprep.framework.pipeline_manager.PipelineManager`. The error
output connector is instantiated and setup only once during the initialization of the pipeline manager
together with :py:class:`~logprep.framework.pipeline_manager.ComponentQueueListener`.
The listener is used to listen on the populated error queue and to send the log messages to the
:code:`store` method of the error output connector.
The error queue is given to the listener and to all pipelines instantiated by the pipeline manager.


Factory
-------

Expand Down
13 changes: 8 additions & 5 deletions doc/source/development/processor_how_to.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ process

This method is implemented in the :py:class:`~logprep.abc.processor.Processor` and is called for every log message.
To process the event it invokes the processors `apply_rules` method.
If you want to do somthing to the event after all rules have been applied, then you could overwrite this method and implement your code after calling the `super().process(event)`.
If you want to do something to the event after all rules have been applied, then you could overwrite this method and implement your code after calling the `super().process(event)`.
The log message is being passed as a dictionary and modified 'in place', meaning that modifications are being performed directly on the input event.

.. code-block:: python
Expand All @@ -112,11 +112,14 @@ The log message is being passed as a dictionary and modified 'in place', meaning
Exceptions/Error Handling
~~~~~~~~~~~~~~~~~~~~~~~~~

An exception should be thrown if an error occurs during the processing of a log message.
All exceptions are being logged and should return a helpful error message with `str(exception)`.
Exceptions derived from `ProcessorWarningError` have no impact on the operation of the processor.
An exception should not been thrown on processor level. Instead it should be added to
the :py:class:`~logprep.abc.processor.ProcessorResult` if an error occurs during the processing of a log message.
All exceptions are being logged in :py:class:`~logprep.framework.pipeline.Pipeline` and should provide a helpful
error message with `str(exception)`. You do not have to log the exception in the processor itself.
Exceptions derived from :py:class:`~logprep.processor.base.exceptions.ProcessingWarning` have no impact on the
operation of the processor.
Other exceptions stop the processing of a log message.
However, the log message will be separately stored as failed (see :ref:`connector_output`, `store_failed``).
However, the log message will be separately stored as failed (see :ref:`error_output`).


Metrics
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/metrics/test_exporter.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@


@mock.patch(
"logprep.metrics.exporter.PrometheusExporter._prepare_multiprocessing",
"logprep.metrics.exporter.PrometheusExporter.prepare_multiprocessing",
new=lambda *args, **kwargs: None,
)
class TestPrometheusExporter:
Expand Down

0 comments on commit 5905de4

Please sign in to comment.