0.26.0

Summary

🎉 This release is mainly focused on improving observability when operating servies. It introduces OpenTelemetry instrumentation and a completely refactored interface for logging.

New and updated

• Major refactoring of how logging is done, introducing tomodachi.get_logger() built on structlog. Contextual logger available for all handlers, etc.
• Provides the option of hooking in tomodachi to a custom logger the user provides.
• Adds instrumentation for opentelemetry (OTEL / OpenTelemetry) that can be enabled if tomodachi is installed using the opentelemetry extras. The OpenTelemetry instrumentation usage is explained in detail in #1790.
• OTEL auto instrumentation can be achieved by starting services using either the tomodachi run argument --opentelemetry-instrument (equivalent to setting env: TOMODACHI_OPENTELEMETRY_INSTRUMENT=1) or using the opentelemetry-instrument CLI.
• An experimental meter provider for exporting OpenTelemetry metrics on a Prometheus server can be used by installing the opentelemetry-exporter-prometheus extras and using the OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus environment value in combination with OTEL instrumentation.
• Adds the option to enable exemplars in the Prometheus client for OpenTelemetry metrics, to be able to link to traces from collected metrics in Grafana, et al.
• The HTTP body for requests with a body is read before the handler is called and if the connection was closed prematurely before the body could be read, the request will be ignored.
• Replaces the banner shown when starting a service without --production. The banner now includes the operating system, architecture, which Python runtime from which virtualenv is used, etc. in order to aid debugging during development for issues caused by environment misconfiguration.
• Updated the CLI usage output from --help.
• Added a value tomodachi.__build_time__ which includes the timestamp when the build for the installed release was done. The time that has passed since build time will be included in the start banner.
• Makes use of asyncio tasks instead of simply awaiting the coroutines so that the context from contextvars will be propagated correctly and not risk being corrupted by handlers.
• Added an internal lazy meta importer to ease renaming and deprecations of modules.
• Added additional lazy loading of submodules.
• Each argument for tomodachi run is now accompanied with an environment variable to do the same. For example --log-level warning can be achieved by setting TOMODACHI_LOG_LEVEL=warning.
• Updated documentation with additional information.

Potentially breaking changes

• The complete refactoring of logging changes how log entries are being emitted, both in the way that the event / message of the log records has changed, but also how a log handler is now also added to the logging.root logger on service start.
• Third party log records will if propagation is enabled also be processed in tomodachi.logging which may cause duplicate log output depending on how the third party logger is configured.
• Removed the log_setup() function that previously was added to the service object on class initialization and that was used to setup log output to disk.
• Tracebacks for uncaught exceptions are now extracted to only include frames relevant to the service application and not the internal tomodachi frames, which usually will be uninteresting for debugging.

Bug fixes

• Fixes exception catching of lifecycle handlers (_start_service, _started_service, etc.) which previously could stall a service raising an exception while starting, instead of exiting with a non-zero exit code.
• Bug fix for an issue which could cause the watcher to fail to restart the service after a syntax error was encountered.
• Adds some missing type hint annotations.
• Added additional logging of uncaught exceptions that previously may have been silenced.
• Fixed that the --log-level CLI argument value is actually applied to loggers.
• Fix for a race condition which could freeze a process if a service was manually stopped (interrupted with ctrl+c) before it had called its first lifecycle function (_start_service).

Deprecations

• Added deprecation warnings for more legacy functionality to give notice that those functions will be removed in a future release.
• The use of the log() function added to the service object is deprecated. Use the structlog logger from tomodachi.get_logger() instead.
• Using the RequestHandler.get_request_ip is deprecated. Instead use the tomodachi.get_forwarded_remote_ip() function.
• Deprecated the use of the CLI argument -c (--config) which could be used to set object attributes from a JSON file. A better pattern is to read optional config data from an environment variable.

0.25.1

• Fix for an issue where a wrapped function is used as a handler function, which would then cause the keyword argument provided transport values to rely on the keyword arguments from the wrapped function's signature to be used instead of the keyword arguments from the wrapper function's signature. Described with examples in #1781.

  The bug was found to be present in the 0.25.0 release, which included major refactoring of the keyword argument provided transport values functionality.

0.25.0

• The middleware execution logic has been improved to handle different argument types and edge cases more smoothly. Enhanced the way arguments are passed to middlewares and handlers, allowing better flexibility.

• Function handlers, middlewares and envelopes can all now specify additional keyword arguments in their signatures and receive transport centric values.

  Previously a few of these keyword values could be used for function handlers or envelopes, but not within middlewares. With this update these keywords can be used across all kind of handler functions to allow for more flexibility in how to structure apps, logging, tracing, authentication, etc.

• Resolved an edge case where a service could end up calling SNS.CreateTopic numerous times due to thousands of messages simultanously being published to a topic that were previously unknown to the service. Described in the detail in #1768.

• The aws_sns_sqs_publish function will now return the SNS message identifier as a str value if it is called with wait=True (default), or instead return an asyncio.Task object if called with wait=False.

---

Updated documentation on middlewares and function signature keyword arguments:

• See examples of custom middleware: https://tomodachi.dev/docs/middlewares.
• See the full list of function signature keywords that can be used: https://tomodachi.dev/docs/function-keywords.

---

This is an example of a middleware function for AWS SNS SQS messaging that adds trace spans on receiving messages, with a context extracted from a message attribute holding with a "traceparent" value. The topic name and SNS message identifier is also added as attributes to the trace span.

async def trace_middleware(
    func: Callable[... Awaitable],
    *,
    topic: str,
    message_attributes: dict,
    sns_message_id: str
) -> None:
    ctx: Context | None = None

    if carrier_traceparent := message_attributes.get("telemetry.carrier.traceparent"):
        carrier: dict[str, list[str] | str] = {"traceparent": carrier_traceparent}
        ctx = TraceContextTextMapPropagator().extract(carrier=carrier)

    with tracer.start_as_current_span(f"SNSSQS handler '{func.__name__}'", context=ctx) as span:
        span.set_attribute("messaging.system", "AmazonSQS")
        span.set_attribute("messaging.operation", "process")
        span.set_attribute("messaging.source.name", topic)
        span.set_attribute("messaging.message.id", sns_message_id)

        try:
            # Calls the handler function (or next middleware in the chain)
            await func()
        except BaseException as exc:
            logging.getLogger("exception").exception(exc)
            span.record_exception(exc, escaped=True)
            span.set_status(StatusCode.ERROR, f"{exc.__class__.__name__}: {exc}")
            raise exc

0.24.3

• Fixes an issue in the internal retry logic when using aws_sns_sqs_publish if calls to the AWS API SNS.Publish would intermittently respond with 408 response without any body, which previously would've resulted in a AWSSNSSQSException("Missing MessageId in response") immediately without retries.

  This was previously attempted to be fixed in #1664 (released in 0.23.0), but due to an error it fell through to become an exception with the "Missing MessageId in response" message instead.

  The publish function will now catch exceptions from botocore of type ResponseParserError to which botocore has added that "Further retries may succeed". tomodachi will retry such SNS.Publish calls up to 3 times and if still failing after all retries the library will reraise the original botocore exception.

  It seems that botocore does not automatically retry such errors itself.

• Similar to the above, the same kind of retries will now also be done during AWS API calls for SQS.DeleteMessage, where the botocore.parser.QueryParser would raise an ResponseParserError exception on 408 responses without body.

0.24.2

• Fixes typing syntax for compatibility with Python 3.8 and Python 3.9 to solve the incompatibility for Python 3.8 and Python 3.9 introduced in the the 0.24.1 release.
• Fixes an issue with an AWS SQS queue's message retention period attribute using an incompatible default value for FIFO queues.
• Support for aiobotocore 2.5.x releases.
• README.rst fixes to conform with RST format. (github: @navid-agz)

0.24.1

• Adds max number of messages that the service will consume when using AWS SNS+SQS
  handlers configurable. (github: @navid-agz)
• Changed default retention period of dead-letter-queues on AWS SNS+SQS.
  (github: @PabloAJomer)

0.24.0

• cchardet is no longer a direct dependency to tomodachi on Python 3.10 and Python 3.11. If you want to use it, you must install it separately, which may require additional build tools when installing on Python 3.10+.
• Updates to the internal tomodachi.envelope.ProtobufBase envelope to now also support protobuf Python bindings versioned >=4.0.0, when running with the (new default) PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb as upb slightly differs in representation of a Message type in relation to cpp and python implementations.
• Python 3.11 added to test matrix and trove classifiers to officially claim support.

0.23.0

• Properly handles aiobotocore client using an async contextmanager. Drops support for aiobotocore versions prior 1.3.0, but will now supporting newer versions. (github: @drestrepom)

• Fixes an issue to now retry calls where AWS SNS intermittently responds with 408 responses without any body, which trips up botocore.parser.QueryParser. (github: @technomunk)

• Refactored options used for AWS SNS+SQS, HTTP, AMQP and the Watcher functionality. Options set on the service class should now be defined as a tomodachi.Options object, which provides type hints and much nicer path traversal of the class.

  Only the specified typed values for options will now be allowed to be set. Setting a non-defined option will raise an AttributeError exception on service start.

  The previous dict based approach is still supported, but will be removed in a future version.

• Dropped support for Python 3.7.

0.22.3

• Support for assigning values to AWS SQS queue attributes value VisibilityTimeout and RedrivePolicy that is used to assign a queue to use a dead-letter queue after a number of failed attempts to consume a message. By default no changes will be done to the existing queue attributes and a change will only be triggered by assigning values to the visibility_timeout or both of dead_letter_queue_name + max_receive_count keyword arguments.

    @tomodachi.aws_sns_sqs(
        topic=None,
        competing=True,
        queue_name=None,
        filter_policy=FILTER_POLICY_DEFAULT,
        visibility_timeout=VISIBILITY_TIMEOUT_DEFAULT,     # affects MessageVisibility
        dead_letter_queue_name=DEAD_LETTER_QUEUE_DEFAULT,  # affects RedrivePolicy
        max_receive_count=MAX_RECEIVE_COUNT_DEFAULT,       # affects RedrivePolicy
        **kwargs,
    )

• Fixes a bug where SQS messages wouldn't get deleted from the queue if a middleware function catches an exception without reraising it. This is because the delete_message is not called from within routine_func (due to the exception breaking normal control flow), but the message deletion from middleware bubble is also skipped, as no exception is propagated from it. (github: @technomunk)
• Adds basic support for FIFO queues & topics on AWS SQS queues managed by a tomodachi service decorated function, which can be used where one needs guaranteed ordering of the consumed messages. (github: @kjagiello)
• Updates to the internal tomodachi.envelope.ProtobufBase envelope to now also support newer versions of protobuf.
• Added documentation to describe the "magic" functions that hooks into the service lifecycle; _start_service, _started_service, _stopping_service, _stop_service.

0.22.2

• Fixes an issue with live reloading on code changes (development mode) with services utilizing protobuf messages, which in same edge cases could trigger a repeated TypeError("A Message class can only inherit from Message") that would prevent the service from restarting correctly.