Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated guidance around time-to-be-received (TTBR) and auditing #6717

Closed
wants to merge 2 commits into from
Closed

Updated guidance around time-to-be-received (TTBR) and auditing #6717

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1fe952c
Updated guidance around time-to-be-received (TTBR) and auditing
ramonsmits Jun 24, 2024
5c312f8
Update nservicebus/operations/auditing.md
ramonsmits Aug 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions Snippets/Core/Core_7/Audit/OverrideTTBR.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
using System;
using NServiceBus;

class OverrideTTBR
{
public OverrideTTBR(EndpointConfiguration endpointConfiguration)
{
#region audit-ttbr-override

var auditTimeToBeReceived = TimeSpan.FromHours(4);
endpointConfiguration.AuditProcessedMessagesTo("targetAuditQueue", timeToBeReceived: auditTimeToBeReceived);

#endregion
}
}
15 changes: 15 additions & 0 deletions Snippets/Core/Core_8/Audit/OverrideTTBR.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
using System;
using NServiceBus;

class OverrideTTBR
{
public OverrideTTBR(EndpointConfiguration endpointConfiguration)
{
#region audit-ttbr-override

var auditTimeToBeReceived = TimeSpan.FromHours(4);
endpointConfiguration.AuditProcessedMessagesTo("targetAuditQueue", timeToBeReceived: auditTimeToBeReceived);

#endregion
}
}
15 changes: 15 additions & 0 deletions Snippets/Core/Core_9/Audit/OverrideTTBR.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
using System;
using NServiceBus;

class OverrideTTBR
{
public OverrideTTBR(EndpointConfiguration endpointConfiguration)
{
#region audit-ttbr-override

var auditTimeToBeReceived = TimeSpan.FromHours(4);
endpointConfiguration.AuditProcessedMessagesTo("targetAuditQueue", timeToBeReceived: auditTimeToBeReceived);

#endregion
}
}
44 changes: 10 additions & 34 deletions nservicebus/messaging/discard-old-messages.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Discarding Old Messages
title: Discarding Old Messages (time-to-be-received or time-to-live)
summary: Automatically discard messages if they have not been processed within a given period of time.
reviewed: 2023-11-30
component: Core
Expand All @@ -11,7 +11,10 @@ redirects:

A message sent through the Particular Service Platform may have Time-To-Be-Received (TTBR) set, according to the users’ decision. TTBR indicates to the platform that a delayed message can be discarded, if not handled within a specified period. A discarded message might no longer have any business value, and discarding it frees up system resources.

Setting TimeToBeReceived might be beneficial in environments with high volumes of messages where there is little business value in processing a delayed message since it will already be replaced by a newer, more relevant version.
> [!NOTE]
> Time-to-be-received (TTBR) is also know as Time-to-live (TTL)

Setting TTBR might be beneficial in environments with high volumes of messages where there is little business value in processing a delayed message since it will already be replaced by a newer, more relevant version.

TTBR applies only to messages that have not been handled. A failed message moved to the error queue or a successfully processed message is considered handled, as well as audit message when generated. Removing TTBR from handled messages ensures no messages are lost. The TTBR value from the original message can be inspected by looking at the `NServiceBus.TimeToBeReceived` [header](/nservicebus/messaging/headers.md).

Expand All @@ -23,12 +26,10 @@ A TTBR interval for a message can be specified:

snippet: DiscardingOldMessagesWithAnAttribute


## Using a custom convention

snippet: DiscardingOldMessagesWithCode


## Clock synchronization issues

When sending a message with a TimeToBeReceived value, it could happen that the receiver drops all messages due to clocks of the sender and the receiver being too much out of sync. For example, if TimeToBeReceived is 1 minute and the receiver's clock is 1 minute ahead compared to the sender’s the message becomes immediately stale - thus never processed.
Expand All @@ -37,7 +38,6 @@ In most deployments clocks are at most a few minutes out of sync. As a result, t

It is advised to add the maximum amount of allowed clock offset, called clock drift, to the TTBR value. For example, when using a TimeToBeReceived value of 90 seconds, one should allow for 300 seconds of maximum clock drift, so the TTBR value becomes 90 + 300 = 390 seconds.


## Discarding messages at startup

In certain situations, it may be required that messages in the incoming queue should not be processed after restarting the endpoint. This usually applies to development and test environments, but may also be appropriate for messages containing information that gets outdated or otherwise unneeded, e.g. change notifications, readings from sensors in IoT apps, etc.
Expand All @@ -49,36 +49,12 @@ To discard all existing messages in the incoming queue at startup:

snippet: PurgeMessagesAtStartup

## Caveats

TimeToBeReceived relies on the transport infrastructure to discard expired messages. As a result runtime behavior is highly affected by the implementation in the different transports.


### MSMQ transport

MSMQ continuously checks the TimeToBeReceived of all queued messages. As soon as the message has expired, it is removed from the queue, and disk space gets reclaimed.

> [!NOTE]
> MSMQ enforces a single TimeToBeReceived value for all messages in a transaction. To prevent message loss, `TimeToBeReceived` is not supported for endpoints with [transaction mode](/transports/transactions.md) `SendsAtomicWithReceive` or `TransactionScope` by default.

> [!WARNING]
> Due to a bug in Version 6 `TransportTransactionMode.ReceiveOnly` wrongly enlisted all outgoing messages in the same transaction causing the issues described above.

For more details about how the MSMQ transport handles TimeToBeReceived, see [discarding expired messages in MSMQ](/transports/msmq/discard-expired-messages.md).


### RabbitMQ transport

RabbitMQ continuously checks the TimeToBeReceived, but only for the first message in each queue. Expired messages are not removed from the queue, and their disk space is not reclaimed until they reach the front of the queue. Using TimeToBeReceived as a disk-saving measure on RabbitMQ is recommended only when all messages in the queue use the same TimeToBeReceived value. Otherwise, messages in front of the queue may prevent other, stale messages from being cleaned.


### Azure transports

The Azure transports evaluate the TimeToBeReceived for a message only when the message is requested by the client. Expired messages are not removed from the queue and their disk space will not be reclaimed until they reach the front of the queue and a consumer tries to read them. Using TimeToBeReceived as a storage saving measure on the Azure transports is not a good choice for queues with long-lived messages like audit and forward.

## Behavior

### SQL transport
Time-to-be-received relies on the transport infrastructure to discard expired messages. As a result runtime behavior is highly affected by the implementation in the different transports.

The SQL Server transport runs a periodic task that removes expired messages from the queue. The task is first executed when the endpoint starts and is subsequently scheduled to execute 5 minutes after the previous run when the task has been completed. Expired messages are not received from the queue and their disk space will be reclaimed when the periodic task executes.
Review the **Transport at a glance** section, specifically the **time-to-be-received (TTBR)** feature for each [transport](/transports/).

## Auditing

Auditing can also use TTBR but the [auditing TTBR behavior is version specific](/nservicebus/operations/auditing.md#configuring-auditing-time-to-be-received).
37 changes: 22 additions & 15 deletions nservicebus/operations/auditing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,34 +60,41 @@ Audit messages can be handled in a variety of ways: Save them in a database, do

partial: configuration

## Audit configuration options

There two settings that control auditing:

### Queue Name

The queue name to forward audit messages.

### OverrideTimeToBeReceived

To force a [TimeToBeReceived](/nservicebus/messaging/discard-old-messages.md) on audit messages by setting `OverrideTimeToBeReceived` use the configuration syntax below.
### Time-to-be-received

Note that while the phrasing is "forwarding a message" in the implementation it is actually "cloning and sending a new message". This is important when considering TimeToBeReceived since the time taken to receive and process the original message is not part of the TimeToBeReceived of the new audit message. In effect the audit message receives the full time allotment of whatever TimeToBeReceived is used.
The [Time-to-be-received (TTBR)](/nservicebus/messaging/discard-old-messages.md) for audit messages can be overriden.

> [!WARNING]
> MSMQ forces the same TimeToBeReceived on all messages in a transaction. Therefore, OverrideTimeToBeReceived is not supported when using the [MSMQ Transport](/transports/msmq/). If OverrideTimeToBeReceived is detected when using MSMQ an exception will be thrown with the following text:
>
> ```
> Setting a custom OverrideTimeToBeReceived for audits is not supported on transactional MSMQ
> ```
> [!NOTE]
> What happens to messages for which the TTBR expires is different. Most transports will **not** auto deleted expired message in the backlog of the queue and purge when dequeued and are received by the client. More details at [discarding old messages behavior](/nservicebus/messaging/discard-old-messages.md#behavior)

#### Default Value

If no OverrideTimeToBeReceived is defined then:
#if-version [, 6.0)

Until NServiceBus version 6.0 the TTBR of the original message will be used.

#end-if

#if-version [6.0, )

**Versions 5 and below**: TimeToBeReceived of the original message will be used.
Audit message since NServiceBus 6.x by default will have no TTBR set.

#end-if

#### Override TimeToBeReceived

The TimeToBeReceived (TTBR) on audit messages can be set using the following configuration syntax:

snippet: audit-ttbr-override

Note that while the phrasing is "forwarding a message" in the implementation it is actually "cloning and sending a new message". This is important when considering TimeToBeReceived since the time taken to receive and process the original message is not part of the TimeToBeReceived of the new audit message. In effect the audit message receives the full time allotment of whatever TimeToBeReceived is used.

**Versions 6 and above**: No TimeToBeReceived will be set.

## Filtering audit messages

Expand Down
25 changes: 15 additions & 10 deletions transports/azure-service-bus/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,21 @@ The Azure Service Bus transport leverages the [Azure.Messaging.ServiceBus](https

## Transport at a glance

|Feature | |
|:--- |---
|Transactions |None, ReceiveOnly, SendsWithAtomicReceive
|Pub/Sub |Native
|Timeouts |Native
|Large message bodies | with Premium tier or data bus
|Scale-out |Competing consumer
|Scripted Deployment |Supported using `NServiceBus.Transport.AzureServiceBus.CommandLine`
|Installers |Optional
|Native integration |[Supported](native-integration.md)
| Feature | |
|:---------------------------------------------------|---------------------------------------------------------------------|
| Transactions | None, ReceiveOnly, SendsWithAtomicReceive |
| Pub/Sub | Native |
| Timeouts | Native |
| Large message bodies | with Premium tier or data bus |
| Scale-out | Competing consumer |
| Scripted Deployment | Supported using `NServiceBus.Transport.AzureServiceBus.CommandLine` |
| Installers | Optional |
| Native integration | [Supported](native-integration.md) |
| [time-to-be-received](#time-to-be-received) (TTBR) | Once the message is fetched by the consumer |

> [!NOTE]
> The Azure Service Bus transport only supports the Standard and Premium tiers of the Microsoft Azure Service Bus service. Premium tier is recommended for production environments.

## Time-to-be-received

The Azure broker evaluate the TimeToBeReceived for a message only when the message is requested by the client. Expired messages are not removed from the queue and their disk space will not be reclaimed until they reach the front of the queue and a consumer tries to read them. Using TimeToBeReceived as a storage saving measure on the Azure transports is not a good choice for queues with long-lived messages like audit and forward.
41 changes: 31 additions & 10 deletions transports/azure-storage-queues/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,41 @@ include: azure-transports

## Transport at a glance

|Feature | |
|:--- |---
|Transactions |None, ReceiveOnly (Message visibility timeout)
|Pub/Sub |Native (Requires Storage Table)
|Timeouts |Native (Requires Storage Table)
|Large message bodies |Data bus
|Scale-out |Competing consumer
|Scripted Deployment |Not supported
|Installers |Mandatory
|Native integration |[Supported](native-integration.md)
| Feature | |
|:---------------------------|---------------------------------------------------|
| Transactions | None, ReceiveOnly (Message visibility timeout) |
| Pub/Sub | Native (Requires Storage Table) |
| Timeouts | Native (Requires Storage Table) |
| Large message bodies | Data bus |
| Scale-out | Competing consumer |
| Scripted Deployment | Not supported |
| Installers | Mandatory |
| Native integration | [Supported](native-integration.md) |
| time-to-be-received (TTBR) | Deleted after expiration, TTL depends per version |

## Configuring the endpoint

partial: endpointconfig

> [!NOTE]
> When using Azure Storage Queues transport, a serializer must be configured explicitly [by the `UseSerialization` API](/nservicebus/serialization/).

## Time-to-be-received

The term [Azure Storage queues uses for time-to-be-received is time-to-live (TTL)](https://learn.microsoft.com/en-us/rest/api/storageservices/put-message#uri-parameters)

Azure Storage Queues will automatically delete the message from the queue once the TTL value expires.

#if-version ASQN [8.1,)

Since Azure Storage Queues transport version 8.1.0 it will use a default TTL of 30 days

#end-if

#if-version ASQ [,9.0)

Before Azure Storage Queues transport version 8.1.0 it will use the Azure Storage Queues default TTL which is 7 days.

Since Azure Storage Queues transport version 8.1.0 it will use a default TTL of 30 days

#end-if
21 changes: 11 additions & 10 deletions transports/learning/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,17 @@ include: learning-usages

## Transport at a glance

|Feature | |
|:--- |---
|Transactions |None, ReceiveOnly, SendsWithAtomicReceive
|Pub/Sub |Native
|Timeouts |Native
|Large message bodies |LearningTransport can handle arbitrary message sizes within available resources
|Scale-out |Competing consumer
|Scripted Deployment |Not supported
|Installers |Not supported, the transport always creates the required folder structure
|Native integration |Not supported
| Feature | |
|:---------------------------|---------------------------------------------------------------------------------|
| Transactions | None, ReceiveOnly, SendsWithAtomicReceive |
| Pub/Sub | Native |
| Timeouts | Native |
| Large message bodies | LearningTransport can handle arbitrary message sizes within available resources |
| Scale-out | Competing consumer |
| Scripted Deployment | Not supported |
| Installers | Not supported, the transport always creates the required folder structure |
| Native integration | Not supported |
| time-to-be-received (TTBR) | Not supported, ignored after fetching |

### Publish and subscribe

Expand Down
24 changes: 22 additions & 2 deletions transports/msmq/discard-expired-messages.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,39 @@ related:
- nservicebus/messaging/discard-old-messages
---

The MSMQ transport can handle messages with a [Time-To-Be-Received (TTBR)](/nservicebus/messaging/discard-old-messages.md) set in two ways.
The MSMQ transport can handle messages with a [Time-To-Be-Received (TTBR)](/nservicebus/messaging/discard-old-messages.md) set in two ways:

| Transaction mode | MSMQ TTBR mode |
|------------------------|----------------|
| TransactionScope | Non-native |
| SendsAtomicWithReceive | Non-native |
| ReceiveOnly | Native |
| None | Native |

## Native

When a message with a TTBR value is sent, NServiceBus translates the value to the [native TTBR property](https://docs.microsoft.com/en-us/dotnet/api/system.messaging.message.timetobereceived) of the MSMQ message. MSMQ continuously checks the Time-To-Be-Received of all queued messages. As soon as the message has expired, it is removed from the queue, and disk space gets reclaimed.
When a message with a TTBR value is sent, NServiceBus translates the value to the [native TTBR property](https://docs.microsoft.com/en-us/dotnet/api/system.messaging.message.timetobereceived) of the MSMQ message. MSMQ continuously checks the TTBR of all queued messages. As soon as the message has expired, it is removed from the queue, and disk space gets reclaimed.

> [!NOTE]
> MSMQ enforces a single Time-To-Be-Received value for all messages in a transaction. If multiple messages enlist in a single transaction, then the TTBR from the first message will be used for all messages, leading to potentially unintentional message expiration. To prevent message loss, TTBR is not supported for endpoints with [transaction mode](/transports/transactions.md) `SendsAtomicWithReceive` or `TransactionScope` by default.

partial: ttbr-send

> [!WARNING]
> Due to a bug in Version 6 `TransportTransactionMode.ReceiveOnly` wrongly enlisted all outgoing messages in the same transaction causing the issues described above.

### Validation

The MSMQ tranport forces the same TimeToBeReceived on all messages in a transaction. If OverrideTimeToBeReceived is detected when using MSMQ an exception will be thrown with the following text:

```txt
Setting a custom OverrideTimeToBeReceived for audits is not supported on transactional MSMQ
```

## Non-native

NServiceBus also annotates outgoing messages with an `NServiceBus.TimeToBeReceived` [header](/nservicebus/messaging/headers.md).

## Ignore the `NServiceBus.TimeToBeReceived` header

partial: ttbr-receive
Loading