From 4ac01f5b08d879d96004928453eef0ab23e6d7c0 Mon Sep 17 00:00:00 2001 From: Francis Roberts <111994975+franrob-projects@users.noreply.github.com> Date: Tue, 23 Sep 2025 13:01:33 +0200 Subject: [PATCH 01/14] Can I delete messages which have been persisted through the history API? --- src/pages/docs/storage-history/storage.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/pages/docs/storage-history/storage.mdx b/src/pages/docs/storage-history/storage.mdx index 049f754672..5eaf4a970d 100644 --- a/src/pages/docs/storage-history/storage.mdx +++ b/src/pages/docs/storage-history/storage.mdx @@ -27,6 +27,10 @@ The time that messages will be stored for depends on your account package: There is a cost associated with storing messages for longer than the minimum time period. [Contact us](https://ably.com/support) to discuss enabling long term storage. +### Message deletion + +Ably does not currently provide an API to delete persisted messages from the history. Once messages are stored with persisted history enabled, they will remain for the entire configured storage period. If you need to delete specific messages from history, please [contact us](https://ably.com/support) to discuss your requirements. + Messages can be retrieved using the [history](/docs/storage-history/history) feature. This is illustrated in the following diagram: ![Persist All Messages](../../../images/content/diagrams/history-persist-all-messages.png) From cf80031f7134687e906ec53eb6d222d87ed424f0 Mon Sep 17 00:00:00 2001 From: Francis Roberts <111994975+franrob-projects@users.noreply.github.com> Date: Tue, 23 Sep 2025 13:13:26 +0200 Subject: [PATCH 02/14] Is it possible to check if a specific message has been delivered to a device? --- src/pages/docs/messages/index.mdx | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/pages/docs/messages/index.mdx b/src/pages/docs/messages/index.mdx index e30609c3ce..511c4f9306 100644 --- a/src/pages/docs/messages/index.mdx +++ b/src/pages/docs/messages/index.mdx @@ -34,6 +34,12 @@ The following are the properties of a message: | **extras** | A JSON object of arbitrary key-value pairs that may contain metadata, and/or ancillary payloads. Valid payloads include those related to [Push Notifications](/docs/push), [deltas](/docs/channels/options/deltas) and headers | | **encoding** | This is typically empty, as all messages received from Ably are automatically decoded client-side using this value. However, if the message encoding cannot be processed, this attribute contains the remaining transformations not applied to the data payload | +## Message delivery tracking + +You can ensure a message was successfully published by checking the [history](/docs/storage-history/history) of the channel for your message. However, it is only possible to check if a device has received a message from the device itself. + +Ably does not store per-message delivery logs, nor logs of who is subscribed to a channel at any point in time. This means it is not possible to check which users have received messages retroactively. + ## Message conflation Use message conflation to ensure that clients only ever receive the most up-to-date message, by removing redundant and outdated messages. Message conflation will aggregate published messages for a set period of time and evaluate all messages against a [conflation key](#routing). All but the latest message for each conflation key value will be discarded, and the resulting message, or messages, will be delivered to subscribers as a single batch once the period of time elapses. From 48500cda20e272477bc51e2cdf5a2c367c74a1eb Mon Sep 17 00:00:00 2001 From: Francis Roberts <111994975+franrob-projects@users.noreply.github.com> Date: Tue, 23 Sep 2025 13:18:10 +0200 Subject: [PATCH 03/14] Reliable message ordering for connected clients --- .../architecture/message-ordering.mdx | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/pages/docs/platform/architecture/message-ordering.mdx b/src/pages/docs/platform/architecture/message-ordering.mdx index 3cdd3bca52..458799b0a5 100644 --- a/src/pages/docs/platform/architecture/message-ordering.mdx +++ b/src/pages/docs/platform/architecture/message-ordering.mdx @@ -58,3 +58,26 @@ When building globally distributed applications, developers should understand th The dual ordering system allows applications to choose the appropriate consistency model for their needs. For highly interactive applications where responsiveness is critical, realtime order minimizes latency. For applications that require a consistent historical record, the canonical global order provides a stable view that all components can agree on. To support both ordering systems efficiently, messages are stored with metadata that tracks their position in multiple sequences. This allows the platform to rapidly retrieve messages in either realtime or canonical global order as needed for different API operations or connection recovery scenarios. + +## Message ordering guarantees + +Ably ensures that messages are delivered to persistently connected subscribers on a channel in the order they were published when using the Realtime libraries. Each message sent on a realtime connection has a unique incrementing serial number, enabling this ordering guarantee. + +For example, if you publish messages using a Realtime client library, subscribers on that channel anywhere in the world will receive those messages in the same order they were originally published. + +### REST publishing considerations + +When publishing using REST client libraries, Ably still guarantees that the order messages are received will be honoured for all subscribers. However, if you are sending messages at a high rate with separate REST requests, it is possible that a later HTTP request may reach Ably before a previous request due to variable network factors outside of Ably's control. + +If ordering is important to you when using REST, consider these approaches: +- Rate limit your HTTP requests +- [Batch messages](/docs/messages/batch) together in a single request +- Use Realtime client libraries to publish messages instead + +### Connection recovery edge cases + +Ably supports connection state recovery, so even if a connection is lost and re-established, messages replayed when reconnected will be in sequential order. + +However, there is a rare situation where ordering may not be maintained: if a client is disconnected and the server maintaining that client's connection state is terminated or recycled during connection state recovery, messages may potentially be replayed out of order. This behaviour is by design to ensure that the backlog of messages held in the connection state do not prevent the new server from sending realtime messages to clients. + +If message ordering is required in all cases, consider catching the connection disconnected event and either re-establishing a new connection or manually re-ordering incoming messages following disconnection. From ee1d62856e2a81a427758c9e321a83e5ef507291 Mon Sep 17 00:00:00 2001 From: Francis Roberts <111994975+franrob-projects@users.noreply.github.com> Date: Tue, 23 Sep 2025 14:10:28 +0200 Subject: [PATCH 04/14] Client-specified message ID restrictions for multiple messages published atomically --- src/pages/docs/pub-sub/advanced.mdx | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/src/pages/docs/pub-sub/advanced.mdx b/src/pages/docs/pub-sub/advanced.mdx index 46f2c736be..168c19edaa 100644 --- a/src/pages/docs/pub-sub/advanced.mdx +++ b/src/pages/docs/pub-sub/advanced.mdx @@ -954,7 +954,32 @@ In some cases you may wish to set the unique message ID yourself to achieve idem * To ensure idempotency when a publisher instance might be restarted, and continuous activity cannot be guaranteed. * To integrate with an upstream system that uses message IDs, to ensure idempotency across an entire message processing pipeline. -If setting your own message IDs be aware of the [restrictions](https://faqs.ably.com/client-specified-message-id-restrictions-for-multiple-messages-published-atomically) on its format when publishing messages atomically. +If setting your own message IDs be aware of the restrictions on its format when publishing messages atomically. + +#### Client-specified message ID restrictions for multiple messages published atomically + +When publishing multiple messages in a single publish request (via REST or realtime), those messages are published atomically by Ably. They are bundled together, and will either all succeed or all fail. + +##### Idempotency with atomic publishes + +From the point of view of idempotency, there is only a single idempotency 'key' for the entire array of messages. Ably prevents you from including a different, unrelated `id` in each message to avoid the mistaken impression that the messages will be individually idempotent. + +For example, if Ably permitted `publish([{ id: 'foo', data: 'first' },{ id: 'bar', data: 'second' }])`, it might seem like you could later call `publish([{ id: 'bar', data: 'second' }])` and the second publish would be ineffective due to idempotency; but that is not the case. + +##### ID format requirements + +For messages published atomically, all messages must have an `id` derived from a single base ID. Each message must contain an `id` of the form `:` where `idx` is a zero-based index into the array of messages. + +For example, to publish 3 messages with a base ID of `foo`, the messages must have IDs `foo:0`, `foo:1`, `foo:2`. This emphasizes that the messages share a single 'idempotency key' (which will be `foo`). + +##### Publishing messages with separate idempotency + +If you want messages to be separately and individually idempotent, you should publish them non-atomically: + +* Issue multiple `publish()` calls serially instead of a single `publish()` call with an array of messages. + +* If publishing over REST and you want all publishes to happen with a single HTTP request, use the batch publishing API. Each message is contained in its own `batchspec` object. All messages in a single `batchspec` are published atomically (for each channel in that `batchspec`), so by specifying multiple `batchspecs`, you can publish messages non-atomically, allowing you to specify a different idempotency ID for each message. +