Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gNMI config subscription extension documentation #213

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

gNMI config subscription extension documentation #213

Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f6d20ec
gNMI config subscription extension documentation
karimra Jul 29, 2024
532ed52
addressing comments
karimra Aug 20, 2024
ddac514
Update rpc/gnmi/gnmi-config-subscriptions.md
karimra Dec 4, 2024
56c5312
Update rpc/gnmi/gnmi-config-subscriptions.md
karimra Dec 4, 2024
7c754d7
updates based on comments
karimra Dec 5, 2024
f53dbd6
add `done` field
karimra Dec 9, 2024
d0a4930
add appendix about concurrent commits handling
karimra Jan 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 187 additions & 0 deletions rpc/gnmi/gnmi-config-subscriptions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
# gNMI Config Subscription Extension

**Contributors:** Roman Dodin, Matthew MacDonald

**Date:** July 29th, 2024

**Version:** 0.1.0

## 1. Purpose

Performing configuration management and handling configuration drift is
robshakir marked this conversation as resolved.
Show resolved Hide resolved
one of the main features of a higher-level management system or orchestrator.
The configuration management tasks are not concerned about the state data
and focus on effective retrieval and push of the configuration values.

Thus, having a synchronized configuration view between the management
system and the network elements is key to enabling robust and near real-time
configuration management.
To enable this synchronization of configuration data, gNMI Subscribe RPC
karimra marked this conversation as resolved.
Show resolved Hide resolved
can be used. The bidirectional streaming nature of this RPC enables fast
and reliable sync between the management system and the devices it manages.

Unfortunately, gNMI Subscribe RPC does not have an embedded mechanism to
stream updates for the configuration values only as opposed to the Get RPC,
which makes this RPC rather ineffective on YANG schemas that do not employ
a separation of config and state elements by using distinct containers.

This proposal introduces the Config Subscription extension that allows clients
to indicate to servers that they are interested in configuration values only.

gNMI Config Subscription Extension proto specification is defined in
[gnmi_ext.proto](
https://github.com/openconfig/gnmi/blob/master/proto/gnmi_ext/gnmi_ext.proto
).

## 2. Definition

A `ConfigSubscription` message is embedded as an extension message in the
`SubscribeRequest` or `SubscribeResponse` proto messages.
If the extension is embedded in a `SubscribeRequest`, the action field
must be a `ConfigSubscriptionStart`.
The presence of such an extension indicates to the target that the client
wants to start a ConfigSubscription. The target must return notifications
pertaining to data leaves that the target considers to be writable.
If the subscription type is `ON_CHANGE`, the target must separate the
notifications triggered by different commits using a
`ConfigSubscriptionSyncDone` in a `SubscribeResponse` message.
On the other hand, if the extension is embedded in a `SubscribeResponse`, the
action field must be a `ConfigSubscriptionSyncDone`. This action is used by a
target to indicate a commit boundary to the client.

## 2.1 Proto

The extension contains a message called `ConfigSubscription` that carries
one of 2 types of actions. `ConfigSubscriptionStart` or
`ConfigSubscriptionSyncDone`

```proto
// ConfigSubscription extension allows clients to subscribe to configuration
// schema nodes only.
message ConfigSubscription {
oneof action {
// ConfigSubscriptionStart is sent by the client in the SubscribeRequest
ConfigSubscriptionStart start = 1;
// ConfigSubscriptionSyncDone is sent by the server in the SubscribeResponse
ConfigSubscriptionSyncDone sync_done = 2;
}
}

// ConfigSubscriptionStart is used to indicate to a target that for a given set
// of paths in the SubscribeRequest, the client wishes to receive updates
// for the configuration schema nodes only.
message ConfigSubscriptionStart {}

// ConfigSubscriptionSyncDone is sent by the server in the SubscribeResponse
// after all the updates for the configuration schema nodes have been sent.
message ConfigSubscriptionSyncDone {
// ID of a commit confirm operation as assigned by the client
// see Commit Confirm extension for more details.
string commit_confirm_id = 1;
// ID of a commit as might be assigned by the server
// when registering a commit operation.
string server_commit_id = 2;
}
```

## 2.2 Actions

### 2.2.1 ConfigSubscriptionStart

A `ConfigSubscriptionStart` message is used by a gNMI client in a
`SubscribeRequest` to indicate that it wants to start a ConfigSubscription.
The target must respond exclusively with configuration data relevant to the
dplore marked this conversation as resolved.
Show resolved Hide resolved
created subscription.
The target must respond with an initial set of updates followed by a
karimra marked this conversation as resolved.
Show resolved Hide resolved
`SubscribeResponse` where the `sync_update` field is set to `true`.
karimra marked this conversation as resolved.
Show resolved Hide resolved
However, if the `updates_only` field in the `SubscribeRequest` was set to
`true`, the target should skip sending the initial updates and only send
a `SubscribeResponse` with `sync_update` set to `true` as per the base gNMI
robshakir marked this conversation as resolved.
Show resolved Hide resolved
specification.

### 2.2.2 ConfigSubscriptionSyncDone

A `ConfigSubscriptionSyncDone` message is used by a gNMI target in a
`SubscribeResponse` to indicate a commit boundary to the client.
A commit boundary marks the completion point of a specific set of
robshakir marked this conversation as resolved.
Show resolved Hide resolved
configuration changes.
It indicates that all changes within that set have been committed and
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does committed mean here? To me, I would interpret this as meaning that all of the values have been applied to the intended configuration -- but there seems a need to define these semantics clearly to allow for interoperable implementations.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Committed means the interface (gNMI, netconf, CLI,...) that was used to change the values returned a nil/zero error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's clarify this in the text.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added:

It indicates that all changes within that set have been committed -- meaning
the mechanism used to apply the changes reported no errors -- and
that all notifications triggered by the commit have been streamed to
the client.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the fact that the 'commit' semantics are defined by the definition per interface -- so can we say "gNMI, NETCONF, CLI etc., returned a non-zero value" explicitly?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

that all notifications triggered by the commit have been streamed to
the client.

The `ConfigSubscriptionSyncDone` message includes two optional fields:

* `commit_confirm_id`: A commit confirm ID assigned by the client which
initiated the commit.
The commit can be initiated via gNMI (using the CommitConfirmed Extension),
NETCONF, or any other management interface. Applicable only if the commit
confirmed option is used.
* `server_commit_id`: An optional internal ID assigned by the target.

In the case a commit happens before the `sync_response: true` and the server
robshakir marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a second thread here -- what happens if we have two configuration changes -- i.e.:

  • client A makes a change
  • client B makes a change
  • client C has a ConfigSubscription open

It seems like, in this case, sync_response: true might be returned at any point in the lifetime of the subscription (when whatever existed in the data tree at the time client C created its subscription). We want ConfigSubscriptionSyncDone to be returned at the point that client A and client B have had their changes "committed" (let's clarify semantics here) -- this is rather independent to when sync_response: true is sent.

Another implication here is that we cannot tell the exact provenance of an update within the Subscribe stream if there are changes being made that do not block on the "completion" of a previous change.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal here is to delay any ConfigSubscriptionSyncDone until after sync_response: true is received.
As for overlapping updates from 2 different commits, typically config stores do not support parallel commits. Assuming updates to be streamed are added to the send buffer in sequence, I see little chance for overlapping updates.

If we want to keep the door open for concurrent commits we have to add the commit_confirmed_id and and server_commit_id to every update. This won't solve the case none of those 2 IDs exist like for a change from the CLI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the config store does not need to support parallel commits for this situation to happen.

assume we take N seconds to stream all updates corresponding to a config change.

t=0 - config change 1 happens and is finished
t=1 - streaming for updates from config change 1 starts
t=Y (Y<N) - config change 2 happens and is finished
t=Y+1 - streaming for updates from config change 2 starts
t=N - streaming for updates from config change 1 ends
t=Y+N - streaming for updates from config change 2 ends

in this scenario -- it's the streaming of the updates that overlaps, not the commits.

when the client receives the ConfigSubscriptionSyncDone with the commit ID of config change 1 in it, it may already have received updates for paths that were impacted only by config change 2 -- so the semantic that this marker means "you have the state that reflects the state only after change 1" is broken.

the only way to avoid this is by saying that the config store locks until all updates are streamed (i.e., N seconds) (which seems bad for performance), or there is a strict send buffer with no coalescing possible (which I don't think is the right behaviour -- and is already broken by existing gnmi implementations).

why does coalescing matter? well, if both change 1 and change 2 change /foo/bar and the update for /foo/bar hasn't been sent, then it's acceptable for an gNMI server to coalesce the two, and only send the value for /foo/bar that corresponds to the value after the second change.

i think the semantics that you're trying to define here for ConfigSubscriptionSyncDone are problematic with any kind of concurrency. to me, it's better to say that the gNMI server gives looser semantics about what it can guarantee -- i.e., it guarantees only that it is "done" with processing the changes that are covered by a particular commit. if the client really wants to ensure that its config subscription reflects exactly that intended state, it can implement locking between changes without adding complexity at the server side.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this definition works: it guarantees only that it is "done" with processing the changes that are covered by a particular commit.. The streaming of changes depends on whether they were coalesced.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK -- suggest to update the text here to describe that this is the semantic meaning of receiving a "done". I might also suggest adding the example of concurrency that means that this design decision is required in an appendix for future readers.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and added appendix

may include the committed changes together with the initial updates but has
to send the related `ConfigSubscriptionSyncDone` after sending the
`sync_response: true` response.

## 3. Configuration changes scenarios

### 3.1 Configuration changes without Commit Confirmed

1) The client subscribes to path P1 with the `ConfigSubscription` extension
present with the action `ConfigSubscriptionStart`.
2) The server processes the subscription request as usual but will only send
updates for the configuration schema nodes under the path P1.
3) The client sends a Set RPC with the configuration changes to the path P1
**without** the `CommitConfirm` extension.
4) The server processes the Set RPC as usual and sends the updates for the
configuration schema nodes under the path P1.
5) After all the configuration updates are sent, the server sends the
`ConfigSubscriptionSyncDone` message to the client in a SubscribeResponse
message.
This message does not contain a `commit_confirmed_id` and may contain a
`server_commit_id`
robshakir marked this conversation as resolved.
Show resolved Hide resolved

### 3.2 Configuration changes with Commit Confirmed

1) The client subscribes to the path P1 with the `ConfigSubscription`
extension present with the action `ConfigSubscriptionStart`.
2) The server processes the subscription request as usual but will only send
updates for the configuration schema nodes under the path P1.
3) The client sends a Set RPC with the configuration changes to the path
P1 and **with** the `CommitConfirm` extension present.
4) The server processes the Set RPC as usual and sends the updates for
the configuration schema nodes under the path P1.
5) As all the configuration updates are sent, the server sends the
`ConfigSubscriptionSyncDone` message to the client in a SubscribeResponse
message.
This message must contain the the value of the `commit_confirmed_id`
received in the Set RPC in step 4 and may contain a `server_commit_id`.
6) When the client sends the commit confirm message, the server confirms
the commit and does not send any extra SubscribeResponse messages with the
`ConfigSubscriptionSyncDone` message.

### 3.3 Configuration changes with Commit Confirmed and rollback/cancellation

1) The client subscribes to path P1 with the `ConfigSubscription` extension
present with the action `ConfigSubscriptionStart`.
2) The server processes the subscription request as usual but will only send
updates for the configuration schema nodes under the path P1.
3) The client sends a Set RPC with the configuration changes to the path P1
and **with** the `CommitConfirm` extension present.
4) The server processes the Set RPC as usual and sends the updates for the
configuration schema nodes under the path P1.
5) After all the configuration updates are sent, the server sends the
`ConfigSubscriptionSyncDone` message to the client in a SubscribeResponse
message.
This message must contain the the value of the `commit_confirmed_id` received
in the Set RPC in step 4.
6) When the commit confirmed rollback timer expires or a commit cancel message
is received, the server:
i. rolls back the configuration changes as per the Commit Confirm extension
specification.
ii. sends the new configuration updates for the path P1 as the configuration
has changed/reverted.
iii. sends the ConfigSubscriptionSyncDone message to the client in a
`SubscribeResponse` message.
This message must contain the the value of the `commit_confirmed_id`
received in the Set RPC in step 4 and may contain a `server_commit_id`.