diff --git a/doc/SONiC_OC_NTP_HLD.md b/doc/SONiC_OC_NTP_HLD.md new file mode 100644 index 0000000000..b29cb8d564 --- /dev/null +++ b/doc/SONiC_OC_NTP_HLD.md @@ -0,0 +1,1014 @@ +# Feature Name +NTP Support in Management Framework + +# High Level Design Document + +#### Rev 0.5 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 05/03/2020 | Bing Sun | Initial version | +| 0.2 | 06/15/2020 | Bing Sun | Update based on comments | +| 0.3 | 09/21/2020 | Bing Sun | Add dhcp behavior | +| 0.4 | 11/02/2020 | Bing Sun | Add support for NTP authentication| +| 0.5 | 11/08/2020 | Bing Sun | Allow configuration of multiple NTP source interfaces| +| 0.6 | 12/07/2020 | Bing Sun | Updated section 3.2 configDB changes | +| | | | Minor change for Yang models | + + +# About this Manual + +This document introduces the support of NTP configuration using management framework. It also describes the corresponding backend NTP configuration changes(/etc/ntp.conf and /etc/ntp.keys) as well as ntp service restart upon configuration changes. + +# Scope + +This document covers NTP "configuration" and "show" commands based on the OpenConfig YANG model. In addition, it decribes the backend mechanism required to support each command. +A summary of NTP unit test cases is presented at the end. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| NTP | Network Time Protocol | +| ntpd | NTP Daemon | +| mgmt VRF | Management VRF | + + +# 1 Feature Overview + +NTP stands for Network Time Protocol. It is used to synchronize the time of a computer or server to another server or reference time source. + +Today, SONiC click CLI provides commands to +- add and delete remote NTP servers with IPv4 or IPv6 addresses +- display NTP synchronization status with show command (output of "ntpq -pn") + +This feature provides the the same above mentioned capabilities via Management CLI, REST and gNMI using OpenConfig YANG models. +In addition, it provides the following configuration, +- add remote NTP server with hostname +- NTP source interfaces +- NTP vrf +- NTP authentication + +## 1.1 Requirements + +### 1.1.1 Front end configuration and get capabilities + +#### 1.1.1.1 add/delete NTP server +``` +ntp server 99.1.1.1 +ntp server pool.ntp.org +``` +Add/delete NTP server information in the Redis ConfigDB and in /etc/ntp.conf. +The NTP server can be IPv4 address, IPv6 address , or hostname. +Mutliple NTP servers can be configured. + +#### 1.1.1.2 add/delete NTP source interface +``` +ntp source-interface Ethernet36 +``` + +``` +ntp source-interface PortChannel 100 +``` + +``` +ntp source-interface Vlan 100 +``` + +``` +ntp source-interface Management 0 +``` + + +Add/delete the global NTP source interface in the Redis ConfigDB and in /etc/ntp.conf. The ip address of this interface will be used by ntpd as source ip for all NTP packets. +Multiple NTP source interfaces can be configured. +Following interface types can be used as NTP source interface, +- Ethernet interface +- PortChannel +- Vlan interface +- Loopback interfacee +- eth0(management interface) + +#### 1.1.1.3 add/delete VRF name +``` +ntp vrf default +``` + +``` +ntp vrf mgmt +``` + +Add/delete the global NTP VRF information in the Redis ConfigDB. It is used by /etc/init.d/ntp script to start the ntpd in a specific VRF context. +For this release, only Management VRF and default instance are supported. + +#### 1.1.1.4 Get NTP association +``` +show ntp association +``` + +This command displays the output of "ntpq -np" command. + +#### 1.1.1.5 Overall Behavior related to NTP source interface and NTP vrf +##### When mgmt VRF is configured +a.if no ntp vrf is configured, ntp service starts in mgmt VRF context by default +b.if "mgmt" is configured as NTP vrf, ntp service starts in mgmt VRF context +c.if "default" is again configured as NTP vrf, ntp service starts in default vrf context + +##### When mgmt VRF is not configured +ntp service always starts in default vrf context + +##### NTP source interface related +a.if a NTP source interface has IP address configured, the IP address will be used as source ip for all NTP packets exchanged with the respective NTP servers/clients +b.if a NTP source interface has no IP address configured, it is not being considered as an NTP source interface + +#### 1.1.1.6 NTP authentication configuration +NTP authentication enables an NTP client or peer to authenticate time received from their servers and peers. + +##### ntp authenticate +``` +ntp authenticate +``` + +NTP server and NTP client use this command to enable the NTP authentication feature. + +##### ntp authentication-key +``` +ntp authentication-key 1 md5 "ntp client 1" + +ntp authentication-key 2 md5 ntp_client2 +``` + +NTP client uses this command to define an authentication key with key number, authentication type and password. +The key number is from 1 to 65535. +The authentication type supported is MD5, SHA1 and SHA2-256. +The password is configured with plaintext the first time. In runnning-configuration, it is encrypted and is indicated with "encrypted" at the end. Authentication key can then be configured with the encrypted format and "encrypted" flag. + + +##### ntp trusted key +``` +ntp trusted-key 1 + +ntp trusted-key 2 +``` + +NTP server uses this command to add a list of key numbers that the NTP server must provide in its NTP packets in order for the NTP clients to synchronize to it. +NTP client must configure a list with this command if it desires to authenticate the NTP server with any of the key number. + +##### ntp server key +``` +ntp server 99.1.1.1 key 1 +``` + +NTP client uses this command to configure the key expected from a specific NTP server. + +### 1.1.2 Backend mechanisms to support configuration and get + +#### 1.1.2.1 add/delete NTP server +This creates or deletes a NTP server entry in the Redis ConfigDB. + +``` + "NTP_SERVER|10.11.0.1": { + "type": "hash", + "value": { + "NULL": "NULL" + } + }, + "NTP_SERVER|2001:aa:aa::a": { + "type": "hash", + "value": { + "NULL": "NULL" + } + }, + "NTP_SERVER|pool.ntp.org": { + "type": "hash", + "value": { + "NULL": "NULL" + } + } +``` + +A change in the NTP_SERVER entry triggers hostcfgd to start the NTP configuration script, which in turn writes each NTP server to the ntp.conf and then restart the ntp service. + + +#### 1.1.2.2 add/delete NTP source + +This creates or deletes a global NTP source interface entry in the Redis ConfigDB. + +``` + "NTP|global": { + "type": "hash", + "value": { + "src_intf": "Ethernet36" + } + } +``` + +A change in this entry triggers hostcfgd to start the NTP configuration script, which in turn writes the ntp source interface to the ntp.conf and then restart the ntp service. + +SONiC click CLI can be extended to include this configuration. + + +#### 1.1.2.3 add/delete NTP VRF + +This creates or deletes a global NTP vrf entry in the Redis ConfigDB. For this release, it can only be "mgmt" or "default". + +``` + "NTP|global": { + "type": "hash", + "value": { + "vrf": "mgmt" + } + } +``` + +A change in this DB entry triggers hostcfgd to restart ntp service. + +SONiC click CLI can be extended to include this configuration. + + +#### 1.1.2.4 get NTP associations + +Transformer function issues "ntpq -pn" command, parses the response and maps the outputs to the OpenConfig system YANG NTP states. + +#### 1.1.2.5 NTP authentication + +##### 1.1.2.5.1 Enable or disable ntp authenticate + +When "authenticate" is enabled, "enable-ntp-auth" field is set to "true" in the NTP global entry, + +``` +"NTP|global": { + "type": "hash", + "value": { + "enable-ntp-auth": "true", + } +} +``` + +This change triggers /etc/ntp.conf to get generated with the line indicating where to find the configured keys +``` +key /etc/ntp.keys +``` + +When "authenticate" is removed, the same attribute is set to "false". The file /etc/ntp.conf is generated without "key /etc/ntp.keys" but with the line +``` +disable auth +``` + +The file /etc/ntp.keys will be created with the configured authentication keys if "authenticate" is enabled, and removed if "authenticate" is disabled. + +##### 1.1.2.5.2 Add or delete ntp authentication key + +When an authentication key is configured with a key number, authentication type and password in plaintext, a transformer function will change the plaintext password to the encrypted format and puts the key in an NTP_AUTHENTICATION_KEY ConfigDB entry. A boolean "key_encrypted" is set to true and added in the same entry as well. This is done so that "show running-configuration" from CLI or GET from REST/gNMI will be able to display the password in encrypted format. For example, +``` + "NTP_AUTHENTICATION_KEY|1": { + "type": "hash", + "value": { + "key_encrypted": "true", + "key_type": "MD5", + "key_value": "3b88c0eb8406a9e76722b84baf1d94e5e185eb7f64f8dd46c759719c33557876" + } + } +``` + +If "authenticate" is enabled, the file /etc/ntp.keys is populated with the configured authentication keys. The password in this file is in the plaintext format. Only root user can read /etc/ntp.keys. + +When an ntp authentication key is removed, the ConfigDb and /etc/ntp.keys will be updated accordingly. + +##### 1.1.2.5.3 Add or delete ntp trusted key + +When a ntp trusted key number is configured, the key number is added to the "trustedkeys" list in the NTP global entry, e.g +``` + "NTP|global": { + "type": "hash", + "value": { + "authenticat": enabled, + "trustedkeys@": "1,2" + } + } +``` + +/etc/ntp.conf will be generated with the line +``` +trustedkey 1 2 +``` + +When a ntp trusted key number is removed, the key number is removed from the "trustedkey" list. + + +##### 1.1.2.5.4 Add a key for NTP server + +When a ntp server is created with a key number, the "key_id" with the key number will be added as a field for the NTP server ConfigDb entry, e.g +``` +"NTP_SERVER|99.1.1.1": { + "type": "hash", + "value": { + "key_id": "1" + } + } +``` + +The file /etc/ntp.conf will be generated with the same key number for that NTP server, e.g +``` +server 99.1.1.1 iburst key 1 +``` + +##### 1.1.2.5.5 Sample NTP authentication CLI commands on NTP server and NTP client +``` +NTP master ------------------------- SONiC switch ----------------------------------server +Mgmt. IP: 100.94.121.15 mgmt. IP: 100.94.122.16 + Loopback100: 2001:aa:aa::1 + +``` + +Here the SONiC switch is a NTP client to the NTP master. It is also a NTP server to the downstream servers. +As a NTP client, SONiC switch uses NTP authentication to validate its NTP server. +As a NTP server, the downstream servers are the NTP client. It is up to the NTP client (server) whether NTP authentication is desired with its NTP server (SONiC switch). +SONiC switch can server as a NTP client and a NTP server simultaneously, with or without NTP authentication with either a remote NTP server or NTP client. +The NTP master reaches SONiC switch via its management interface. The downstream servers reach the SONiC switch via its front panel ports. + +###### Relevant CLI commmands on SONiC switch as NTP server + +``` +sonic(config)# ntp source-interface Loopback 100 +sonic(config)#ip vrf mgmt +``` + +###### Relevant CLI commands on SONiC switch as NTP client +``` +sonic(config)#ntp authenticate +sonic(config)#ntp authentication-key 1 MD5 force +sonic(config)#ntp trusted-key 1 +sonic(config)#ntp server 100.94.121.15 key 1 +sonic(config)# ntp source-interface Management 0 +``` + +###### Relevant CLI commands on the server as NTP client without NTP authentication +``` +sonic(config)#ntp server 2001:aa:aa::1 +sonic(config)# ntp source-interface Vlan 100 +``` + +###### Relevant CLI commands for the server and SONiC switch with NTP authentication +``` +On the SONiC switch + +sonic(config)#ntp authenticate +sonic(config)#ntp authentication-key 2 MD5 jungle +sonic(config)#ntp trusted-key 2 +sonic(config)# ntp source-interface Loopback 100 +sonic(config)#ip vrf mgmt +``` + +``` +On the server + +sonic(config)#ntp authenticate +sonic(config)#ntp authentication-key 2 MD5 jungle +sonic(config)#ntp trusted-key 2 +sonic(config)#ntp server 2001:aa:aa::1 key 2 +sonic(config)# ntp source-interface Vlan 100 + +``` + +### 1.1.3 Functional Requirements + +Provide management framework support to +- configure NTP server +- configure NTP source interface +- configure NTP vrf +- configure NTP authentication + +### 1.1.4 Configuration and Management Requirements +- CLI style configuration and show commands +- REST API support +- gNMI Support + +Details described in Section 3. + +### 1.1.5 Configurations not supported by this feature using management framework: +- configure local server as a NTP master +- broadcast mode + +### 1.1.6 Scalability Requirements +Interface listening will not be enabled on all L3 interfaces. User can configure NTP source interfaces which are only a few. +Ntpd runs in one VRF context, default vrf or mgmt vrf. +Multiple ntp servers supported, also only a few. + +### 1.1.7 Warm Boot Requirements +NA + +## 1.2 Design Overview + +### 1.2.1 Basic Approach +Implement NTP support using transformer in sonic-mgmt-framework. + +### 1.2.2 Container +The front end code change will be done in management-framework container including: +- XML file for the CLI +- Python script to handle CLI request (actioner) +- Jinja template to render CLI output (renderer) +- front-end code to support "show running-configuration" +- OpenConfig YANG model for NTP openconfig-system.yang and openconfig-system-ext.yang +- SONiC NTP model for NTP based on Redis DB schema of NTP +- transformer functions to + * convert OpenConfig YANG model to SONiC YANG model for NTP related configurations + * convert from Linux command "ntpq -p" output to OpenConfig NTP state YANG model + +### 1.2.3 SAI Overview + +# 2 Functionality + +## 2.1 Target Deployment Use Cases +Manage/configure NTP via gNMI, REST and CLI interfaces. + +## 2.2 Functional Description +Provide CLI, gNMI and REST supports for NTP related configurations. + +## 2.3 Backend change to support new configurations +Provide changes in hostcfgd, ntp.conf.j2, ntp.keys.j2 and /etc/init.d/ntp. +SONiC click CLI enhancement if possible. + +## 2.4 Behavior when Management IP Address is acquired via DHCP +If the management IP address is acquired via DHCP, and if the NTP server option specifies the NTP server, /etc/dhcp/dhclient-exit-hooks.d/ntp script will generate the file /var/lib/ntp/ntp.conf.dhcp. This file is a copy of the default /etc/ntp.conf with a modified server list from the DHCP server. +NTP daemon only uses one of the 2 files, and /var/lib/ntp/ntp.conf.dhcp takes precedence over the default /etc/ntp.conf. It is the existing behavior and is out of the scope of this HLD. + +NTP source-interface, NTP vrf and NTP authentication discussed in the HLD are only guaranteed to take effect on the static configured NTP servers. +For acquired NTP servers from DHCP server, NTP source-interface, NTP vrf and NTP authentication will only take effect if /var/lib/ntp/ntp.conf.dhcp is generated based on the /etc/ntp.conf with user configured NTP source-interface and NTP authentication. + +Applying the configured NTP source-interface, NTP vrf and NTP authentication to acquired NTP servers from the DHCP server is not a requirement for this release. + +# 3 Design + +## 3.1 Overview + +Enhancing the management framework backend code and transformer methods to add support for NTP. + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB +This feature will allow users to make NTP configuration changes to CONFIG DB, and get NTP configurations. + +``` +NTP server + + "NTP_SERVER": { + "2.2.2.2": { + "key_id": "1" + }, + "3.3.3.3": { + "key_id": "2" + }, + "4.4.4.4": { + "key_id": "3" + }, + "10.14.8.140": {} + } +``` + +``` +NTP authentication key + + "NTP_AUTHENTICATION_KEY": { + "1": { + "encrypted": "true", + "type": "MD5", + "value": "U2FsdGVkX18LP3kIv47lRKCboUop/+0YyacH2UT2WJ0=" + }, + "2": { + "encrypted": "true", + "type": "SHA1", + "value": "U2FsdGVkX1+DU7geMDXVvCOJjZQyP1zTT4vRbHFqsZo=" + }, + "3": { + "encrypted": "true", + "type": "SHA2_256", + "value": "U2FsdGVkX19yHcvrGFDKJb80FRY+cnmO1+yv6SGkao8=" + } + } +``` + +``` +NTP global configuration + + "NTP": { + "global": { + "auth_enabled": "true", + "src_intf": [ + "eth0", + "Loopback99" + ], + "trusted_key": [ + "1", + "2", + "3" + ], + "vrf": "mgmt" + } + } +``` + + +### 3.2.2 APP DB + +### 3.2.3 STATE DB + +### 3.2.4 ASIC DB + +### 3.2.5 COUNTER DB + +## 3.3 Switch State Service Design + +### 3.3.1 Orchestration Agent + +### 3.3.2 Other Process + +## 3.4 SyncD + +## 3.5 SAI + +## 3.6 User Interface + +### 3.6.1 Data Models + +YANG models needed for NTP handling in the management framework: +1. **openconfig-system.yang** + +2. **openconfig-system-ext.yang** + +3. **sonic-system-ntp.yang** + +Supported yang objects and attributes: +```diff + +--rw ntp + | +--rw config + | | +--rw enabled? boolean ++ | | +--rw enable-ntp-auth? boolean ++ | | +--rw oc-sys-ext:ntp-source-interface* oc-if:base-interface-ref ++ | | +--rw oc-sys-ext:vrf? string ++ | | +--rw oc-sys-ext:trusted-key* uint16 + | +--ro state + | | +--ro enabled? boolean ++ | | +--ro enable-ntp-auth? boolean + | | +--ro auth-mismatch? oc-yang:counter64 ++ | | +--ro oc-sys-ext:ntp-source-interface? oc-if:base-interface-ref ++ | | +--ro oc-sys-ext:vrf? string ++ | | +--rw oc-sys-ext:trusted-key* uint16 ++ | +--rw ntp-keys ++ | | +--rw ntp-key* [key-id] ++ | | +--rw key-id -> ../config/key-id ++ | | +--rw config ++ | | | +--rw key-id? uint16 ++ | | | +--rw key-type? identityref ++ | | | +--rw key-value? string ++ | | | +--rw oc-sys-ext:encrypted? boolean ++ | | +--ro state ++ | | +--ro key-id? uint16 ++ | | +--ro key-type? identityref ++ | | +--ro key-value? string ++ | | +--rw oc-sys-ext:encrypted? boolean + | +--rw servers + | +--rw server* [address] ++ | +--rw address -> ../config/address ++ | +--rw config ++ | | +--rw address? oc-inet:host + | | +--rw port? oc-inet:port-number + | | +--rw version? uint8 + | | +--rw association-type? enumeration + | | +--rw iburst? boolean + | | +--rw prefer? boolean ++ | | +--rw oc-sys-ext:key-id? uint16 ++ | +--ro state ++ | +--ro address? oc-inet:host + | +--ro port? oc-inet:port-number + | +--ro version? uint8 + | +--ro association-type? enumeration + | +--ro iburst? boolean + | +--ro prefer? boolean ++ | +--ro stratum? uint8 + | +--ro root-delay? uint32 + | +--ro root-dispersion? uint64 + | +--ro offset? uint64 ++ | +--ro poll-interval? uint32 ++ | +--rw oc-sys-ext:key-id? uint16 ++ | +--ro oc-sys-ext:peerdelay? decimal64 ++ | +--ro oc-sys-ext:peeroffset? decimal64 ++ | +--ro oc-sys-ext:peerjitter? decimal64 ++ | +--ro oc-sys-ext:selmode? string ++ | +--ro oc-sys-ext:refid? inet:host ++ | +--ro oc-sys-ext:peertype? string ++ | +--ro oc-sys-ext:now? uint32 ++ | +--ro oc-sys-ext:reach? uint8 +``` + +```diff +module: sonic-system-ntp ++ +--rw NTP ++ | +--rw NTP_LIST* [global_key] ++ | +--rw global_key enumeration ++ | +--rw src_intf* union ++ | +--rw vrf? union ++ | +--rw auth_enabled? boolean ++ | +--rw trusted_keys* -> /sonic-system-ntp/NTP_AUTHENTICATION_KEY/NTP_AUTHENTICATION_KEY_LIST/id ++ +--rw NTP_AUTHENTICATION_KEY ++ | +--rw NTP_AUTHENTICATION_KEY_LIST* [id] ++ | +--rw id uint16 ++ | +--rw type enumeration ++ | +--rw value string ++ | +--rw encrypted? boolean ++ +--rw NTP_SERVER ++ +--rw NTP_SERVER_LIST* [server_address] ++ +--rw server_address inet:host ++ +--rw key_id? -> /sonic-system-ntp/NTP_AUTHENTICATION_KEY/NTP_AUTHENTICATION_KEY_LIST/id +``` + +### 3.6.2 CLI + + +#### 3.6.2.1 Configuration Commands +All commands are executed in `configuration-view`: +``` +sonic# configure terminal +sonic(config)# + +sonic(config)# ntp + authenticate Authenticate time sources + authentication-key Authentication key for trusted time sources + server Configure NTP server + source-interface Configure NTP source interface to pick the source IP, used for the NTP packets + trusted-key Key numbers for trusted time sources + vrf Enable NTP on VRF + +``` + +##### 3.6.2.1.1 Configure NTP server +``` +sonic(config)#ntp + server Configure NTP server +sonic(config)#ntp server +String NTP server address or name + +sonic(config)# ntp server 10.11.0.1 + +sonic(config)# ntp server 2001:aa:aa::a + +sonic(config)# ntp server pool.ntp.org + +``` + +##### 3.6.2.1.2 Delete NTP server + +``` +sonic(config)# no ntp server + String NTP server address or name + +sonic(config)# no ntp server 10.11.0.1 + +sonic(config)# no ntp server 2001:aa:aa::a + +sonic(config)# no ntp server pool.ntp.org + +``` + +##### 3.6.2.1.3 Configure NTP source interface + +``` +sonic(config)# ntp source-interface + Ethernet Ethernet interface + Loopback Loopback interface + Management Management Interface + PortChannel PortChannel interface + Vlan Vlan interface + +sonic(config)# ntp source-interface Ethernet 48 +sonic(config)# +sonic(config)# + +sonic(config)# ntp source-interface Loopback 100 +sonic(config)# +sonic(config)# +sonic(config)# +sonic(config)# ntp source-interface Management 0 +sonic(config)# +sonic(config)# +sonic(config)# +sonic(config)# ntp source-interface PortChannel 100 +sonic(config)# +sonic(config)# +sonic(config)# +sonic(config)# ntp source-interface Vlan 100 +sonic(config)# + +``` + +##### 3.6.2.1.4 Delete NTP source interface + +``` +sonic(config)# no ntp source-interface PortChannel 100 + +``` + +``` +sonic(config)# no ntp source-interface +``` + +##### 3.6.2.1.5 Configure NTP vrf + +``` +sonic(config)# + vrf Enabling NTP on a VRF + +sonic(config)#ntp vrf + mgmt Enable NTP on management VRF + default Enable NTP on default VRF + +``` + +##### 3.6.2.1.6 Delete NTP vrf + +``` +sonic(config)# no ntp + vrf Disable NTP on a VRF + +sonic(config)# no ntp vrf + +``` + +##### 3.6.2.1.7 Enable NTP authentication +``` +sonic(config)#ntp + authenticate Authenticate time sources +sonic(config)#ntp authenticate +``` + +##### 3.6.2.1.8 Disable NTP authentication +``` +sonic(config)#no ntp authenticate +``` + +##### 3.6.2.1.9 Configure NTP authentication-key +``` +sonic(config)#ntp authentication-key + <1-65535> Key number + +sonic(config)#ntp authentication-key 1 + md5 MD5 authentication + sha1 SHA1 authentication + sha2-256 SHA2-256 authentication + +sonic(config)#ntp authentication-key 1 md5 + String Authentication key (max 64 chars, keys longer than 20 chars must be hex) + +sonic(config)#ntp authentication-key 1 md5 "ntp client 1" + +``` + +##### 3.6.2.1.10 Delete NTP authentication-key +``` +sonic(config)#no ntp authentication-key 1 +``` + +##### 3.6.2.1.11 Configure NTP trusted-key +``` +sonic(config)#ntp trusted-key + <1-65535> Key number + +sonic(config)#ntp trusted-key 1 +``` + +##### 3.6.2.1.12 Delete NTP trusted-key +``` +sonic(config)no ntp trusted-key 1 +``` + +##### 3.6.2.1.13 Add NTP server with key +``` +sonic(config)#ntp server 99.1.1.1 + key Configure peer authentication key + +sonic(config)#ntp server 99.1.1.1 key 1 +``` + +##### 3.6.2.1.14 Delete NTP server with key +``` +sonic(config)#no ntp server 99.1.1.1 +``` + +#### 3.6.2.2 Show ntp +``` +sonic# show ntp + associations Display NTP associations + global Display NTP global configuration + server Display NTP server configuration + +``` + +##### 3.6.2.2.1 show ntp associations + +``` +sonic# show ntp associations + remote refid st t when poll reach delay offset jitter +============================================================================== +*10.11.0.1 10.11.8.1 4 u 28 64 1 0.183 1.499 2.625 ++2001:aa:aa::b 60.39.129.68 10 u 27 64 1 0.638 2171.31 0.411 ++10.11.0.2 10.11.8.1 4 u 24 64 1 0.240 -13.957 12.786 +* master (synced), # master (unsynced), + selected, - candidate, ~ configured + +``` + +##### 3.6.2.2.2 Show configured ntp servers +``` +sonic# show ntp server +-------------------------------- +NTP Servers +-------------------------------- +10.11.0.1 +10.11.0.2 +``` + +##### 3.6.2.2.3 Show global ntp configurations +``` +sonic# show ntp global +---------------------------------------------- +NTP Global Configuration +---------------------------------------------- +NTP source-interface: eth0 + Loopback100 + +NTP vrf: mgmt + +``` + +##### 3.6.2.2.4 Show running-configuration + +``` +sonic(config)#ntp authenticate +sonic(config)#ntp authentication-key 1 md5 "ntp client 1" +sonic(config)#ntp authentication-key 1 md5 ntp_client_2 +sonic(config)#ntp server 99.1.1.1 key 1 +sonic(config)#ntp trusted-keys 1 +sonic(config)#ntp trusted-keys 2 +sonic(config)# do show running-configuration +! +ntp authenticate +ntp authentication-key 1 md5 3b88c0eb8406a9e76722b84baf1d94e5e185eb7f64f8dd46c759719c33557876 encrypted +ntp authentication-key 2 md5 771de7710005c5d6aa5b3313812b721d5d0d4a93fb1548572994464495476c4e encrypted +ntp server 99.1.1.1 key 1 +ntp trusted-keys 1 +ntp trusted-keys 2 +! + +sonic(config)#ntp server 10.11.0.1 +sonic(config)#ntp server pool.ntp.org +sonic(config)#ntp source-interface Management 0 +sonic(config)#ntp source-interface Loopback 100 +sonic(config)#do show running-configuration +! +ntp server 10.11.0.1 +ntp server pool.ntp.org +ntp source-interface Management 0 +ntp source-interface Loopback 100 +! + +sonic(config)# no ntp source-interface Loopback 100 +sonic(config)# ntp vrf mgmt +sonic(config)# do show running-configuration +! +ntp server 10.11.0.1 +ntp server pool.ntp.org +ntp source-interface Management 0 +ntp vrf mgmt +! + +sonic(config)# ntp vrf default +sonic(config)# do show running-configuration +! +ntp server 10.11.0.1 +ntp server pool.ntp.org +ntp source-interface Management 0 +ntp vrf default +! + +``` + +#### 3.6.2.3 Debug Commands +``` +From KLISH: + +show ntp associations + +show ntp server + +show ntp global + +``` + +``` +From shell: + +servcie ntp status + +check /etc/ntp.conf + +check /var/log/syslog and look for ntp + +check "docker exec -it mgmt tail -f /var/log/rest_server/rest_server.log" for rest logs + +ifconfig lo + +ifconfig lo-m + +show mgmt-vrf + +ntpq -pn + +``` + +#### 3.6.2.4 IS-CLI Compliance + +### 3.6.3 REST API Support +``` +GET - Get existing NTP configuration information from CONFIG DB. + Get NTP peer states +PUT - Create NTP configuration into CONFIG DB. +POST - Add NTP configuration into CONFIG DB. +PATCH - Update existing NTP configuraiton information in CONFIG DB. +DELETE - Delete a existing NTP configuration from CONFIG DB. +``` + +# 4 Flow Diagrams + +# 5 Error Handling + +# 6 Serviceability and Debug + +# 7 Warm Boot Support +NA + +# 8 Scalability + +# 9 Unit Test + +The unit-test for this feature will include: +#### Configuration via CLI + +| Test Name | Test Description | +| :-------- | :----- | +| Configure NTP server | Verify NTP servers are installed correctly in the configDB and reflected in the NTP peer status | +| Delete NTP server | Verify NTP server is deleted from the configDB and reflected in the NTP peer status | +| Configure NTP source interface| Verify NTP source interface is installed correctly in the configDB, NTP packets are transmitted and received over this source | +| | Verify that multiple NTP source interfaces can be configured, for both default and mgmt vrf cases| +| Delete NTP source interface| Verify that NTP source interface is removed from the configDB, NTP packets are transmitted and received over the default interface| +| Configure NTP vrf| Verify that NTP vrf is installed correctly in the configDB and ntp service is running in the specified VRF| +| | Verify that only default and mgmt can be configured as NTP vrf| +| Delete NTP vrf| Verify that NTP vrf is removed from the configDB and ntp service is running in the default instance| +| Configure NTP authentication for NTP server| Verify that NTP authentication-key can be created correctly| +| | Verify that NTP trusted-keys can be added correctly| +| | Verify that NTP authentication can be enabled and disabled| +| Configure NTP authentication for NTP client| Verify that NTP authentication-key can be created correctly| +| | Verify that NTP trusted-keys can be added correctly| +| | Verify that key number can be added to a NTP server | +| | Verify that NTP authentication can be enabled and disabled| +| | Verify NTP server is accepted if authentication keys match on NTP server and NTP client| +| | Verify NTP server is rejected if authentication keys mismatch on NTP server and NTP client| +| show ntp associations | Verify ntp associations are displayed correctly | +| show ntp server | Verify ntp servers are displayed correctly | +| show ntp global | Verify ntp global configurations are displayed correctly | + +#### Configuration via gNMI + +Same test as CLI configuration Test but using gNMI request. +Additional tests will be done to set NTP configuration at different levels of YANG models. + +#### Get configuration via gNMI + +Same as CLI show test but with gNMI request, will verify the JSON response is correct. +Additional tests will be done to get NTP configuration and NTP states at different levels of YANG models. + +#### Configuration via REST (POST/PUT/PATCH) + +Same test as CLI configuration Test but using REST POST/PUT/PATCH request. +Additional tests will be done to set NTP configuration at different levels of YANG models. + + +#### Get configuration via REST (GET) + +Same as CLI show test but with REST GET request, will verify the JSON response is correct. +Additional tests will be done to get NTP configuration and NTP states at different levels of YANG models. + + +# 10 Internal Design Information + + diff --git a/doc/SONiC_QoS_Mgmt_HLD-1.0.md b/doc/SONiC_QoS_Mgmt_HLD-1.0.md new file mode 100644 index 0000000000..24d5752639 --- /dev/null +++ b/doc/SONiC_QoS_Mgmt_HLD-1.0.md @@ -0,0 +1,499 @@ +# QoS + +# High Level Design Document +#### Rev 1.0 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 1.0 | 10/15/2019 | Oliver Hu, Ashok Daparthi, Srinadh Penugondaa, Eddy Lem | Initial version | + + +# About this Manual +This document provides general information about QoS configuration in SONiC using the management framework. + +# Scope +Covers Northbound interface for the QoS feature, as well as Unit Test cases. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|:-------------------------|:------------------------------------| +| QoS | Quality of Service | + +# 1 Feature Overview + +Provide management framework capabilities to handle: +- Traffic classification +- Queue management +- Traffic scheduling +- Buffer Threshold configuration +- Watermark statistics +- Associated show commands. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide management framework support to existing SONiC capabilities with respect to QoS. + +### 1.1.2 Configuration and Management Requirements +- CLI configuration and show commands +- REST API support +- gNMI Support + +Details described in Section 3. + +## 1.2 Design Overview + +### 1.2.1 Basic Approach +QoS deals with the traffic prioritization and scheduling. + +The basic approach is to classify different traffic to different service category, ie. putting traffic to different queues during ingress packet processing. + +In addition defining different queue behavior controls how the traffic is serviced during egress packet processing. + +### 1.2.2 Container +Management container + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases + +Anywhere user wants to give different traffic with different treatment. + +## 2.2 Functional Description + +Functionally QoS provides user the capability to handle different traffic with different level of service, from separate queuing to different scheduling behavior. + +# 3 Design +## 3.1 Overview + +To support quality of service, first user has to have a way to differentiate different types of traffic. Provided here is the method based on the DSCP value of the packet. Packets are classified into different internal traffic class based on their DSCP value and will be sent to different queues for scheduling. + +To manage the queue overflow situtation, WRED configuration is provided to allow queue to drop packet early before the queue is completely full. Or else packet will always be dropped when the queue is full. + +Once packets enter the queue, they will be eligible to be scheduled to be sent out to the wire. Scheduling the queue for transmit is done via Strict Priority (SP) method or Weighted Round Robin (WRR). Minimum guaranteed bandwidth, maximum allowed bandwidth and burst size are configurable for each queue to ensure guaranteed service and allow oversubscription for extra bandwidth. + + + +## 3.2 User Interface +### 3.2.1 Data Models + +Open config defines a data model for [QoS](https://github.com/openconfig/public/tree/master/release/models/qos) + +This data model lacks certain definitions for SONIC QoS feature. Some features that the data model supports also has limitations for future feature expansion. + +As a result, our data model will be loosely based on Open Config YANG, with our enhancement and modifications on top of it. +Here is a list of proposed new data model or existing data models from Open Config. + +- WRED profile + No Open Config YANG available. + Augment Open Config YANG model with new definitions similar to [SONIC YANG for WRED profile](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-wred-profile.yang) + + +- Scheduler Policy + openconfig-qos-elements.yang::qos-scheduler-policy-config + + +- DSCP to TC map + No Open Config YANG available. + Augment Open Config YANG model with new definitions similar to [SONIC YANG for DSCP-to-TC-map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-dscp-tc-map.yang) + + +- TC to Queue map + No Open Config YANG available. + Augment Open Config YANG model with new definition similar to [SONIC YANG for TC-to-Queue-map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-tc-queue-map.yang) + + +- Interface QoS + openconfig-qos-interfaces.yang:: qos-interfaces-config + openconfig-qos-elements.yang:: qos-queue-config + + + +- QoS Statistics + openconfig-qos-interfaces.yang:: qos-interfaces/interface/interface-id/output/queues/queue/state + Augment Open Config YANG model with new definitions for queue statistics and watermark statistics. + + +- Threshold + Augment Open Config YANG model with new definitions of Queue Thresholds. + + + +### 3.2.2 CLI + +#### 3.2.2.1 Configuration Commands + +- Configuring WRED +```` +sonic(config)# wred +sonic(conf-wred-)# random-detect color [green|yellow|red] minimum-threshold 100 maximum-threshold 300 +drop-probability 40 +sonic(conf-wred-)# random-detect ecn + + sonic(config)# no wred +```` + +- Configure Scheduler Policy +```` + sonic(config)# scheduler-policy + sonic(conf-sched-policy)# scheduler + sonic(conf-sched-policy-q)# priority level + sonic(conf-sched-policy-q)# shape-bandwidth min min-burst max max-burst + sonic(conf-sched-policy-q)# bandwidth-percent + + sonic(conf-sched-policy)# no scheduler + sonic(config)# no scheduler-policy +```` + + +- Config DSCP to Traffic Class map +```` + sonic(config)# qos-map dscp-to-tc-map + sonic(conf-qos-map)# traffic-class <1..8> dscp {} + + sonic(config)# no dscp-to-tc-map +```` + +- Config Traffic Class to Queue map +```` + sonic(config)# qos-map tc-to-queue-map + sonic(conf-qos-map)# queue 3 traffic-class 0-3 +```` + +- Config interface QoS +```` + sonic(config)# interface + sonic(conf-if-name)# dscp-to-tc-map + sonic(conf-if-name)# tc-to-queue-map + sonic(conf-if-name)# scheduler-policy + sonic(conf-if-name)# queue wred-profile + sonic(conf-if-name)# queue [ucast|mcast] threshold + + sonic(conf-if-name)# no dscp-to-tc-map + sonic(conf-if-name)# no tc-to-queue-map + sonic(conf-if-name)# no scheduler-policy + sonic(conf-if-name)# no queue wred-profile + sonic(conf-if-name)# no queue [ucast|mcast] threshold + +```` +- Config Watermark polling interval +```` + sonic(config)# qos watermark polling-interval +```` + + +#### 3.2.2.2 Show Commands + +- Show WRED +```` + >show wred {} + Sample Output: + > show wred wred_1 + Profile Name | Green | Yellow | Red | | + |------------------|------------------|------------------| | + | MIN MAX DROP-RATE| MIN MAX DROP-RATE| MIN MAX DROP-RATE| ECN | + | KB KB % | KB KB % | KB KB % | | + -------------|------------------|------------------|------------------|-------| + wred_1 | 100 1000 100 | 50 100 100 | 50 100 100 | All | + -------------|------------------|------------------|------------------|-------| +```` + + +- Show Scheduler Policy +```` + >show scheduler-policy + Sample output: + Scheduler : + Type: Priority + Bandwidth Percent: N/A + Priority Level: 1 + Shape Bandwidth: min 100mbps, min-burst 16kBytes, max 200mbps, max-burst 16kBytes + Scheduler : + Type: WRR + Bandwidth Percent: 40 + Shape Bandwidth: min 200mbps, min-burst 16kBytes, max 1000mbps, max-burst 16kBytes + ... +```` + + +- Show interface QoS configuration +```` + >show qos interface + Sample Output: + Ethernet 1/1/10 + DSCP-to-Traffic-Class map: dscp-to-tc-map_1 + Traffic-Class-to-Queue map: tc-to-queue-map_1 + Scheduler policy: scheduler-policy_1 + Q1 WRED profile: wred-profile_1 + Q5 WRED profile: wred-profile_2 + Q1 threshold: 2500 + Q5 threshold: 2500 +```` + +- Show DSCP to TC map +```` + >show qos-map dscp-to-tc-map + Sample Output: + DSCP Priority to Traffic-Class Map : dscp-trustmap1 + Traffic-Class | DSCP Priority + --------------|--------------- + 0 | 8-15 + 2 | 16-23 + 1 | 0-7 +```` + +- Show TC to Queue map +```` + >show qos-map tc-to-queue-map + Sample Output: + Traffic-Class to Queue Map: queue-map1 + Queue | Traffic-Class + ------|------------------- + 1 | 5 + 2 | 6 + 3 | 7 +```` + +- Show QoS statistics + +```` + > show queue counters {interface ethernet {queue }} + Sample Output: + + +sonic$ show queue counters + Port TxQ Counter/pkts Counter/bytes Drop/pkts Drop/bytes +--------- ----- -------------- --------------- ----------- ------------ +Ethernet0 UC0 0 0 0 0 +Ethernet0 UC1 0 0 0 0 +Ethernet0 UC2 0 0 0 0 +Ethernet0 UC3 0 0 0 0 +Ethernet0 UC4 0 0 0 0 +Ethernet0 UC5 0 0 0 0 +Ethernet0 UC6 0 0 0 0 +Ethernet0 UC7 0 0 0 0 +Ethernet0 UC8 0 0 0 0 +Ethernet0 UC9 0 0 0 0 +Ethernet0 MC0 0 0 0 0 +Ethernet0 MC1 0 0 0 0 +Ethernet0 MC2 0 0 0 0 +Ethernet0 MC3 0 0 0 0 +Ethernet0 MC4 0 0 0 0 +Ethernet0 MC5 0 0 0 0 +Ethernet0 MC6 0 0 0 0 +Ethernet0 MC7 0 0 0 0 +Ethernet0 MC8 0 0 0 0 +Ethernet0 MC9 0 0 0 0 +Ethernet4 UC0 0 0 0 0 +Ethernet4 UC1 0 0 0 0 +Ethernet4 UC2 0 0 0 0 +Ethernet4 UC3 0 0 0 0 +... + + + Sample output: + +sonic# show queue counters Ethernet 0 queue 3 +------------------------------------------------------------------- +TxQ Counter/pkts Counter/bytes Drop/pkts Drop/bytes +------------------------------------------------------------------- +UC3 0 0 0 0 + + + > show queue wred statistics interface ethernet queue (*) + Sample Output: + > show queue wred statistics interface ethernet 1/1/1 + Interface ethernet1/1/1 (All queues) + Description Packets Bytes + Output 0 0 + Dropped 0 0 + Green Drop 0 0 + Yellow Drop 0 0 + Red Drop 0 0 + ECN marked count 0 0 + (*) WRED statistics is not supported in SONiC. It is not supported in Buzznik release. + + > show queue (watermark|persistent-watermark) (multicast | unicast) [] + +Sample output +sonic$ show queue watermark unicast +Egress shared pool occupancy per unicast queue: + Port UC0 UC1 UC2 UC3 UC4 UC5 UC6 UC7 +----------- ----- ----- ----- ----- ----- ----- ----- ----- + Ethernet0 0 0 0 0 0 0 0 0 + Ethernet4 0 0 0 0 0 0 0 0 + Ethernet8 0 0 0 0 0 0 0 0 + Ethernet12 0 0 0 0 0 0 0 0 + + + > show queue buffer-threshold-breaches + Sample Output: + >show queue buffer-threshold-breaches + Interface | Queue | breach-value | watermark-bytes | time-stamp + Ethernet0 | UC3 | 82 | 8100 | 2019-06-14 - 11:29:33 + Ethernet1 | UC1 | 80 | 8100 | 2019-06-14 - 11:20:19 + ... + + > show priority-group (watermark | persistent-watermark) (headroom | shared) [] + + Sample Output: + +sonic$ show priority-group watermark shared +Ingress shared pool occupancy per PG: + Port PG0 PG1 PG2 PG3 PG4 PG5 PG6 PG7 +----------- ----- ----- ----- ----- ----- ----- ----- ----- + Ethernet0 0 0 0 0 0 0 0 0 + Ethernet4 0 0 0 0 0 0 0 0 + Ethernet8 0 0 0 0 0 0 0 0 + Ethernet12 0 0 0 0 0 0 0 0 + + +```` +#### 3.2.2.3 Clear QoS statistis +```` + sonic# clear queue statistics [interface ethernet ] + sonic# clear queue wred statistics [interface ethernet ] + sonic# clear queue [buffer-watermark | buffer-persistent-watermark] [interface ethernet ] + +```` +#### 3.2.2.4 Debug Commands +N/A + +#### 3.2.2.5 IS-CLI Compliance +N/A + +### 3.2.3 REST API Support + +The "PATCH" operation is available to the YANG model attributes except those */state/*. + +The "GET" operation is available to the YANG model attributes with */state/*. + +The "DELETE" operation is available to the key(s) of a list in the YANG model. + +- WRED profile + + [SONiC YANG for WRED profile](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-wred-profile.yang) will be converted to Open Config YANG format following Open Config convention. + + +- Scheduler Policy + Scheduler Policy configuration uses the following Open Config YANG: +```` + openconfig-qos-elements.yang::qos-scheduler-policy-config +```` + + A summary of the supported attributes or new attributes added to the tree: +```` + /scheduler-policy/name + /scheduler-policy/name/config/scheduler/sequence + /scheduler-policy/name/config/scheduler/sequence/config/priority + /scheduler-policy/name/config/scheduler/sequence/config/weight (New!) + /scheduler-policy/name/config/scheduler/sequence/cir + /scheduler-policy/name/config/scheduler/sequence/pir + /scheduler-policy/name/config/scheduler/sequence/bc + /scheduler-policy/name/config/scheduler/sequence/be + /scheduler-policy/name/state/* +```` + + +- DSCP to TC map + + [SONiC YANG for DSCP-to-TC map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-dscp-tc-map.yang) will be converted to Open Config YANG format following Open Config convention. + + +- Interface QoS + + The following two YANG containers are used to model interface QoS configuration: +```` + openconfig-qos-interfaces.yang:: qos-interfaces-config + openconfig-qos-elements.yang:: qos-queue-config +```` + + A summary of the supported attributes or new attributes added to the tree: +```` + /qos-interfaces/interface/interface-id + /qos-interfaces/interface/interface-id/input/dscp-to-tc-map (New!) + /qos-interfaces/interface/interface-id/output/scheduler-policy/name + + /qos-queue/queues/queue/name = "intf-name + q" + /qos-queue/queues/queue/name/config/queue-type + /qos-queue/queues/queue/name/config/wred-profile (New!) + + /qos-interfaces/interface/interface-id/input/state/* + /qos-interfaces/interface/interface-id/output/state/* + /qos-queue/queues/queues/name/state/* +```` + + +- QoS Statistics + The following Open Config YANG container will be used and augmented to support QoS Statistics +```` + openconfig-qos-interfaces.yang:: qos-interfaces/interface/interface-id/output/queues/queue/state +```` + +# 4 Flow Diagrams +N/A + +# 5 Error Handling +N/A + +# 6 Serviceability and Debug +N/A + +# 7 Warm Boot Support +N/A + +# 8 Scalability +N/A + +# 9 Unit Test +- Create WRED profile, verify it with Show command +- Delete WRED profile that is not used by any queue +- Delete WRED profile while it is being used by some queue. Should reject the operation. +- Create Scheduler Policy, verify it with Show command +- Delete Scheduler Policy that is not used by any queue +- Delete Scheduler Policy while it is being used by some queue. Should reject the operation +- Create DSCP-to-TC-map, verify it with Show command +- Delete DSCP-to-TC-map that is not used by any interface +- Delete DSCP-to-TC-map while it is being used by some interface. Should reject the operation +- Configure DSCP-to-TC-map on an interface; check it with Show command +- Create TC-to-Queue-map, verify it with Show command +- Delete TC-to-Queue-map that is not used by any interface +- Delete TC-to-Queue-map while it is being used by some interface. Should reject the operation +- Configure TC-to-Queue-map on an interface; check it with Show command +- Configure WRED profile on an interface queue; check it with Show command +- Configure Scheduler Policy on an interface; check it with Show command. +- Update DSCP-to-TC-map content; check the map with Show command +- Update WRED-profile content; check interface queue is updated with new WRED parameters +- Update Scheduler-policy content; check interface is updated with new scheduler policy settings. +- Read QoS Queue statistics +- Clear QoS Queue statistics + + +# 10 Internal Design Information + +For Scheduler Policy created via Open Config YANG model, each Scheduler within a Scheduler Policy is transformed into a Scheduler Profile defined by SONiC DB. + +The Scheduler Profile is identified as "Policy_name + q". Such naming convention makes it possible to associate each interface queue with a unique Scheduler Profile later on. + +When a scheduler Policy is configured on an interface in Open Config YANG model, backend creates SONiC "interface.q#" in SONiC QueueDB if such entity has not yet been created. Backup also looks up the scheduler Profiles within the Scheduler Policy and attaches the Scheduler Profiles to the corresponding interface queues. + +When WRED profile is configured on an interface queue in Open Config YANG model, backend writes the information into SONiC Queue DB. + +When queue state query comes from Open Config YANG, a valid queue name can be in the form of "Q#", "UC-Q#" or "MC-Q#". "Q#" will include both Unicast and Multicast queue statistics, while "UC-Q#" and "MC-Q#" will be specific to either Unicast or Multicast queue. + diff --git a/doc/SONiC_QoS_Mgmt_HLD-1.1.md b/doc/SONiC_QoS_Mgmt_HLD-1.1.md new file mode 100644 index 0000000000..6cb935b9b9 --- /dev/null +++ b/doc/SONiC_QoS_Mgmt_HLD-1.1.md @@ -0,0 +1,712 @@ +# QoS + +# High Level Design Document +#### Rev 1.0 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 1.0 | 10/15/2019 | Oliver Hu, Ashok Daparthi, Srinadh Penugondaa, Eddy Lem | Initial version | + + +# About this Manual +This document provides general information about QoS configuration in SONiC using the management framework. + +# Scope +Covers Northbound interface for the QoS feature, as well as Unit Test cases. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|:-------------------------|:------------------------------------| +| QoS | Quality of Service | + +# 1 Feature Overview + +Provide management framework capabilities to handle: +- Traffic classification +- Queue management +- Traffic scheduling +- Buffer Threshold configuration +- Watermark statistics +- Associated show and clear commands. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide management framework support to existing SONiC capabilities with respect to QoS. + +### 1.1.2 Configuration and Management Requirements +- IS-CLI style configuration and show commands +- REST API support +- gNMI Support + +Details described in Section 3. + +## 1.2 Design Overview + +### 1.2.1 Basic Approach +QoS deals with the traffic prioritization and scheduling. + +The basic approach is to classify different traffic to different service category, ie. putting traffic to different queues. + +In addition defining different queue behavior controls how the traffic is serviced. + +### 1.2.2 Container +Management container + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases + +Anywhere user wants to give different traffic with different treatment. + +## 2.2 Functional Description + +Functionally QoS provides user the capability to handle different traffic with different level of service, from separate queuing to different scheduling behavior. + +# 3 Design +## 3.1 Overview + +To support quality of service, first user has to have a way to differentiate different types of traffic. Provided here is the method based on the DSCP value of the packet. Packets are classified into different internal traffic class based on their DSCP value and will be sent to different queues for scheduling. + +To manage the queue overflow situtation, WRED configuration is provided to allow queue to drop packet early before the queue is completely full. Or else packet will always be dropped when the queue is full. + +Once packets enter the queue, they will be eligible to be scheduled to be sent out to the wire. Scheduling the queue for transmit is done via Strict Priority (SP) method or Weighted Round Robin (WRR). Minimum guaranteed bandwidth, maximum allowed bandwidth and burst size are configurable for each queue to ensure guaranteed service and allow oversubscription for extra bandwidth. + + + +## 3.2 User Interface +### 3.2.1 Data Models + +Open config defines a data model for [QoS](https://github.com/openconfig/public/tree/master/release/models/qos) + +This data model lacks certain definitions for SONIC QoS feature. Some features that the data model supports also has limitations for future feature expansion. + +As a result, our data model will be loosely based on Open Config YANG, with our enhancement and modifications on top of it. +Here is a list of proposed new data model or existing data models from Open Config. + +- WRED Policy + + No Open Config YANG available. + Augment Open Config YANG model with new definitions similar to [SONIC YANG for WRED profile](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-wred-profile.yang) + Configuring WRED on Green color only supported in this release. + + +- Scheduler Policy + + openconfig-qos-elements.yang::qos-scheduler-policy-config + + +- DSCP to TC map + No Open Config YANG available. + Augment Open Config YANG model with new definitions similar to [SONIC YANG for DSCP-to-TC-map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-dscp-tc-map.yang) + + +- Dot1p to TC map + + No Open Config YANG available. + Augment Open Config YANG model with new definitions similar to [SONIC YANG for DOT1P-to-TC-map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-dot1p-tc-map.yang) + + +- TC to Queue map + + No Open Config YANG available. + Augment Open Config YANG model with new definition similar to [SONIC YANG for TC-to-Queue-map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-tc-queue-map.yang) + + + +- TC to Priority group + + No Open Config YANG available. + Augment Open Config YANG model with new definition similar to [SONIC YANG for TC-to-priority-group-map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-tc-priority-group-map.yang) + + +- PFC priority to Queue + + No Open Config YANG available. + Augment Open Config YANG model with new definition similar to [SONIC YANG for PFC-pirority-to-queue-map](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/cvl/testdata/schema/sonic-pfc-priority-queue.yang) + + +- PFC Configuration + + No open config support for enable PFC priorities on interface. + Augment Open Config YANG model with new attribute for PFC priorities and PFC Asynchronous mode. + + +- Interface QoS + + openconfig-qos-interfaces.yang:: qos-interfaces-config + openconfig-qos-elements.yang:: qos-queue-config + + +- QoS Statistics + + openconfig-qos-interfaces.yang:: qos-interfaces/interface/interface-id/output/queues/queue/state + Augment Open Config YANG model with new definitions for queue statistics and watermark statistics. + + +- QoS Clear Statistics + + No Open Config YANG available. + RPC based sonic YANG model with new definitions for clear QOS statistics and watermark statistics. + + +- Threshold Breach + + Augment Open Config YANG model with new definitions of Thresholds breach get. + + + +### 3.2.2 CLI + +#### 3.2.2.1 Configuration Commands + +- Configuring WRED +```` + sonic(config)# qos wred-policy + sonic(conf-wred-)# [green] minimum-threshold 100 maximum-threshold 300 + drop-probability 40 + sonic(conf-wred-)# ecn + + sonic(config)# no qos wred-policy +```` +- Configure Scheduler Policy for queues +```` + sonic(config)# qos scheduler-policy + sonic(conf-sched-policy)# queue + sonic(conf-sched-policy-queue-)# type + sonic(conf-sched-policy-queue-)# cir + sonic(conf-sched-policy-queue-)# cbs + sonic(conf-sched-policy-queue-)# pir + sonic(conf-sched-policy-queue-)# pbs + sonic(conf-sched-policy-queue0)# weight + + sonic(conf-sched-policy)# no queue + sonic(config)# no qos scheduler-policy +```` +- Configure Scheduler Policy for port shaper + +```` + sonic(config)# qos scheduler-policy + sonic(conf-sched-policy)# port + sonic(conf-sched-policy-port)# pir + sonic(conf-sched-policy-port)# pbs + + sonic(conf-sched-policy)# no port + +```` + +- Config DSCP to Traffic Class map +```` + sonic(config)# qos map dscp-tc + sonic(conf-qos-map)# dscp {} traffic-class <0..7> + sonic(conf-qos-map)# no dscp {} + + sonic(config)# no qos map dscp-tc +```` +- Config DOT1P to Traffic Class map +```` + sonic(config)# qos map dot1p-tc + sonic(conf-qos-map)# dot1p {} traffic-class <0..7> + sonic(conf-qos-map)# no dot1p {} + + sonic(config)# no qos map dot1p-tc +```` + +- Config Traffic Class to Queue map +```` + sonic(config)# qos map tc-queue + sonic(conf-qos-map)# traffic-class {} queue <0..7> + + sonic(config)# no qos map tc-queue +```` + +- Config Traffic Class to Priority Group map +```` + sonic(config)# qos map tc-pg + sonic(conf-qos-map)# traffic-class {} pg <0..7> + + sonic(config)# no qos map tc-pg +```` + +- Config PFC priority to queue map +```` + sonic(config)# qos map pfc-priority-queue + sonic(conf-qos-map)# pfc-priority {} queue <0..7> + + sonic(config)# no qos map pfc-priority-queue +```` +- Config interface QoS +```` + sonic(config)# interface + sonic(conf-if-name)# qos-map dscp-tc + sonic(conf-if-name)# qos-map dot1p-tc + sonic(conf-if-name)# qos-map tc-queue + sonic(conf-if-name)# qos-map tc-pg + sonic(conf-if-name)# qos-map pfc-priority-queue + sonic(conf-if-name)# pfc priority <0-7> + sonic(conf-if-name)# pfc asymmetric + sonic(conf-if-name)# scheduler-policy + sonic(conf-if-name)# queue wred-policy + + sonic(conf-if-name)# no qos-map dscp-tc + sonic(conf-if-name)# no qos-map dot1p-tc + sonic(conf-if-name)# no qos-map tc-pg + sonic(conf-if-name)# no qos-map tc-queue + sonic(conf-if-name)# no qos-map pfc-priority-queue + sonic(conf-if-name)# no scheduler-policy + sonic(conf-if-name)# no queue wred-policy +```` + +#### 3.2.2.2 Show Commands + +- Show WRED + +```` + >show qos wred-policy {} + Sample Output: +--------------------------------------------------- +Profile : test +--------------------------------------------------- +ecn : ecn_green +green-min-threshold : 1 KBytes +green-max-threshold : 1 KBytes +green-drop-probability : 10 + +```` +- Show Scheduler Policy + +```` +>show qos scheduler-policy {} +Sample output: +sonic# show qos scheduler-policy + Scheduler Policy: test + Queue: 0 + type: strict + weight: 10 + cir: 10000 Kbps + cbs: 100 Bytes + pir: 200000 Kbps + pbs: 200 Bytes + Port: + pir: 100 Kbps + pbs: 200 Bytes + +```` + +- Show interface QoS configuration +```` + >show qos interface + Sample Output: + sonic# show qos interface Ethernet 0 + scheduler policy: p + dscp-tc-map: test + dot1p-tc-map: test + tc-queue-map: test + tc-pg-map: test + pfc-priority-queue-map: test + pfc-asymmetric: off + pfc-priority : 3,4 + + sonic# show qos interface Ethernet 0 queue 0 + interface queue 0 config: + WRED profile: test +```` + +- Show DSCP to TC map + +```` +>show qos map dscp-tc +Sample Output: +sonic# show qos map dscp-tc +DSCP-TC-MAP: test +---------------------------- + DSCP TC +---------------------------- + 0 0 + 1 1 +---------------------------- + +```` +- Show DOT1P to TC map + +```` +>show qos map dot1p-tc + Sample Output: + sonic# show qos map dot1p-tc + DOT1P-TC-MAP: test + ---------------------------- + DOT1P TC + ---------------------------- + 0 0 + 1 1 +---------------------------- + +```` +- Show TC to Queue map + +```` +>show qos map tc-queue + Sample Output: + sonic# show qos map tc-queue + TC-Q-MAP: test + ---------------------------- + TC Q + ---------------------------- + 0 0 + 1 1 +---------------------------- + +```` + +- Show TC to PG map + +```` +>show qos map tc-pg + Sample Output: + sonic# show qos map tc-pg + TC-PG-MAP: test + ---------------------------- + TC PG + ---------------------------- + 0 0 + 1 1 +---------------------------- + +```` + +- Show PFC priority to queue map + +```` +>show qos map pfc-priority-queue + Sample Output: + sonic# show qos map pfc-priority-queue + PFC-PRIORITY-Q-MAP: test + ---------------------------- + PRIORITY Q + ---------------------------- + 0 0 + 1 1 +---------------------------- + +```` +- Show QoS statistics + + +show queue counters {interface (ethernet | CPU) {queue }} + +show queue (watermark|persistent-watermark) (percentage) (multicast | unicast | CPU) interface [] + +show priority-group (watermark|persistent-watermark) (percentage) (headroom | shared) interface [] + +show queue buffer-threshold-breaches + +```` + > show queue counters interface ethernet queue + Sample Output: + > show queue counters interface ethernet 1/1/1 queue 3 + sonic# show queue counters interface Ethernet 0 queue 0 + ------------------------------------------------------------------- + TxQ Counter/pkts Counter/bytes Drop/pkts Drop/bytes + ------------------------------------------------------------------- + UC0 0 0 0 0 + + sonic# show queue watermark unicast interface Ethernet 0 + Egress queue watermark per unicast queue: + ----------------------------------------------- + UC0 UC1 UC2 UC3 UC4 UC5 UC6 UC7 + ----------------------------------------------- + 0 0 0 0 0 0 0 0 + + sonic# show queue watermark multicast interface Ethernet 0 + Egress queue watermark per multicast queue: + ----------------------------------------------- + MC8 MC9 MC10 MC11 MC12 MC13 MC14 MC15 + ----------------------------------------------- + 0 0 0 0 0 0 0 0 + + sonic# show priority-group watermark headroom/shared interface Ethernet 0 + Ingress headroom watermark per PG: + ------------------------------------------------------------------- + PG0 PG1 PG2 PG3 PG4 PG5 PG6 PG7 + ------------------------------------------------------------------- + 0 0 0 0 0 0 0 0 + + > show queue buffer-threshold-breaches + Sample Output: + >show queue buffer-threshold-breaches + Interface | Queue | breach-value | watermark-bytes | time-stamp + Ethernet0 | UC3 | 82 | 8100 | 2019-06-14 - 11:29:33 + Ethernet1 | UC1 | 80 | 8100 | 2019-06-14 - 11:20:19 + ... + +```` +#### 3.2.2.3 Clear QoS statistis +```` + sonic# clear queue counters [interface ethernet | CPU ] + sonic# clear queue [watermark | persistent-watermark] (unicast|multicast|CPU) [interface ethernet ] + sonic# clear priority-group [watermark | persistent-watermark] (headroom|shared) [interface ethernet ] +```` +#### 3.2.2.4 Debug Commands +N/A + +#### 3.2.2.5 IS-CLI Compliance +N/A + +### 3.2.3 REST API Support + +The "PATCH" operation is available to the YANG model attributes except those */state/*. + +The "GET" operation is available to the YANG model attributes with */state/*. + +The "DELETE" operation is available to the key(s) of a list in the YANG model. + +- WRED profile + + ```` + +--rw oc-qos-ext:wred-profiles + | +--rw oc-qos-ext:wred-profile* [name] + | +--rw oc-qos-ext:name -> ../config/name + | +--rw oc-qos-ext:config + | | +--rw oc-qos-ext:name? string + | | +--rw oc-qos-ext:green_min_threshold? uint64 + | | +--rw oc-qos-ext:green_max_threshold? uint64 + | | +--rw oc-qos-ext:ecn? enumeration + | | +--rw oc-qos-ext:wred_green_enable? boolean + | | +--rw oc-qos-ext:green_drop_rate? uint64 + | +--ro oc-qos-ext:state + | +--ro oc-qos-ext:name? string + | +--ro oc-qos-ext:green_min_threshold? uint64 + | +--ro oc-qos-ext:green_max_threshold? uint64 + | +--ro oc-qos-ext:ecn? enumeration + | +--ro oc-qos-ext:wred_green_enable? boolean + | +--ro oc-qos-ext:green_drop_rate? uint64 +```` +- Scheduler Policy + Scheduler Policy configuration uses the following Open Config YANG: + +```` + openconfig-qos-elements.yang::qos-scheduler-policy-config + +```` + A summary of the supported attributes or new attributes added to the tree: + +```` + /scheduler-policy/name + /scheduler-policy/name/config/scheduler/sequence + /scheduler-policy/name/config/scheduler/sequence/config/priority + /scheduler-policy/name/config/scheduler/sequence/config/weight (New!) + /scheduler-policy/name/config/scheduler/sequence/cir + /scheduler-policy/name/config/scheduler/sequence/pir + /scheduler-policy/name/config/scheduler/sequence/bc + /scheduler-policy/name/config/scheduler/sequence/be + /scheduler-policy/name/state/* + +```` + +- DSCP to TC map + +```` ++--rw oc-qos-maps-ext:dscp-maps + | +--rw oc-qos-maps-ext:dscp-map* [name] + | +--rw oc-qos-maps-ext:name -> ../config/name + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:name? string + | +--ro oc-qos-maps-ext:state + | | +--ro oc-qos-maps-ext:name? string + | +--rw oc-qos-maps-ext:dscp-map-entries + | +--rw oc-qos-maps-ext:dscp-map-entry* [dscp] + | +--rw oc-qos-maps-ext:dscp -> ../config/dscp + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:dscp? uint8 + | | +--rw oc-qos-maps-ext:fwd_group -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/name + | +--ro oc-qos-maps-ext:state + | +--ro oc-qos-maps-ext:dscp? uint8 + | +--ro oc-qos-maps-ext:fwd_group -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/nam +```` + +- DOT1P to TC map + +```` ++--rw oc-qos-maps-ext:dot1p-maps + | +--rw oc-qos-maps-ext:dot1p-map* [name] + | +--rw oc-qos-maps-ext:name -> ../config/name + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:name? string + | +--ro oc-qos-maps-ext:state + | | +--ro oc-qos-maps-ext:name? string + | +--rw oc-qos-maps-ext:dot1p-map-entries + | +--rw oc-qos-maps-ext:dot1p-map-entry* [dot1p] + | +--rw oc-qos-maps-ext:dot1p -> ../config/dot1p + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:dot1p? uint8 + | | +--rw oc-qos-maps-ext:fwd_group -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/name + | +--ro oc-qos-maps-ext:state + | +--ro oc-qos-maps-ext:dot1p? uint8 + | +--ro oc-qos-maps-ext:fwd_group -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/name + +```` + +- Traffic class to queue map + +```` ++--rw oc-qos-maps-ext:forwarding-group-queue-maps + | +--rw oc-qos-maps-ext:forwarding-group-queue-map* [name] + | +--rw oc-qos-maps-ext:name -> ../config/name + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:name? string + | +--ro oc-qos-maps-ext:state + | | +--ro oc-qos-maps-ext:name? string + | +--rw oc-qos-maps-ext:forwarding-group-queue-map-entries + | +--rw oc-qos-maps-ext:forwarding-group-queue-map-entry* [fwd_group] + | +--rw oc-qos-maps-ext:fwd_group -> ../config/fwd_group + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:fwd_group? -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/name + | | +--rw oc-qos-maps-ext:output-queue-index uint8 + | +--ro oc-qos-maps-ext:state + | +--ro oc-qos-maps-ext:fwd_group? -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/name + | +--ro oc-qos-maps-ext:output-queue-index uint8 + +```` + +- Traffic class to priority group map + +```` ++--rw oc-qos-maps-ext:forwarding-group-priority-group-maps + | +--rw oc-qos-maps-ext:forwarding-group-priority-group-map* [name] + | +--rw oc-qos-maps-ext:name -> ../config/name + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:name? string + | +--ro oc-qos-maps-ext:state + | | +--ro oc-qos-maps-ext:name? string + | +--rw oc-qos-maps-ext:forwarding-group-priority-group-map-entries + | +--rw oc-qos-maps-ext:forwarding-group-priority-group-map-entry* [fwd_group] + | +--rw oc-qos-maps-ext:fwd_group -> ../config/fwd_group + | +--rw oc-qos-maps-ext:config + | | +--rw oc-qos-maps-ext:fwd_group? -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/name + | | +--rw oc-qos-maps-ext:priority-group-index uint8 + | +--ro oc-qos-maps-ext:state + | +--ro oc-qos-maps-ext:fwd_group? -> ../../../../../../oc-qos:forwarding-groups/forwarding-group/config/name + | +--ro oc-qos-maps-ext:priority-group-index uint8 +```` + +- PFC priority to queue map + +```` ++--rw oc-qos-maps-ext:pfc-priority-queue-maps + +--rw oc-qos-maps-ext:pfc-priority-queue-map* [name] + +--rw oc-qos-maps-ext:name -> ../config/name + +--rw oc-qos-maps-ext:config + | +--rw oc-qos-maps-ext:name? string + +--ro oc-qos-maps-ext:state + | +--ro oc-qos-maps-ext:name? string + +--rw oc-qos-maps-ext:pfc-priority-queue-map-entries + +--rw oc-qos-maps-ext:pfc-priority-queue-map-entry* [dot1p] + +--rw oc-qos-maps-ext:dot1p -> ../config/dot1p + +--rw oc-qos-maps-ext:config + | +--rw oc-qos-maps-ext:dot1p? uint8 + | +--rw oc-qos-maps-ext:output-queue-index uint8 + +--ro oc-qos-maps-ext:state + +--ro oc-qos-maps-ext:dot1p? uint8 + +--ro oc-qos-maps-ext:output-queue-index uint8 +```` + +- Interface QoS + + The following two YANG containers are used to model interface QoS configuration: + openconfig-qos-interfaces.yang:: qos-interfaces-config + openconfig-qos-elements.yang:: qos-queue-config + + A summary of the supported attributes or new attributes added to the tree: +```` + /qos-interfaces/interface/interface-id + /oc-qos:qos/oc-qos:interfaces/oc-qos:interface/interface-maps/config/dscp-to-forwarding-group + /oc-qos:qos/oc-qos:interfaces/oc-qos:interface/interface-maps/config/dot1p-to-forwarding-group + /oc-qos:qos/oc-qos:interfaces/oc-qos:interface/interface-maps/config/forwarding-group-to-queue + /oc-qos:qos/oc-qos:interfaces/oc-qos:interface/interface-maps/config/pfc-priority-to-queue + /oc-qos:qos/oc-qos:interfaces/oc-qos:interface/interface-maps/config/forwarding-group-to-priority-group + + /qos-interfaces/interface/interface-id/output/scheduler-policy/name + + /qos-queue/queues/queue/name = intf-name:q + /qos-queue/queues/queue/name/config/queue-type + /qos-queue/queues/queue/name/config/wred-profile (New!) + + /qos-interfaces/interface/interface-id/input/state/* + /qos-interfaces/interface/interface-id/output/state/* + /qos-queue/queues/queues/name/state/* +```` + +- QoS Statistics + + The following Open Config YANG container will be used and augmented to support QoS Statistics + + openconfig-qos-interfaces.yang:: qos-interfaces/interface/interface-id/output/queues/queue/state + + + +# 4 Flow Diagrams +N/A + +# 5 Error Handling +N/A + +# 6 Serviceability and Debug +N/A + +# 7 Warm Boot Support +N/A + +# 8 Scalability +N/A + +# 9 Unit Test +- Create WRED profile, verify it with Show command +- Delete WRED profile that is not used by any queue +- Delete WRED profile while it is being used by some queue. Should reject the operation. +- Create Scheduler Policy, verify it with Show command +- Delete Scheduler Policy that is not used by any queue +- Delete Scheduler Policy while it is being used by some queue. Should reject the operation +- Create DSCP-to-TC-map, verify it with Show command +- Delete DSCP-to-TC-map that is not used by any interface +- Delete DSCP-to-TC-map while it is being used by some interface. Should reject the operation +- Configure DSCP-to-TC-map on an interface; check it with Show command +- Create TC-to-Queue-map, verify it with Show command +- Delete TC-to-Queue-map that is not used by any interface +- Delete TC-to-Queue-map while it is being used by some interface. Should reject the operation +- Configure TC-to-Queue-map on an interface; check it with Show command +- Configure WRED profile on an interface queue; check it with Show command +- Configure Scheduler Policy on an interface; check it with Show command. +- Update DSCP-to-TC-map content; check the map with Show command +- Update WRED-profile content; check interface queue is updated with new WRED parameters +- Update Scheduler-policy content; check interface is updated with new scheduler policy settings. +- Read QoS Queue statistics +- Clear QoS Queue statistics + + +# 10 Internal Design Information + +For Scheduler Policy created via Open Config YANG model, each Scheduler within a Scheduler Policy with sequence number is transformed into a Scheduler Profile defined by SONiC DB. Each scheduler policy sequence is assumed as queue number. For interface scheduler sequence 0-7 is valid and CPU interface sequence 0-31 are valid. In ethernet interface scheduler with sequence number > 8 are ignored. Sequence 255 is reserved for port scheduler. + +The Scheduler Profile is identified as "Policy_name@q#"". Such naming convention makes it possible to associate each interface queue with a unique Scheduler Profile later on. + +When a scheduler Policy is configured on an interface in Open Config YANG model, backend associates each scheduler under scheduler policy to corresponding queue(sequence = queueid) in SONiC QueueDB if such entity has not yet been created. Backup also looks up the scheduler Profiles within the Scheduler Policy and attaches the Scheduler Profiles to the corresponding interface. + +When WRED profile is configured on an interface queue in Open Config YANG model, backend writes the information into SONiC Queue DB. + +When queue state query comes from Open Config YANG, a valid queue name can be in the form of "Q#", "UC-Q#" or "MC-Q#". "Q#" will include both Unicast and Multicast queue statistics, while "UC-Q#" and "MC-Q#" will be specific to either Unicast or Multicast queue. diff --git a/doc/VRRP_HLD_Buzznik.md b/doc/VRRP_HLD_Buzznik.md new file mode 100755 index 0000000000..6981680e84 --- /dev/null +++ b/doc/VRRP_HLD_Buzznik.md @@ -0,0 +1,730 @@ +# Virtual Router Redundency Protocol (VRRP) HLD + +#### Rev 0.1 + + + +[TOC] + + + +# List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:--:|:--------:|:-----------------:|:------------------------------------------------------------:| +| 0.1 | 09/05/2019 | Dilip Kumar | Initial version | +| 0.2 | 09/11/2019 | Dilip Kumar | Addressed review comments from Ben and others. Minor edits to complete the unifished sections. | +| 0.3 | 10/10/2019 | Vijay Kumar | Added VRRP version 3 and VARP support | +| | | | | +| | | | | + +# Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +| -------- | ------------------------------------------- | +| ARP | Address Resolution Protocol | +| IP | Internet Protocol | +| LAG | Link Aggregation Group | +| LAN | Local Area Network | +| MAC | Media Access Control addresses | +| VIP | Virtual IP address | +| VLAN | Virtual Local Area Network | +| VMAC | Virtual MAC address | +| VRF | Virtual Routing and Forwarding | +| VRID | Virtual Router Identifier | +| VRRP | Virtual Router Redundency Protocol | +| VRRPv3 | Virtual Router Redundency Protocol verson 3 | +| IPv6 | Internet Protocol version 6 | +| VARP | Virtual Address Resolution Protocol | + +# About this Manual + +This document provides general overview of VRRP feature implementation based on RFC 5798 and VARP feature implementation in SONiC. + + + +# 1 Introduction and Scope + +Virtual Router Redundancy Protocol (VRRP) functionality is designed to eliminate the single point of +failure inherent in the static default routed environment. VRRP specifies an election protocol that +dynamically assigns responsibility of gateway router to one of the VRRP routers on a LAN. The VRRP +router controlling the IP(v6) address(es) associated with a virtual router is called the Master, and routes the +traffic. The election process provides dynamic fail-over in the forwarding responsibility should the Master +become unavailable. Any of the virtual router's IP(v6) addresses on a LAN can then be used as the default first +hop router by end-hosts. The advantage gained from using VRRP is a higher availability default path +without requiring configuration of dynamic routing or router discovery protocols on every end-host. + +Virtual-ARP (VARP) allows multiple switches to simultaneously route packets from a common IP address in an active-active router configuration. Each switch is configured with the same set of virtual IP addresses on corresponding VLAN interfaces and a common virtual MAC address. + + +# 2 Feature Requirements + +## 2.1 Functional Requirements + +Following requirements are addressed by the design presented in this document: + +1. Support VRRPv2 (IPv4), VRRPv3(IPv4, IPv6), VARP(IPv4, IPv6) + +2. Support multiple VRRP instances (groups) per interface + +3. Support VRRP on VLAN, PortChannel and Ethernet interfaces + +4. Support Uplink interface tracking + +5. Support preemption of a Master when a high priority VRRP node comes up + +6. Support configurable priority for VRRP instance + +7. Support configuration and management of various VRRP parameters + +8. VRRP support on non-default VRF + +9. Support REST access to VRRP objects + + + +Following requirements are beyond scope of this release. + +2. SNMP and gNMI access support to VRRP objects + + + +## 2.2 Configuration and Management Requirements + +This feature will support configuration and display CLIs to control and monitor VRRP parameters + +1. Support configuration of VRRP instances per interface + +2. Support configuration of VIP, priority, hello interval and preemption for VRRP instance + +3. Support configuration of uplink interface track + +4. Support configuration of VRRP protocol version + +5. Allow users to configure track interface weight to provide flexible policy of Master to Backup switchover + +6. Support display of various VRRP parameters as well as states using CLIs. + +7. Support configuration of VARP on interface + + + +## 2.3 Scalability Requirements + +1. Max number of VRRP instances: 128 +2. Max number of VRRP enabled interfaces: 128 +3. Max number of VRRP instances per interface: 16 +4. Max number of tracked interfaces per VRRP Instance: 8 +5. Max IP addresses per VRRP instance: 4 +6. Max number of VIP for VARP instance on an interface: 4 + + + +## 2.4 Warm Boot Requirements + +VRRP module is warm reboot compliant. That is, VRRP docker will be restarted as part of warm-reboot but will come up afresh and will build the state from scratch. The behavior of VRRP will be like Cold reboot. + +VARP module does not store any states. During warm-reboot router will be restarted. After restart VIP will be programmed in forwarding as per configuration. + + + +# 3 Feature Description + +## 3.1 Target Deployment use cases + +The following are some of the deployment use cases for VRRP and VARP + +- The Leaf nodes of the Clos network to provide first hop redundency to connected devices + + + +## 3.2 Functional Description of VRRP + +VRRP specifies an election protocol to provide the virtual router function described earlier. All protocol +messaging is performed using IP multicast datagrams, thus the protocol can operate over a variety of +multiaccess LAN technologies supporting IP multicast. Each VRRP virtual router has a single well-known +MAC address allocated to it. This document currently only details the mapping to networks using the IEEE +802 48-bit MAC address. The virtual router MAC address is used as the source in all periodic VRRP +messages sent by the Master router to enable bridge learning in an extended LAN. +A virtual router is defined by its virtual router identifier (VRID) and a set of IP addresses. A VRRP router +may associate a virtual router with its real addresses on an interface, and may also be configured with +additional virtual router mappings and priority for virtual routers it is willing to backup. The mapping +between VRID and addresses must be coordinated among all VRRP routers on a LAN. However, there is +no restriction against reusing a VRID with a different address mapping on different LANs. The scope of +each virtual router is restricted to a single LAN. +To minimize network traffic, only the Master for each virtual router sends periodic VRRP Advertisement +messages. A Backup router will not attempt to preempt the Master unless it has higher priority. This +eliminates service disruption unless a more preferred path becomes available. It's also possible to +administratively prohibit all preemption attempts. The only exception is that a VRRP router will always +become Master of any virtual router associated with addresses it owns. If the Master becomes unavailable +then the highest priority Backup will transition to Master after a short delay, providing a controlled +transition of the virtual router responsibility with minimal service interruption. +The VRRP protocol design provides rapid transition from Backup to Master to minimize service +interruption, and incorporates optimizations that reduce protocol complexity while guaranteeing +controlled Master transition for typical operational scenarios. The optimizations result in an election +protocol with minimal runtime state requirements, minimal active protocol states, and a single message +type and sender. The typical operational scenarios are defined to be two redundant routers and/or distinct +path preferences among each router. A side effect when these assumptions are violated (i.e., more than two redundant paths all with equal preference) is that duplicate packets may be forwarded for a brief period during Master election. However, the typical scenario assumptions are likely to cover the vast majority of deployments, loss of the Master router is infrequent, and the expected duration in Master election convergence is quite small ( << 1 second ). Thus the VRRP optimizations represent significant +simplifications in the protocol design while incurring an insignificant probability of brief network +degradation. +Though VRRP standard protocol present in RFC 3768 and RFC 5798 are complete in itself, there are few +limitations/drawbacks of the protocol: + +1. The point of failure which VRRP safeguards against is the interface on which VRRP instance is +present and the router as a whole. Even if all of the connectivity of master to the external network +fails, VRRP will still not trigger the failover to the backup gateway. Uplink interface tracking feature has been introduced to overcome this limitation +2. Only master router sends the advertisements, and backups just listen to them. Since backups do +not send any periodic message, it is difficult to ascertain that the backups are still active, and the +healthiness of the backup routers can not be known until the failover happens. +3. VRRP owner - if a VRRP instance's virtual IP address is same as the real interface IP address. VRRP owner has special privileges (priority is 255) and can preempt any other router acting as master. This usually causes unwanted temporary network disruption after the non-owner master had stabilized +after the failover from owner. +4. A non-owner master can not accept packets destined to the virtual IP address. Only the owner can +accept and respond to such packets. +5. Two virtual routers with same VRIDs but on different IP subnets can not co-exist on the same +VLAN because of the virtual MAC address clash. + +### 3.2.1 Virtual Router Owner + +VRRP instance whose virtual IP address (VIP) is same as real interface address is called the owner of virtual router and has the highest priority. VRRP owner is supported in SONIC's VRRP implementation. + +### 3.2.2 Virtual MAC Address + +Following virtual MAC addresses is used by the protocol (as per RFC 5798) + +IPv4 case: **00-00-5e-00-01-{vrid}** + +IPv6 case: **00-00-5e-00-02-{vrid}** + +where, vrid is user configured 1-byte virtual router identifier. VRID has interface scope; that is, VRID has to be unique among the VRRP instances on an interface. However, same VRID can be used for two or more virtual router instances across different interfaces. + +### 3.2.3 Preemption + +Preemption is turned on by default. Even if preemption is disabled, it does not affect the owner router since owner preempts the active master. Mastership switchover causes unnecessary temporary network disruption. + +### 3.2.4 VRRP Advertisement Frame + +VRRP control packets have IP protocol type as 112 (reserved for VRRP). IPv4 and IPv6 VRRP keepalives are sent to VRRP multicast address 224.0.0.18 and FF02::12 respectively. Source MAC in VRRP control packets is virtual MAC address + +### 3.2.5 ARP Request Handling + +Only master responds to the ARP requests for virtual IP address. In ARP replies sent by master, the source MAC in Ethernet header and ARP payload is virtual MAC address. + +### 3.2.6 Uplink Interface Tracking + +Interfaces other than the VRRP instance interface can be tracked for up/down events. When interface-tracking is enabled in the VRRP instance configuration, the tracked interface's operational status will be monitored. When a interface operational down event is detected on a tracked-interface, the track-priority/weight is subtracted from the current router’s priority value. Similarly, when interface operational up event is detected on the tracked-interface, the track-priority/weight is added to the router’s current priority value. + +The dynamic change of router priority can trigger mastership switchover if the preemption is enabled. **However, if the router is an owner, the mastership switchover will not happen**. + +Maximum number of interfaces that can be tracked for a virtual router instance is 8. + +## 3.3 Functional Description of VARP + +In most of Leaf-Spine deployments, redundancy in Spine layer is required to achieve high availability and to prevent network service disruption. Modern layer 2 networks adopted loop-free and balanced path networks using Multi Chassis Link Aggregation topologies with LACP port channels, leaving loop control methods (STP) as second protection layer. Spines also supports layer 3 networks, using ECMP in a scalable network topology. For unicast redundancy in layer 3, a common method is use Virtual Router Redundancy Protocol (VRRP) to provide a simple and unique gateway for Leaf level. Although VRRP provides redundancy, it is active-standby protocol and do not provide a balanced data traffic distribution over Multi Chassis Link Aggregated topologies. + +VARP provides better data traffic balancing and faster redundancy convergence, implementing active-active First Hop Router Redundancy to provide active/active unicast IP routing. + +The primary benefit of using VARP is that all configured routers are active and are able to perform routing. VARP also provides rapid failover in the event of a link or switch failure, while enabling the sharing of IP forwarding load between both switches. VARP requires configuring the same virtual-router IP address on the appropriate VLAN interfaces of both peers, as well as a global unique virtual-router MAC address. VARP functions by having both switches respond to ARP requests and GARP for a configured IP address with the “virtual-router” MAC address. This address is receive-only MAC address and no packet is ever sent with this address as its source. If IP routing is enabled, received packets will be routed as follows: when the DMAC of a packet destined to a remote network matches the configured “virtual-router” MAC address, each MLAG peer locally forwards the traffic to its next hop destination. + +### 3.3.1 Virtual MAC Address + +Administrator assigns virtual MAC address to the switch. The switch maps all virtual router IP addresses to this MAC address. The address is receive-only; the switch never sends packets with this address as the source. + +# 4 Feature Design + + + +## 4.1 Design Overview + +### 4.1.1 Basic Approach + +Keepalived (https://www.keepalived.org/) open source code is chosen for VRRP control plane code. + +![VRRP Keepalived Design](images/VRRP_Keepalived_design.PNG "Figure : Design") + + +### 4.1.2 Container + +A new container "vrrp" has been added to host VRRP protocol operation. VRRP source code is not maintained in SONIC repos. Instead, VRRP code is downloaded from keeplaived project repo at the compile time, patches are applied to it (for any fixes by us) and then compiled/linked with SONIC binary. + +New container "vrrp" handles both VRRP and VARP functionality. For VARP the configuration is programmed in kernel through 'vrrpmgrd' module and 'vrrpsyncd' programs the hardware entry in SAI. + +### 4.1.3 SAI Overview + +VRRP virtual router interface can be created by setting the attribute “SAI_ROUTER_INTERFACE_ATTR_IS_VIRTUAL” with **create_router_interface** API. + + * @brief RIF creation is a virtual RIF. + * + * Create a Virtual RIF object, which only programs the ingress router MAC. + * This simplifies the management of VRRP master router's configuration in + * SAI adapter, as defined by RFC 5798 (or similar proprietary protocols). + * Using a Virtual RIF allows SAI to optimize resources, so neighbor entries + * cannot be learned on a Virtual RIF. On a virtual RIF following attributes + * are invalid: ADMIN state, MTU size, packet action and multicast enable. + * Alternatively VRRP can also be configured using native RIF objects without + * using VIRTUAL attribute, with the expectation that SAI adapter will consume + * resources that will not be used. + * + * @type bool + * @flags CREATE_ONLY + * @default false + */ + + SAI_ROUTER_INTERFACE_ATTR_IS_VIRTUAL, +## 4.2 DB Changes + +At a high level below are some of the interactions between relevant components and the DB involved for VRRP support in SONiC architecture. + + + +![VRRP Arch](images/VRRP_architecture.PNG "Figure : Arch") +__Figure 1: VRRP Architecture__ + +### 4.2.1 CONFIG_DB changes + +**VRRP_TABLE** + +Producer: config manager + +Consumer: VrrpMgr + +Description: New table that stores VRRP configuration for per interface + VRID. + +Schema: + +``` +;New table +;holds the VRRP configuration per interface and VRID + +key = VRRP_TABLE:interface_name:vrid:address_family + ; Interface name string like Vlan1 or PortChannel002 or Ethernet4 + ; vrid is an integer +; field = value +vrid = 1*3DIGIT ; VRRP Instance Identifier +address_family = "IPv4"/"IPv6"; Address Family of VRRP instances +vip = ip_address ; Virtual IPv4/IPv6 address. This is a list of IPv4/IPv6 addresses +priority = vrrp_priority ; Priority of VRRP instance +adv_interval = 1*3DIGITS ; Advertisement interval for VRRP. Default = 1sec +state = vrrp_state ; String denoting the state of VRRP instance +version = vrrp_version ; VRRP version. VRRP for IPv6 will always be version 3 +pre_empt = "true"/"false" ; VRRP premption is enabled? Default is True +track_interface = track_interface ; List of interfaces tracked by a VRRP instance + |weight|; This is repeated for the configured tracking interfaces +``` + +Example:- + +**Key**: VRRP_TABLE:Vlan11:1 + +**Value**: + +​ 'vrid': '1', +​ 'vip': '4.1.1.100,', + +​ 'priority': '50', + +​ 'adv_interval': '1', +​ 'state': '', + +​ 'version': '2', +​ 'pre_empt': 'True', +​ 'track_interface': +​ 'Ethernet7|weight|20, + +​ PortChannel001|weight|40 + + + +Example:- Entery with multiple virtual IPs + +**Key**: VRRP_TABLE:Vlan11:1 + +**Value**: + +​ 'vrid': '1', +​ 'vip': '4.1.1.100,4.1.1.200,4.1.1.201,4.1.1.202,', + +​ 'priority': '50', + +​ 'adv_interval': '1', +​ 'state': '', + +​ 'version': '2', +​ 'pre_empt': 'True', +​ 'track_interface': +​ 'Ethernet7|weight|20, + +​ PortChannel001|weight|40 + + + +**Key**: VRRP_TABLE:Vlan12:2 + +**Value**: + +​ 'vrid': '2', +​ 'vip': 'fe80::100,4::100', + +​ 'priority': '50', + +​ 'adv_interval': '1', +​ 'state': '', + +​ 'version': '3', +​ 'pre_empt': 'True', +​ 'track_interface': +​ 'Ethernet8|weight|20, + +​ PortChannel002|weight|40 + + + +Example:- VRRP for IPv6 address + +**VARP_TABLE** + +Producer: config manager + +Consumer: VrrpMgr + +Description: New table that stores VARP configuration for per interface. + +Schema: + +``` +;New table +;holds the VARP configuration per interface + +key = VARP_TABLE:interface_name:address_family + ; Interface name string like Vlan1 or PortChannel002 or Ethernet4 +address_family = "IPv4"/"IPv6"; Address Family of VRRP instances +vip = ip_address ; Virtual IPv4/IPv6 address. This is a list of IPv4/IPv6 addresses +``` + + + +``` +;New table +;holds the VARP_GLOBAL configuration per router + +key = VARP_GLOBAL_TABLE:address_family + ; Interface name string like Vlan1 or PortChannel002 or Ethernet4 +vmac = mac ; Virtual mac address for all the VARP Virtual IP +``` + + + +Example:- + +**Key**: VRRP_TABLE:Vlan15 + +**Value**: +​ 'vip': '15.1.1.100,16.1.1.100', + +### 4.2.2 APP_DB Changes + +**VRRP_TABLE** + +Producer: VrrpMgr + +Consumer: VrrpOrch + +Description: This is a new table that contains VRRP and VARP state information + +Schema: + +``` +; New table +; holds the VRRP state and VMAC information + +key = VRRP_TABLE:interface_name:vip:type + interface_name ; interface name as a string. Vlan, Ethernet or PortChannel + vip ; virtual IP address in a.b.c.d/32 or a:b:c::d format + type ; IP(v6) address type string. + +; field = value +vmac = virtual_mac_address ; Virtual MAC address associated with VRRP instance +``` + +Example:- + +**Key**: VRRP_TABLE:Vlan1000:[40.10.8.101/32:ipv4] + +**Value**: "vmac":"00:00:5e:00:01:08" + +## 4.3 Modules Design and Flows + + + + + + +## 5 CLI + +SONIC Click based configuration and monitoring CLIs have been introduced in SONIC for VRRP + +### 5.1 Configuration Commands + +#### 5.1.1 Configuration Commands for IPv4 VRRP + +``` +configure interface vrrp add +This command adds/enables a IPv4 VRRP instance on an interface. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VRRP is to be enabled. +- vrid - VRRP instance identifier. + +configure interface vrrp remove +This command removes a IPv4 VRRP instance from an interface. + +configure interface vrrp vip add +This command adds a virtual IP address for a VRRP instance on an interface. User is allowed to create multiuple VIPs for a IPv4 VRRP instance. VIP must fall in the interface's subnet. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VIP is being added +- vrid - VRRP instance identifier. +- virtual_ip_address - VIP address in dotted decimal IPv4 address + +configure interface vrrp vip remove +This command deletes a already configured VIP from a IPv4 VRRP instance + +configure interface vrrp priority +This command configures priority for a VRRP instance +- priority - a number between 1 and 254 with 1 being the lowest and 254 being the highest priority. Default is 100. Priority 255 is reserved for owner VRRP router. + +configure interface vrrp adv_interval +This command configures VRRP periodic advertisement interval for a VRRP instance +- interval - a number between 1 and 255. Unit is in seconds. Default is 1sec. + +configure interface vrrp pre_empt enable +This command enables premeption of a Master when a higher priority VRRP router arrives +- enable - Enable premeption. Default is enabled. + +configure interface vrrp pre_empt disable +This command disables premeption of a Master when a higher priority VRRP router arrives +- disable - Disable premeption. Default is enabled. + +configure interface vrrp version +This command configures VRRP protocol version for IPv4 VRRP instances + +configure interface vrrp track_interface add +This command adds a track interface to a VRRP Instance. A maximum of 8 track interfaces can be added to a VRRP instance. +- track_interface - Interface to track. Interface can be Ethernet, Vlan or PortChannel +- weight - weight or importance assigned to the track_interface. When track interface goes down, the priority of VRRP instance will be reduced by weight + +configure interface vrrp track_interface remove +This command removes an already configured track interface from a IPv4 VRRP Instance. +``` + +#### 5.1.1 Configuration Commands for IPv6 VRRP + +``` +configure interface vrrp6 add +This command adds/enables a IPv6 VRRP instance on an interface. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VRRP is to be enabled. +- vrid - VRRP instance identifier. + +configure interface vrrp6 remove +This command removes a IPv6 VRRP instance from an interface. + +configure interface vrrp6 vip add +This command adds a virtual IPv6 address for a IPv6 VRRP instance on an interface. User is allowed to create multiuple VIPs for a VRRP instance. VIP must fall in the interface's subnet. +- interface_name - name of interface (Ethernet or Vlan or PortChannel) over which VIP is being added +- vrid - VRRP instance identifier. +- virtual_ip_address - VIP address in dotted decimal IPv4 address + +configure interface vrrp6 vip remove +This command deletes a already configured VIP from a IPv6 VRRP instance + +configure interface vrrp6 priority +This command configures priority for a VRRP instance +- priority - a number between 1 and 254 with 1 being the lowest and 254 being the highest priority. Default is 100. Priority 255 is reserved for owner VRRP router. + +configure interface vrrp6 adv_interval +This command configures VRRP periodic advertisement interval for a VRRP instance +- interval - a number between 1 and 255. Unit is in seconds. Default is 1sec. + +configure interface vrrp6 pre_empt enable +This command enables premeption of a Master when a higher priority VRRP router arrives +- enable - Enable premeption. Default is enabled. + +configure interface vrrp6 pre_empt disable +This command disables premeption of a Master when a higher priority VRRP router arrives +- disable - Disable premeption. Default is enabled. + +configure interface vrrp6 version +This command configures VRRP protocol version for IPv4 VRRP instances + +configure interface vrrp6 track_interface add +This command adds a track interface to a IPv6 VRRP Instance. A maximum of 8 track interfaces can be added to a VRRP instance. +- track_interface - Interface to track. Interface can be Ethernet, Vlan or PortChannel +- weight - weight or importance assigned to the track_interface. When track interface goes down, the priority of VRRP instance will be reduced by weight + +configure interface vrrp6 track_interface remove +This command removes an already configured track interface from a VRRP Instance. +``` + +SONIC kLISH based configuration and monitoring CLIs have been introduced in SONIC for VRRP + +``` +[no] configure vrrp address-family {ipv4 | ipv6} +This command configures/removes VRRP instance in IPv4 or IPv6 address-family. +- vrid - VRRP instance identifier. + +[no] vip +This command configures/removes a virtual IP address for a VRRP instance on an interface. User is allowed to create multiuple VIPs for a IPv4 VRRP instance. VIP must fall in the interface's subnet. For IPv6 VRRP instance VIP could be any link-local address. +- virtual_ip_address - VIP address + +priority +This command configures priority for a VRRP instance +- priority_value - a number between 1 and 254 with 1 being the lowest and 254 being the highest priority. Default is 100. Priority 255 is reserved for owner VRRP router. + +adv_interval +This command configures VRRP periodic advertisement interval for a VRRP instance +- int_value - a number between 1 and 255. Unit is in seconds. Default is 1sec. + +[no] pre_empt +This command enables/disables premeption of a Master when a higher priority VRRP router arrives. + +[no] vrrpv3 +This command configures/removes VRRP version 3 for a IPv4 VRRP instance. IPv4 VRRP instances are version 2 by default. + +[no] track_interface + +This command adds/removes a track interface to a VRRP Instance. A maximum of 8 track interfaces can be added to a VRRP instance. +- track_ifname - Interface to track. Interface can be Ethernet, Vlan or PortChannel +- weight - weight or importance assigned to the track_interface. When track interface goes down, the priority of VRRP instance will be reduced by weight + +``` + +SONIC kLISH based configuration and monitoring CLIs have been introduced in SONIC for VARP + +``` +[no] ip virtual-router address +This command configures/removes virtual IPv4 on an interface. +- vip - IPv4 virtual address. + +[no] ipv6 virtual-router address +This command configures/removes virtual IPv6 on an interface. +- vip - IPv6 virtual address. + +[no] ip virtual-router mac-address +This command configures/removes virtual mac address for all the IPv4/IPv6 virtual addresses. + + +``` + +#### + +### 5.2 Show Commands + +``` +show vrrp + - lists all the VRRP instances including their current state + Sample output:- +admin@sonic:~$ show vrrp +Interface_Name VRID State VIP Cfg_Prio Curr_Prio + Vlan1 1 Backup 4.1.1.100 100 120 + Vlan2 2 Master 4.1.2.100 100 100 + Vlan3 3 Backup 4.1.3.100 100 100 + Vlan4 4 Backup 4.1.4.100 100 100 + Vlan5 5 Master 4.1.5.100 100 100 + + +show vrrp + - This command displays data about a VRRP instance in detail + Sample output:- +admin@sonic:~$ show vrrp Vlan1 1 +Vlan1, VRID 1 +Version is 2 +State is Backup +Virtual IP address: + 4.1.1.100 +Virtual MAC address is 0000.5e00.0101 +Track interface: + Intfname State Priority + Ethernet7 Up 10 + PortChannel001 Up 10 +Configured Priority is 100, Current Priority is 120 +Advertisement interval is 1 sec +Preemption is enabled +``` + +## + +``` +show varp + - lists all the VARP instances including their current state + Sample output:- +IP virtual router is configured with MAC address: 24cd.5a29.cc31 +Interface IP Address Virtual IP Address Status Protocol +Vlan15 10.1.1.3/24 10.1.1.15 up up +Vlan15 10.1.1.3/24 10.1.1.16 up up +Vlan15 10.1.1.3/24 10.1.1.17 up up +Vlan20 10.12.1.6/24 10.12.1.51 up up +Vlan20 10.12.1.6/24 10.12.1.53 up up +Vlan20 10.12.1.6/24 10.12.1.55 up up + + +``` + + +## 6 Serviceability and Debug + +The existing logging mechanisms shall be used. Proposed debug framework shall be used for internal state dump. + +## 7 Warm Reboot Support + +Currently, warm-reboot is not supported for VRRP. That is, warm-reboot will simply restart the VRRP docker without VRRP storing any data for warm restart. + +VARP is stateless, during warm-reboot router will go down and restart with new configuration. + +## 8 Unit Test cases + +| Test Cases | UT Action and observation | Remarks | +| ---------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------- | +| Configuration | | | +| | Validate IPv4 VRRP config on Physical Interface | | +| | Validate IPv4 VRRP config on Ve Interface | | +| | Ensure that VMAC is generated based on VRID for IPv4 VRRP | | +| | Ensure VMAC is programmed in FDB and VIP in LPM | | +| | Validate IPv6 VRRP config on Physical Interface | | +| | Validate IPv6 VRRP config on Ve Interface | | +| | Ensure that VMAC is generated based on VRID for IPv6 VRRP | | +| | Ensure VMAC is programmed in FDB and VIP in LPM for IPv6 VRRP | | +| | Validate VRRP on more than one interface | | +| | Validate VRRP with more than one VIP per instance | | +| | Validate multiple VRRP instance per interface | | +| Failover | | | +| | Validate VRRP master failover with priority change | | +| | Validate VRRP master failover by disabling a vrrp group | Add 'disable' feature if needed and feasible | +| | Validate VRRP master failover by removing all virtual-ip | | +| | Validate VRRP master failover by removing interface IPv4/IPv6 address | | +| | Validate VRRP master failover by deleting master VRRP session | | +| | Validate VRRP master failover by bringing down interface | | +| | Validate VRRP master failover by deleting interface | | +| VRRP parameter changes | | | +| | Validate gratutious ARP | | +| | Validate adv interval | | +| | Validate version change for IPv4 VRRP instance | | +| ARP/ND resolve to VIP | | | +| | validate ARP resolution from host to VRRP session VIP | Hosts should get arp response with VMAC as source MAC | +| | Perform Master failover and check that ARP resolution to VIP will happen with VMAC | | +| Ping to VIP | | | +| | Validate ping to IVRRP from host, backup & non vrrp router works | | +| L3 Forwarding with VIP as GW | | | +| | Validate that IPv4/IPv6 traffic forwarding with VIP as gateway works from source to destination host | | +| | Perform VRRP master failover and check if traffic forwarding continues with new master | | +| VRRP Owner | | | +| | Configure VRRP for IPv4/IPv6 session with VIP same as interface IP and ensure this session becomes master and priority of session becomes max=255 | | +| | Ensure ping to VIP works in owner case | | +| | Ensure arp/nd resolution with VIP as gateway in owner case | ARP/ND should get resolved with VMAC and not interface MAC | +| | Ensure ping and traffic forwarding continues after owner failover and comes back | | +| | Traffic forwarding works with VIP as gateway | | +| Interface tracking | | | +| | Run below tests with Ping and traffic running | | +| | Enable interface tracking in a VRRP group with priority, bring down the port and ensure VRRP session priority comes down and session transition from Master to Backup State | | +| | Enable the tracked interface and check if session priority gets restored and session becomes master | | +| | Repeat this test on Port-channel interface | | +| | Repeat this test by deleting and re-adding port-channel interface | | \ No newline at end of file diff --git a/doc/aaa/Persistent HTTP Connections from CLI.md b/doc/aaa/Persistent HTTP Connections from CLI.md new file mode 100644 index 0000000000..4d455c44f7 --- /dev/null +++ b/doc/aaa/Persistent HTTP Connections from CLI.md @@ -0,0 +1,60 @@ +Persistent HTTP connections from CLI +==================================== + +The SONiC CLI provided by the management framework container runs an instance of +klish, which provides a fixed set of commands. The CLI is simply a front-end to +the REST server, and each command is mapped to a corresponding REST endpoint. + +# Command Flow + +When the user enters a specific command on the CLI, the corresponding `ACTION` +tag in the CLI XML specification shells out to a Python script with any +arguments and an optional template to format the returned values. This script +connects to the REST server on the local machine over HTTPS, retrieves and +formats the JSON response. The Python script then exits, terminating any HTTP +connection that had been set up. + +This is the current behavior, even without RBAC support, which means that every +command will need to set up a new HTTPS connection. However, when RBAC is +enabled, it is not likely to cause a noticeable performance impact, since the +system is already incurring the TLS overhead. + +As can be seen from the flow above, it is not possible to set up a persistent +HTTP connection, since every command spawns a new connection. + +# Alternative Designs + +This section describes some alternative designs that will enable the CLI to +create a persistent connection. + +## Proxy service + +As part of the management framework, we can add a "proxy" service that is +spawned with the CLI. This service will set up a secure HTTP connection for the +authenticated user, and create a local unix socket that is accessible only by +that user. The CLI XML will remain unchanged, but the Python ApiClient class +will be changed to connect to the local socket. + +This will still create independent HTTP connections, but they can be insecure +connections, while the proxy service will transfer the connections from the +insecure unix socket to the secure tunnel, which will reduce the TLS connection +time. + +**Note:** The security considerations have not been completely mapped out, and +this may open the system up to security holes. + +## Klish modification + +This approach considers modifying the Klish executable. When Klish is spawned, +it will set up the HTTPS connection and keep it alive as long as the CLI is +active. Each `ACTION` tag will call into klish functions that will connect to +the existing HTTPS connection. + +This approach is the most secure option, however, it needs heavy modification to +klish, and there are several unknowns at this time. + +## No modification - Buzznik + +This approach leaves the design as is for the Buzznik release. Every command +will continue to create a new HTTPS connection, as it does today, and will tear +down the connection on completion of the request. diff --git a/doc/aaa/SONiC RBAC HLD.md b/doc/aaa/SONiC RBAC HLD.md new file mode 100644 index 0000000000..2f6d4a3e37 --- /dev/null +++ b/doc/aaa/SONiC RBAC HLD.md @@ -0,0 +1,460 @@ + + +# Authentication and Role-Based Access Control + +# High Level Design Document +#### Rev 0.1 + +# Table of Contents +- [Revision History](#revision-history) +- [About this Manual](#about-this-manual) +- [Scope](#scope) +- [Definitions/Abbreviations](#definitionsabbreviations) +- [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + + [1.1.1 Functional Requirements](#111-functional-requirements) + - [1.1.1.1 NBI Authentication](#1111-nbi-authentication) + - [1.1.1.2 CLI Authentication to REST server](#1112-cli-authentication-to-rest-server) + - [1.1.1.3 Translib Enforcement of RBAC](#1113-translib-enforcement-of-rbac) + - [1.1.1.4 Linux Groups](#1114-linux-groups) + - [1.1.1.5 Certificate-based Authentication for gNMI and REST](#1115-certificate-based-authentication-for-gnmi-and-rest) + - [1.1.1.6 Local User Management and UserDB Sync](#1116-local-user-management-and-userdb-sync) + + [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + + [1.1.3 Scalability Requirements](#113-scalability-requirements) + - [1.1.3.1 REST Server](#1131-rest-server) + - [1.1.3.2 gNMI Server](#1132-gnmi-server) + - [1.1.3.3 Translib](#1133-translib) + + [1.1.4 Warm Boot Requirements](#114-warm-boot-requirements) + * [1.2 Design Overview](#12-design-overview) + + [1.2.1 Basic Approach](#121-basic-approach) + + [1.2.2 Container](#122-container) + + [1.2.3 SAI Overview](#123-sai-overview) +- [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + * [2.2 Functional Description](#22-functional-description) +- [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + + [3.2.1 CONFIG DB](#321-config-db) + + [3.2.2 APP DB](#322-app-db) + + [3.2.3 STATE DB](#323-state-db) + + [3.2.4 ASIC DB](#324-asic-db) + + [3.2.5 COUNTER DB](#325-counter-db) + * [3.3 Switch State Service Design](#33-switch-state-service-design) + + [3.3.1 Orchestration Agent](#331-orchestration-agent) + + [3.3.2 Other Process](#332-other-process) + * [3.4 SyncD](#34-syncd) + * [3.5 SAI](#35-sai) + * [3.6 User Interface](#36-user-interface) + + [3.6.1 Data Models](#361-data-models) + + [3.6.2 CLI](#362-cli) + - [3.6.2.1 Configuration Commands for User Management](#3621-configuration-commands-for-user-management) + - [3.6.2.2 Show Commands](#3622-show-commands) + - [3.6.2.3 Debug Commands](#3623-debug-commands) + - [3.6.2.4 IS-CLI Compliance](#3624-is-cli-compliance) + + [3.6.3 REST API Support](#363-rest-api-support) +- [4 Flow Diagrams](#4-flow-diagrams) +- [5 Error Handling](#5-error-handling) + * [5.1 REST Server](#51-rest-server) + * [5.2 gNMI server](#52-gnmi-server) + * [5.3 CLI](#53-cli) + * [5.4 Translib](#54-translib) +- [6 Serviceability and Debug](#6-serviceability-and-debug) +- [7 Warm Boot Support](#7-warm-boot-support) +- [8 Scalability](#8-scalability) +- [9 Unit Test](#9-unit-test) +- [10 Internal Design Information](#10-internal-design-information) + +# Revision History +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/22/2019 | Jeff Yin | Initial version | +| 0.2 | 10/30/2019 | Jeff Yin | Revision after joint review with Broadcom/Dell | + +# About this Manual +This document provides a high-level design approach for authentication and RBAC in the SONiC Management Framework. + +For authentication, this document describes how the CLI and programmatic interfaces (REST, gNMI) -- collectively referred to in this document as the northbound interfaces (NBIs) -- will authenticate users and the supported credentials and methods. + +For authorization, this document describes a centralized authorization approach to be implemented in the Translib component of the SONiC Management Framework. + +# Scope +This document covers the interfaces and mechanisms by which NBIs will authenticate users who wish to access and configure the SONiC system via the Management Framework. It will also cover RBAC enforcement. + +This document will NOT extensively cover (or assumes the pre-existence of): +- Implementation of remote authentication and authorization (RADIUS, TACACS+, etc.) +- Public Key Infrastructure management: X.509v3 certificate installation, deletion, trust store management, etc. + + +# Definitions/Abbreviations + +| **Term** | **Meaning** | +|:--------------------------|:-------------------------------------| +| RBAC | Role-Based Access Control | +| AAA | Authentication, Authorization, Accounting | +| CLI | Command-Line Interface | +| REST | REpresentational State Transfer | +| gNMI | gRPC Network Management Interface | +| NBI | North-bound Interfaces (CLI, REST, gNMI) | + + + +# 1 Feature Overview + +## 1.1 Requirements + +- The SONiC Management Framework must support authenticated access for the various supported northbound interfaces (NBIs): CLI, REST, and gNMI. Since the CLI works with the REST server, it must also be authenticated with the REST server. +- The NBIs must pass along the username info to Translib, so that Translib can enforce role-based access (RBAC). +- For RBAC, Linux Groups will facilitate authorization. Initially, two roles will be supported: read/write and read-only. Additionally, remotely-authenticated users who map to a defined role will be authenticated as a static global user on the system. These limitations should be revisited at a later date. +- Local user management: CLIs and APIs for creating and managing local users on the system -- their usernames, passwords, and roles. + +### 1.1.1 Functional Requirements + +#### 1.1.1.1 NBI Authentication +A variety of authentication methods must be supported: +* **CLI** authentication is handled via the same mechanisms supported by SSH + - Password-based Authentication + - Public-key Authentication +* **REST** authentication + - Password-based Authentication with JWT token-based authentication + - Certificate-based Authentication with JWT token-based authentication + - The REST server must be enhanced to accept all types of authentication concurrently +* **gNMI** Authentication + - Password-based Authentication with Token-based authentication + - Certificate-based Authentication + +#### 1.1.1.2 CLI Authentication to REST server +Given that the CLI module works by issuing REST transactions to the Management Framework's REST server, the CLI module must also be able to authenticate with the REST server when REST authentication is enabled. This can be accomplished via the following steps: +1. When a user is created on the system, a self-signed certificate is generated with the username embedded into the Subject field. That self-signed certificate will be stored in the user's home directory, with access permissions limited to that user only. The self-signed certificate will be used for mutual authentication from the KLISH shell to the REST server. +**Design note:** Certificate-based authentication is desired over password-based authentication for the CLI-to-REST connection because it prevents the need to store and forward any plaintext passwords. Per-user certificates with strict file permissions are desired so as to ensure that users cannot drop into the Linux shell and perform operations as other than themselves (e.g., via `curl` with a shared certificate, or someone else's certificate). +2. When a user logs into the switch, the switch will authenticate the user by using the username and password credentials. +3. When the KLISH process is started, its UID and GID are set to those of the user that was authenticated by `sshd` or `login` (as through the console). +4. When the KLISH shell is spawned, it will create a persistent, local HTTPS connection over an internal socket, and send the REST request to authenticate this KLISH session as the logged-in user, which can be looked up through NSS by using `getpwuid()`. This authentication request will contain the user's client certificate. +5. The REST server will authenticate the user and return a token with the username encoded in it. The KLISH CLI must cache this token and use it for all future requests from this KLISH session. The REST server will also maintain this token to username mapping with it so as to identify all future requests as well. This will also allow the REST server in creating audit logs for all the requests sent from KLISH, as well as pass the username to Translib for RBAC enforcement. +6. KLISH CLI session will store the authentication token, and from then on, KLISH CLI will send REST requests using the persistent connection with the authentication token in the HTTP header to the REST server for all the CLI commands. +7. The KLISH session must be able to cleanly handle ctrl-c breaks while it is busy waiting on a response from the REST server. +8. When the user exits the KLISH CLI session, the HTTP persistent connection is closed. The REST server will clean up the corresponding authentication token for its corresponding KLISH CLI Client. + +#### 1.1.1.3 Translib Enforcement of RBAC +The Translib module in the [management framework](https://github.com/project-arlo/SONiC/blob/master/doc/mgmt/Management%20Framework.md) is a central point for all commands, regardless of the interface (CLI, REST, gNMI). Therefore, the RBAC enforcement will be done in Translib library. + +The CLI, REST, and gNMI will result in Translib calls with xpath and payload. Additionally, the REST and gNMI server modules must pass the username to the Translib. The RBAC checks will be enforced in the **Request Handler** of the Translib. The Request Handler processes the entire transaction atomically. Therefore, if even one operation in the transaction is not authorized, the entire transaction is rejected with the appropriate "Not authorized" error code. If the authorization succeeds, the transaction is passed to the Common Application module for further ‘transformation’ into the ABNF format. The rest of the flow of the transaction through the management framework is unmodified. + +At bootup time of the manageability framework (or gNMI container), it is recommended to cache the Privilege, Tenant and Resource Tables(RBAC tables) stored in the Config DB. This is because every command needs to access the information in the RBAC tables in order to authorize the access for the information in other tables. Alternately, instead of caching the entire RBAC tables, the information can be cached once the record is read from the DB. Additionally, the Translib must listen to change notifications on these RBAC Tables in order to keep its cache current. + +As described in section [1.1.1.4 Linux Groups](#1114-linux-groups), the enforcement of the users and roles in Linux will be done via the Linux groups. A user can use Linux commands on the Linux shell and create users and Linux groups (which represent the roles). This will mean that the information in the RBAC tables is no longer current. In order to keep the information in the RBAC tables and the Linux `/etc/passwd` file in sync, a service must run on the host (not the container) to keep the databases in sync. + +Since Translib is the main authority on authorized operations, this means that the NBIs cannot render to the user what they are and are not allowed to do. The CLI, therefore, renders the entire command tree to a user, even the commands they are not authorized to execute. + +#### 1.1.1.4 Linux Groups +Initially, only two roles will be supported: +- `admin` -- can perform read and write functions across all attributes (i.e., GET/PUT/PATCH/etc.) +- `operator` -- can only perform read functions on all attributes (i.e., GET only) +These privileges will be enforced by Translib. + +RBAC will be facilitated via Linux Groups: +- Local users + - Local user with `Operator` role is added into `operator` group + - Local user with `Admin` role is added into `admin` group and is a `sudoer`. +- Remote users + - Remote users with `Operator` role are mapped to the same global `remote-user` user who is part of `operator` group + - Remote users with `Admin` role are mapped to the same global `remote-user-su` user who is part of `admin` group and is a `sudoer` + - This means that all remote users will share the same accounts on the system, which also means they will share the same /home directory and certificate to log into the REST server. + + In the future, this "global user" approach will be revisited so that remote users are authenticated with their own username so that their activities may be properly audited. + +#### 1.1.1.5 Certificate-based Authentication for gNMI and REST +For the initial release, it will be assumed that certificates will be managed outside of the NBIs. That is, no CLIs or REST/gNMI interfaces will be implemented to support public key infrastructure management for certificates and certificate authorities. Certificates will be manually generated and copied into the system via Linux utilities. + +The exception to this is the self-signed certificates used for CLI authentication to the REST server. Those certificates will be auto-generated when a user is created on the system. These certificates should be copied to a user's home directory _as well as_ the trust store so that the REST server can use them for authenticating local CLI sessions. + +User certificates must be stored in a user's home directory. Their corresponding private keys must also be stored in the user's home directory, albeit with restricted permissions so that they are not readable by other users. + +The gNMI server will use a trust store for CA certificates from a location such as `/usr/local/share/ca-certificates`. The trust store itself must be managed by [existing Linux tools](https://manpages.debian.org/jessie/ca-certificates/update-ca-certificates.8.en.html). + +The gNMI server must implement a method by which a username can be determined from the presented client certificate, so that the username can thus be passed to Translib for RBAC enforcement. The username will be derived from the Subject field of the X.509v3 certificate. + +Users must be informed by way of documentation so that they know how to manage their certificate infrastructure in order to properly facilitate gNMI communication. + +The REST server must use the same certificate scheme as the gNMI server to validate client certificates. + +#### 1.1.1.6 Local User Management and UserDB Sync +An interface must be developed for local user management, so that administrators can add users and assign passwords and roles to them. Administrators with the appropriate role must be able to add/delete users, modify user passwords, and modify user roles. They must be able to do so through all of the NBIs, meaning that a YANG model and CLI tree must be developed. + +Users must be added to the Linux database (`/etc/passwd`, `/etc/group`, and optionally `/etc/shadow`). That is where a user is mapped to a Linux [User Identifier](https://en.wikipedia.org/wiki/User_identifier) (UID) and primary [Group Identifier](https://en.wikipedia.org/wiki/Group_identifier) (GID). When users are created they also need to be assigned roles. Roles are simply defined as Linux groups (`/etc/group`) and assigned to users as [Supplementary GIDs](https://en.wikipedia.org/wiki/Group_identifier#Supplementary_groups). + +When a user is created it also needs to be assigned certificates that will allow them to communicate with the REST server. Finally, all users need to be added to the REDIS database (see [section 3.2.6]()) where additional information about each user can be stored (e.g. *tenant*). + +Since these operations (i.e. creating Linux users, assigning certificates, etc.) are non-trivial, the process of creating users will be entrusted to the Host Account Management Daemon (**hamd**). + +##### 1.1.1.6.1 Host Account Management Daemon (hamd) + +The **hamd** process runs on the host. It is accessed via a DBus interface that provides the ability to access and/or modify the host's Linux database (`/etc/passwd`, `/etc/group`, and optionally `/etc/shadow`). Since DBus is a secured interface we can control which processes will be allowed to access **hamd**. + +The **hamd** process will provide the following DBus APIs to create/modify/delete user and group (role) accounts: + +- **useradd**: To create new users (similar to GNU [useradd](http://man7.org/linux/man-pages/man8/useradd.8.html)) +- **userdel**: To delete a user (similar to GNU [userdel](http://man7.org/linux/man-pages/man8/userdel.8.html)) +- **passwd**: To change a user password (similar to GNU [passwd](http://man7.org/linux/man-pages/man1/passwd.1.html)) +- **set_roles**: To set a user's roles (similar to GNU [usermod](http://man7.org/linux/man-pages/man8/usermod.8.html)) +- **groupadd**: To create new groups/roles (similar to GNU [groupadd](http://man7.org/linux/man-pages/man8/groupadd.8.html)) +- **groupdel**: To delete groups/roles (similar to GNU [groupdel](http://man7.org/linux/man-pages/man8/groupdel.8.html)) + +##### 1.1.1.6.2 User management with hamd + +Applications that need to manage users (for example **click** or **klish**) can do so by using **hamd**'s DBus **ham.accounts** interface. DBus services such as **hamd** publish their interfaces. This can be retrieved and analyzed at runtime in order to understand the used implementation. The resulting introspection data is in XML format. DBus debug tools such as **qdbus** can be used to retrieve this data. At the time this document was written, the DBus XML definition for the APIs defined in the previous section was: + +> ```xml +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> +> ``` + +##### 1.1.1.6.3 The hamctl shell program + +A utility program, **hamctl**, is provided to make it easier for operators to interact with **hamd** from a Linux shell (e.g. bash). This is primarily a debug tool and *should not be invoked from other programs*. Programs should use the DBus interface described above. + +Users logged in at a shell terminal can control **hamd** (e.g. ask it to create or delete a user) with **hamctl**. **hamctl** is sell-documented. Simply invoke "**hamctl --help**" to get the list of commands available. + +##### 1.1.1.6.4 Name Service Switch + +In addition to providing APIs to create/modify/delete user and group (role) accounts, **hamd** also provides APIs to simply read user and group (role) accounts. Here's the list: + +- **getpwnam**: To retrieve user credentials (similar to POSIX [getpwnam](http://man7.org/linux/man-pages/man3/getpwnam.3.html)) +- **getpwuid**: To retrieve user credentials (similar to POSIX [getpwuid](http://man7.org/linux/man-pages/man3/getpwnam.3.html)) +- **getgrnam**: To retrieve group/role credentials (similar to POSIX [getgrnam](http://man7.org/linux/man-pages/man3/getgrnam.3.html)) +- **getgrgid**: To retrieve group/role credentials (similar to POSIX [getgrgid](http://man7.org/linux/man-pages/man3/getgrnam.3.html)) + +These APIs, however, are meant to be invoked through [NSS](https://en.wikipedia.org/wiki/Name_Service_Switch) (name service switch). That is, applications running in containers can simply continue invoking the standard POSIX APIs (`getpwnam()`, `getgrnam()`, etc) and a Host Account Management NSS module will ensure that the credentials get retrieved from **hamd** running on the host. The HAM NSS module (`libnss_ham.so.2`) need be installed and configured (`/etc/nsswitch.conf`) in the containers that require access to the host's Linux database. + +### 1.1.2 Configuration and Management Requirements +An interface and accompanying CLI must be developed for local user management. Local users should be configurable like any other feature: via CLI, REST, and gNMI. Additionally, users may also be created and managed via Linux commands in the Bash shell. This will add additional complexity and require a service to sync between the Redis DB and the Linux user database. + + +### 1.1.3 Scalability Requirements +Adding authentication to NBIs will result in some performance overhead, especially when doing operations involving asymmetric cryptography. Care should be taken to leverage performance-enhancing features of a protocol wherever possible. + +#### 1.1.3.1 REST Server +- Persistent HTTP connections can be used to preserve TCP sessions, thereby avoiding handshake overhead. +- TLS session resumption can be used to preserve the TLS session layer, thereby avoiding TLS handshake overhead and repeated authentication operations (which can involve expensive asymmetric cryptographic operations) +- Token-based authentication via JSON Web Tokens (JWT) will be used to preserve sessions for users who have already authenticated with password-based authentication, so that they do not need to constantly re-use their passwords. + +#### 1.1.3.2 gNMI Server +- TLS session resumption can be used to preserve the TLS session layer, thereby avoiding TLS handshake overhead and repeated authentication operations (which can involve expensive asymmetric cryptographic operations) + +#### 1.1.3.3 Translib +- Translib will cache all the user information along with the privilege and resource information to avoid the overhead of querying them every time we receive a request. +- Will rely on notification to update any change in the user information, privilege or resource information + +### 1.1.4 Warm Boot Requirements +N/A + +## 1.2 Design Overview +### 1.2.1 Basic Approach +The code will extend the existing Klish (CLI) and REST Server modules in the sonic-mgmt-framework repository. Klish will be extended to enable authentication to the REST server (depending on the ultimately chosen approach), and the REST Server will need to be extended to map transactions to a user and pass that username data to the Translib. + +The gNMI server, which currently exists in the sonic-telemetry repository, needs to support passing the username down to Translib as well. + +The Translib code (also in sonic-mgmt-framework) will be extended to support RBAC via Linux Groups. It will receive username data from the REST/gNMI NBIs and perform the Group lookup for a given user. + +For user management, a service must run on the host to sync the Redis RBAC tables with the Linux user database and vice-versa. + +### 1.2.2 Container +SONiC Management Framework, gNMI Telemetry containers + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Enterprise networks that enforce authentication for their management interfaces. + +## 2.2 Functional Description +This feature enables authentication and Role-Based Access Control (RBAC) on the REST and gNMI programmatic interfaces that are provided by the SONiC Management Framework and Telemetry containers. With respect to authentication, these programmatic interfaces will support password-based authentication with tokens, and certificate-based authentication. + +Since the Klish CLI in the management framework communicates with the REST server in the back-end, the solution will also be extended to support REST authentication. + +RBAC will be enforced centrally in the management framework, so that users accessing the system through varying interfaces will be limited to the same, consistent set of operations and objects depending on their role. Users' roles will be mapped using Linux Groups. + +Users and their role (group) assignments may be managed via Linux tools or the NBIs. + +# 3 Design +## 3.1 Overview +(TODO/DELL: Draw a picture) + +## 3.2 DB Changes +### 3.2.1 CONFIG DB +To support this the following tables will be introduced in the CONFIG DB : +* **UserTable** + + This table contains the username to role mapping needed for enforcing the authorization checks. It has the following columns : + * *user* : This is the username being authorized. This is a string. + * *tenant* : This contains the tenant with which the user is associated. This is a string + * *role* : This specifies the role associated with the username in the tenant. This is a comma separated list of strings. + The UserTable is keyed on <***user, tenant***>. + + **Note**: The UserTable will _not_ store users' salted+hashed passwords due to security concerns surrounding access restrictions to the DB; instead, that information will be maintained in `/etc/shadow` as per Linux convention. + +* **PrivilegeTable** + + This table has provides the information about the type of operations that a particular role is authorized to perform. The authorization can be performed at the granularity of a feature, feature group, or the entire system. The table has the following columns : + * *role* : The role associated with the user that is being authorized. This is a string. + * *feature* : This is feature that the role is being authorized to access. The granularity of the feature can be : + * *feature* - A logical grouping of multiple commands. If the user is authorized to access a particular feature, the column contains the tag associated with that feature. (More on tagging later. This will be implemented in Phase 2 of RBAC.) + * *feature-group* - A logical grouping of multiple features. If the user is authorized to a feature-group, the column contains the name of the feature-group. (More on feature-group later. This will be implemented in Phase 2 of RBAC.) + * *entire-system* - If the user is being granted access to the entire system, the column contains *all* + * *permissions* : Defines the permissions associated with the role. This is a string. + * *none* - This is the default permissions that a role is created with. A role associated with *none* permission cannot access any resources on the system to read, or to modify them. + * *read-only* - The role only has read access to the resources associated with the *feature*. + * *read-write* - The role has permissions to read and write (create, modify and delete) the resources associated with the *feature*. + The PrivilegeTable is keyed on <***role, feature***> + +* **ResourceTable** + (To be implemented in Phase 2) + Though the resources are statically tagged with the features that they belong to, a ResourceTable is still needed so as to allow for future extensibility. It is possible that in the future, a customer wants a more granular control over the authorization and wants to either sub partition the features or override the default tagging associated with a feature. The ResourceTable will allow for this support in the future. In the Phase 2, this table will be create using the default tagging associated with the resources. + * *resource* : The xpath associated with the resource being accessed. This is a string. + * *feature-name* : The tag of the feature this resource belongs to. + The ResourceTable is keyed on <***resource, feature***> + +* **TenantTable** + (To be implemented in Phase 3 or when Multi-tenancy is introduced) + In most systems today, a single SONiC system will serve multiple tenants. A tenant is a group of users who have a different privileges for resource instances. As SONiC becomes multitenant, RBAC needs to account for this when authorizing users. The TenantTable is needed to enable this and has the following columns : + * *resource* : The xpath associated with the resource being accessed. This is a string. + * *tenant* : The tenant for which the resource partitioning is being done. This is a string. + * *instances* : The instances of the *resource* allocated to this *tenant*. This is a list of instances. + The TenantTable is keyed on <***resource, tenant***> + +### 3.2.2 APP DB +N/A + +### 3.2.3 STATE DB +N/A + +### 3.2.4 ASIC DB +N/A + +### 3.2.5 COUNTER DB +N/A + +## 3.3 Switch State Service Design +### 3.3.1 Orchestration Agent +N/A + +### 3.3.2 Other Process +N/A + +## 3.4 SyncD +To facilitate the sync between users in the UserDB and with users in Linux, a service must run on the host that listens for both changes to `/etc/passwd` and changes to UserDB, since users can be created via either interface (UserDB via NBIs / Linux commands). + +This service runs a process that uses the POSIX [inotify APIs](http://man7.org/linux/man-pages/man7/inotify.7.html) to register for file system events like changes to `/etc/passwd` and/or `/etc/shadow`. +Another approach would be to have a process started by `systemd` on changes to `/etc/passwd` or `/etc/group`, and that process would simply reconcile the Redis DB with what is found in those files. `systemd` allows starting processes based on file create/delete/modify. + +A user can be created either via CLI or REST. It is the responsibility of this service to ensure that when the user information is added to the User DB, the appropriate user and groups are also created in the `/etc/passwd` and `/etc/groups` files. +This way, the User DB information and the Linux groups information is always in sync. + +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models +TBD from developer +(TODO/DELL) + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands for User Management +Users may be managed via Linux tools like `useradd`, `usermod`, `passwd`, etc. They may also be managed via configuration. + +##### username +`username password role ` -- Configures a user on the system with a given name, password, and role. +* **name** is a text string of 1-32 alphanumeric characters +* **password-string** is a text string of 1-32 alphanumeric characters +* **role-string** is a text string consisting of a role name. In the initial release, the user is recommended to use "admin" and "operator" roles, as other roles will not be supported. A text string is desired instead of keywords so that in the future, more roles may be implemented and expanded. +* Configuring another a user with the same **name** should result in modification of the existing user. + +`no username ` -- Deletes a user from the system. +* **name** is a text string of 1-32 alphanumeric characters + +#### 3.6.2.2 Show Commands +N/A + +#### 3.6.2.3 Debug Commands +N/A + +#### 3.6.2.4 IS-CLI Compliance +N/A + +### 3.6.3 REST API Support +Ability to configure local users via REST API. + +# 4 Flow Diagrams +N/A + +# 5 Error Handling +## 5.1 REST Server +The REST server should return standard HTTP errors when authentication fails or if the user tries to access a forbidden resource or perform an unauthorized activity. + +## 5.2 gNMI server +The gNMI server should return standard gRPC errors when authentication fails. + +## 5.3 CLI +Authentication errors will be handled by SSH. However, the CLI must gracefully handle authorization failures from the REST server (the authorization failure would originate from Translib of course). While the CLI will render all of the available commands to a user, the user will actually only be able to execute a subset of them. This limitation is a result of the design decision to centralize RBAC in Translib. Nevertheless, the CLI must inform the user when they attempt to execute an unauthorized command. + +## 5.4 Translib +Translib will authorize the user and when the authorization fails will return appropriate error string to the REST/gNMI server. + +If a user authenticates but is not part of one of the pre-defined groups, they will not be allowed to do anything at all on the system. + +# 6 Serviceability and Debug +All operations performed by NBIs (CLI commands, REST/gNMI operations) should be logged/audited with usernames attached to the given operation(s) performed. + +Initially, users who are remotely authenticated will share a common role-specific username, so there will be a limitation here. + +# 7 Warm Boot Support +N/A + +# 8 Scalability +See previous section 1.1.3: Scalability Requirements + +# 9 Unit Test + +### Table 3: Test Cases +| **Test Case** | **Description** | +|:--------------------------|:-------------------------------------| +| REST with password | Authenticate to REST server with username/password and perform some operations | +| REST with token | Perform subsequent operations with token, ensure username/password are not re-prompted | +| REST authorized RBAC | Perform authorized operations as both `Admin` and `Operator` via REST | +| REST unauthorized RBAC | Attempt unauthorized operations as both `Admin` and `Operator` via REST | +| CLI with password | SSH to the system with username/password and execute some commands | +| CLI with RSA | SSH to the system with pubkey and execute some commands | +| CLI authorized RBAC | SSH to the system and perform authorized commands | +| CLI unauthorized RBAC | SSH to the system and perform unauthorized commands | +| RBAC no-group | Create a user and assign them to a non-predefined group; make sure they can't perform any operations | +| gNMI authentication | Test the same authentication methods as REST, but for gNMI instead | +| gNMI authorization | Test the same authorization as REST, but for gNMI instead | diff --git a/doc/aaa/SONiC RBAC.md b/doc/aaa/SONiC RBAC.md new file mode 100644 index 0000000000..194032fdaa --- /dev/null +++ b/doc/aaa/SONiC RBAC.md @@ -0,0 +1,481 @@ +# Authentication and Role-Based Access Control +# High Level Design Document +#### Rev 0.1 + +# Table of Contents +- [Revision History](#revision-history) +- [About this Manual](#about-this-manual) +- [Scope](#scope) +- [Definitions/Abbreviations](#definitionsabbreviations) +- [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + + [1.1.1 Functional Requirements](#111-functional-requirements) + - [1.1.1.1 NBI Authentication](#1111-nbi-authentication) + - [1.1.1.2 CLI Authentication to REST server](#1112-cli-authentication-to-rest-server) + - [1.1.1.3 REST/gNMI server authentication methods CLIs](#1113-restgnmi-server-authentication-methods-clis) + - [1.1.1.4 Linux Groups](#1114-linux-groups) + - [1.1.1.5 Customizable roles](#1115-customizable-roles) + - [1.1.1.6 Local User Management and UserDB Sync](#1116-local-user-management-and-userdb-sync) + + [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + + [1.1.3 Scalability Requirements](#113-scalability-requirements) + - [1.1.3.1 Translib](#1131-translib) + + [1.1.4 Warm Boot Requirements](#114-warm-boot-requirements) + * [1.2 Design Overview](#12-design-overview) + + [1.2.1 Basic Approach](#121-basic-approach) + + [1.2.2 Container](#122-container) + + [1.2.3 SAI Overview](#123-sai-overview) +- [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) + * [2.2 Functional Description](#22-functional-description) +- [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + + [3.2.1 CONFIG DB](#321-config-db) + + [3.2.2 APP DB](#322-app-db) + + [3.2.3 STATE DB](#323-state-db) + + [3.2.4 ASIC DB](#324-asic-db) + + [3.2.5 COUNTER DB](#325-counter-db) + * [3.3 Switch State Service Design](#33-switch-state-service-design) + + [3.3.1 Orchestration Agent](#331-orchestration-agent) + + [3.3.2 Other Process](#332-other-process) + * [3.4 SyncD](#34-syncd) + * [3.5 SAI](#35-sai) + * [3.6 User Interface](#36-user-interface) + + [3.6.1 Data Models](#361-data-models) + + [3.6.2 CLI](#362-cli) + - [3.6.2.1 Configuration Commands for User Management](#3621-configuration-commands-for-user-management) + - [3.6.2.2 Show Commands](#3622-show-commands) + - [3.6.2.3 Debug Commands](#3623-debug-commands) + - [3.6.2.4 IS-CLI Compliance](#3624-is-cli-compliance) + + [3.6.3 REST API Support](#363-rest-api-support) +- [4 Flow Diagrams](#4-flow-diagrams) +- [5 Error Handling](#5-error-handling) + * [5.1 REST Server](#51-rest-server) + * [5.2 gNMI server](#52-gnmi-server) + * [5.3 CLI](#53-cli) + * [5.4 Translib](#54-translib) +- [6 Serviceability and Debug](#6-serviceability-and-debug) +- [7 Warm Boot Support](#7-warm-boot-support) +- [8 Scalability](#8-scalability) +- [9 Unit Test](#9-unit-test) + +# Revision History +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 04/07/2020 | Nirenjan Krishnan | Initial version | +| 0.2 | 06/12/2020 | Nirenjan Krishnan | Update with review comments | + +# About this Manual +This document provides a high-level design approach for authentication and RBAC in the SONiC Management Framework. + + +# Scope + +TBD + +# Definitions/Abbreviations + +| **Term** | **Meaning** | +|:--------------------------|:-------------------------------------| +| RBAC | Role-Based Access Control | +| AAA | Authentication, Authorization, Accounting | +| CLI | Command-Line Interface | +| REST | REpresentational State Transfer | +| gNMI | gRPC Network Management Interface | +| NBI | North-bound Interfaces (CLI, REST, gNMI) | +| HAM | Host Account Manager | + + + +# 1 Feature Overview + +## 1.1 Requirements + +- For RBAC, the roles will be defined in the configuration database. Initially, + two roles will be supported; admin (with read/write privileges), and operator + (with read-only privileges); however, it shall be possible for the users with + admin role to define additional roles. +- HAM will become the single source of truth for user management on SONiC + systems. +- Local user management: CLIs and APIs for creating and managing local users on + the system -- their usernames, passwords, and roles. +- Authentication control: CLIs and APIs to modify the allowed authentication + modes on the REST and gNMI interfaces. +- CLI authentication mechanisms to REST server. + +### 1.1.1 Functional Requirements + +#### 1.1.1.1 NBI Authentication +A variety of authentication methods must be supported: +* **CLI** authentication is handled via user detection on the Unix Socket. +* **REST** authentication + - Password-based Authentication + - JWT token-based authentication + - Certificate-based Authentication +* **gNMI** Authentication + - Password-based Authentication + - JWT token-based authentication + - Certificate-based Authentication + +#### 1.1.1.2 CLI Authentication to REST server +Given that the CLI module works by issuing REST transactions to the Management +Framework's REST server, the CLI module must also be able to authenticate with +the REST server when REST authentication is enabled. This can be accomplished +via the following steps: +1. KLISH must be launched as the corresponding user + - This is performed by the `-u` argument to `docker exec` +2. REST will listen in on a Unix socket. Listener will detect the user at the + remote end of the connection, and use that for authentication/authorization. +3. The REST server will authenticate the user and return a token with the + username and role encoded in it. The KLISH CLI must cache this token and use + it for all future requests from this KLISH session. The REST server will + validate this token to username/role mapping with it so as to identify all + future requests as well. This will also allow the REST server in creating + audit logs for all the requests sent from KLISH, as well as pass the username + and role to Translib for RBAC enforcement. +4. KLISH CLI session will store the authentication token, and from then on, + KLISH CLI will send REST requests using the persistent connection with the + authentication token in the HTTP header to the REST server for all the CLI + commands. +5. The KLISH session must be able to cleanly handle ctrl-c breaks while it is + busy waiting on a response from the REST server. +6. When the user exits the KLISH CLI session, the HTTP persistent connection is + closed. The REST server will clean up the corresponding authentication token + for its corresponding KLISH CLI Client. + +#### 1.1.1.3 REST/gNMI server authentication methods CLIs + +The CLI shall provide commands to modify the authentication methods of both the +REST and gNMI interfaces. + +#### 1.1.1.4 Linux Groups +Roles will not have a direct map to a primary Linux group, however, +supplementary groups are used to allow access to additional functionalities. +- `admin` users will be added to `sudo` and `docker` groups +- `operator` users will be added to `docker` group only. +- All non-admin users will be added to the `docker` group only. + +The default shell for all users is set to a script. This script will determine +if the user is a member of the admin group, and if so, drop them into a Bash +shell, otherwise, it will spawn KLISH to drop them into the CLI. Non-admin users +will only have access to the CLI and not be able to access Bash. Users cannot +run `system` commands from KLISH. + +#### 1.1.1.5 Customizable roles + +**This is planned for future support.** + +Admin users shall be able to create customizable roles for further granularity beyond read-only and read-write. For instance, a `secadmin` role shall allow the users to configure security parameters on the switch. The custom role shall have a default deny policy. Custom roles shall map a set of features, and their access level to the feature. A feature is a set of YANG paths. The default policy is no access to a feature, if it is not specified in the role. + +``` +[ + { + "name": "secadmin", + "features": [ + { + "feature": "aaa", + "access": "rw" + }, + { + "feature": "net", + "access": "ro" + } + ] + } +] +``` + +Features map to a set of YANG paths. The features are saved in the +ResourceTable, but a default feature set is defined in the system to allow users +to start from a known state. Sample feature description is shown below. + +``` +[ + { + "name": "aaa", + "paths": [ + "openconfig-aaa:aaa/system/users/user={username}", + "openconfig-aaa:aaa/system/users" + ] + }, + { + "name": "net", + "paths": [ + "openconfig-net:net/xyz" + ] + } +] +``` + + +#### 1.1.1.6 Local User Management and UserDB Sync +An interface must be developed for local user management, so that administrators can add users and assign passwords and roles to them. Administrators with the appropriate role must be able to add/delete users, modify user passwords, and modify user roles. They must be able to do so through all of the NBIs, meaning that a YANG model and CLI tree must be developed. + +Users must be added to the Linux database (`/etc/passwd`, `/etc/group`, and optionally `/etc/shadow`). That's where a user is mapped to a Linux [User Identifier](https://en.wikipedia.org/wiki/User_identifier) (UID) and primary [Group Identifier](https://en.wikipedia.org/wiki/Group_identifier) (GID). When users are created they also need to be assigned roles. Roles are defined in the REDIS database. Users may also be assigned to supplementary Linux groups (`/etc/group`) in order to allow functionality required for their roles (e.g. Docker group membership in order to spawn the CLI). + +Since these operations (i.e. creating Linux users, assigning roles, supplementary groups, etc.) are non-trivial, the process of creating users will be entrusted to the Host Account Management Daemon (**hamd**). + +##### 1.1.1.6.1 Host Account Management Daemon (hamd) + +The hamd process runs on the host. It is accessed via a DBus interface that provides the ability to access and/or modify the host's Linux database (`/etc/passwd`, `/etc/group`, and optionally `/etc/shadow`). Since DBus is a secured interface we can control which processes will be allowed to access hamd. + +The hamd process will provide the following APIs to create/modify/delete user and group (role) accounts: + +- **useradd**: To create new users (similar to GNU [useradd](http://man7.org/linux/man-pages/man8/useradd.8.html)) +- **userdel**: To delete a user (similar to GNU [userdel](http://man7.org/linux/man-pages/man8/userdel.8.html)) +- **passwd**: To change a user password (similar to GNU [passwd](http://man7.org/linux/man-pages/man1/passwd.1.html)) +- **setroles**: To set a user's roles (similar to GNU [usermod](http://man7.org/linux/man-pages/man8/usermod.8.html)) +- **groupadd**: To create new groups/roles (similar to GNU [groupadd](http://man7.org/linux/man-pages/man8/groupadd.8.html)) +- **groupdel**: To delete groups/roles (similar to GNU [groupdel](http://man7.org/linux/man-pages/man8/groupdel.8.html)) +- **usermod**: To create or modify users. This method will accept a username, + hashed password, and a list of strings corresponding to the roles. + +The hamd process will add the users to the User Table in the Config DB. + +##### 1.1.1.6.2 Name Service Switch + +In addition to providing APIs to create/modify/delete user and group (role) accounts, hamd also provides APIs to simply read user and group (role) accounts. Here's the list: + +- **getpwnam**: To retrieve user credentials (similar to POSIX [getpwnam](http://man7.org/linux/man-pages/man3/getpwnam.3.html)) +- **getpwuid**: To retrieve user credentials (similar to POSIX [getpwuid](http://man7.org/linux/man-pages/man3/getpwnam.3.html)) +- **getgrnam**: To retrieve group/role credentials (similar to POSIX [getgrnam](http://man7.org/linux/man-pages/man3/getgrnam.3.html)) +- **getgrgid**: To retrieve group/role credentials (similar to POSIX [getgrgid](http://man7.org/linux/man-pages/man3/getgrnam.3.html)) + +These APIs, however, are meant to be invoked through [NSS](https://en.wikipedia.org/wiki/Name_Service_Switch) (name service switch). That is, applications running in containers can simply continue invoking the standard POSIX APIs (`getpwnam()`, `getgrnam()`, etc) and a Host Account Management NSS module will ensure that the credentials get retrieved from hamd running on the host. The HAM NSS module (`libnss_ham.so.2`) need be installed and configured (`/etc/nsswitch.conf`) in the containers that require access to the host's Linux database. + +### 1.1.2 Configuration and Management Requirements + +Local users are managed via all the NBIs, i.e., CLI, REST and gNMI. The admin +users may also create or modify local users from the Bash shell. All requests +will be proxied to HAM to perform the actual user management. + +### 1.1.3 Scalability Requirements + +#### 1.1.3.1 Translib +- Translib will cache all the user information along with the privilege and resource information to avoid the overhead of querying them every time we receive a request. +- Will rely on notification to update any change in the user information, privilege or resource information + +### 1.1.4 Warm Boot Requirements +N/A + +## 1.2 Design Overview +### 1.2.1 Basic Approach + +The Translib code (also in sonic-mgmt-framework) will be modified to support RBAC via Roles, rather than Groups. It will receive username data from the REST/gNMI NBIs and perform the role lookup for a given user. + +Translib shall cache the user table and role list to amortize the cost of a +database lookup over several transactions. + +### 1.2.2 Container +SONiC Management Framework, gNMI Telemetry containers + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Enterprise networks that enforce authentication for their management interfaces. + +## 2.2 Functional Description +This feature enables authentication and Role-Based Access Control (RBAC) on the REST and gNMI programmatic interfaces that are provided by the SONiC Management Framework and Telemetry containers. With respect to authentication, these programmatic interfaces will support password-based authentication with tokens, and certificate-based authentication. + +Since the Klish CLI in the management framework communicates with the REST server in the back-end, the solution will also be extended to support REST authentication. + +RBAC will be enforced centrally in the management framework, so that users accessing the system through varying interfaces will be limited to the same, consistent set of operations and objects depending on their role. Users' roles will be mapped using Linux Groups. + +Users and their role (group) assignments may be managed via the NBIs. HAM shall +provide tools to allow administrators to manage the users and roles from the +shell using the hamctl command. + +# 3 Design +## 3.1 Overview +(TODO/DELL: Draw a picture) + +## 3.2 DB Changes +### 3.2.1 CONFIG DB +* **UserTable** + + This table contains the username to role mapping needed for enforcing the authorization checks. It has the following columns : + * *user* : This is the username being authorized. This is a string. + * *tenant* : This contains the tenant with which the user is associated. This is a string + * *role* : This specifies the role associated with the username in the tenant. This is a comma separated list of strings. + The UserTable is keyed on <***user, tenant***>. + + **Note**: The UserTable will _not_ store users' salted+hashed passwords due to security concerns surrounding access restrictions to the DB; instead, that information will be maintained in `/etc/shadow` as per Linux convention. + +**Future Support** + +* **PrivilegeTable** + + This table provides the information about the type of operations that a particular role is authorized to perform. The authorization can be performed at the granularity of a feature, feature group, or the entire system. The table has the following columns : + * *role* : The role associated with the user that is being authorized. This is a string. + * *feature* : This is feature that the role is being authorized to access. The granularity of the feature can be : + * *feature* - A logical grouping of multiple commands. If the user is authorized to access a particular feature, the column contains the tag associated with that feature. + * *entire-system* - If the user is being granted access to the entire system, the column contains *all* + * *permissions* : Defines the permissions associated with the role. This is a string. + * *none* - This is the default permissions that a role is created with. A role associated with *none* permission cannot access any resources on the system to read, or to modify them. + * *read-only* - The role only has read access to the resources associated with the *feature*. + * *read-write* - The role has permissions to read and write (create, modify and delete) the resources associated with the *feature*. + The PrivilegeTable is keyed on <***role, feature***> + +* **ResourceTable** + Though the resources are statically tagged with the features that they belong to, a ResourceTable is still needed so as to allow for future extensibility. It is possible that in the future, a customer wants a more granular control over the authorization and wants to either sub partition the features or override the default tagging associated with a feature. The ResourceTable will allow for this support in the future. In the Phase 2, this table will be create using the default tagging associated with the resources. + * *resource* : The xpath associated with the resource being accessed. This is a string. + * *feature-name* : The tag of the feature this resource belongs to. + The ResourceTable is keyed on <***resource, feature***> + +* **TenantTable** + (To be implemented in Phase 3 or when Multi-tenancy is introduced) + In most systems today, a single SONiC system will serve multiple tenants. A tenant is a group of users who have a different privileges for resource instances. As SONiC becomes multitenant, RBAC needs to account for this when authorizing users. The TenantTable is needed to enable this and has the following columns : + * *resource* : The xpath associated with the resource being accessed. This is a string. + * *tenant* : The tenant for which the resource partitioning is being done. This is a string. + * *instances* : The instances of the *resource* allocated to this *tenant*. This is a list of instances. + The TenantTable is keyed on <***resource, tenant***> + +### 3.2.2 APP DB +N/A + +### 3.2.3 STATE DB +N/A + +### 3.2.4 ASIC DB +N/A + +### 3.2.5 COUNTER DB +N/A + +## 3.3 Switch State Service Design +### 3.3.1 Orchestration Agent +N/A + +### 3.3.2 Other Process +N/A + +## 3.4 SyncD + +N/A. HAM is going to be the single source of truth, and it will take care of +synchronizing the User Table and /etc/passwd + +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models + +TBD from developer +(TODO/DELL) + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands for User Management +Users may be managed via Linux tools like `sonic-useradd`, `sonic-usermod`, `passwd`, etc. They may also be managed via configuration. + +##### username +`username password role ` -- Configures a user on the system with a given name, password, and role. +* **name** is a text string of 1-32 alphanumeric characters +* **password-string** is a text string of 1-32 alphanumeric characters +* **role-string** is a text string consisting of a role name. In the initial release, the user is recommended to use "admin" and "operator" roles, as other roles will not be supported. A text string is desired instead of keywords so that in the future, more roles may be implemented and expanded. +* Configuring another a user with the same **name** should result in modification of the existing user. + +`no username ` -- Deletes a user from the system. +* **name** is a text string of 1-32 alphanumeric characters + +##### aaa authentication +``` +aaa authentication {rest-server | gnmi-server} {[password] [token] [certificate]} +aaa authentication {rest-server | gnmi-server} certificate no-authorization +no aaa authentication {rest-server | gnmi-server} +``` + +The `certificate no-authorization` mode will force users to authenticate via +certificate, however, there will be no role lookup performed. All authenticated +users will be treated as having full administrator access. + +The `no aaa ...` command will disable authentication altogether, and will allow +all users to execute admin level commands via the REST/gNMI interfaces. The CLI +however, will still require a valid password based login, due to it being +accessed via SSH. + +##### userrole +`userrole ` - Creates a new custom role and enters userrole config to +enable/disable individual features. +`(config-userrole)# feature {read-only | read-write}` - Enable +feature `` with read-only or read-write access. +`(config-userrole)# no feature ` - Disable access to feature +``. + +`no userrole ` - Deletes a user role from the system. + +#### 3.6.2.2 Show Commands + +##### show users + +`show users` + +This will display a list of configured users on the system. + +##### show roles + +`show roles` + +This will display a list of configured roles on the system. +#### 3.6.2.3 Debug Commands +N/A + +#### 3.6.2.4 IS-CLI Compliance +N/A + +### 3.6.3 REST API Support +Ability to configure local users via REST API. + +# 4 Flow Diagrams +N/A + +# 5 Error Handling +## 5.1 REST Server +The REST server should return standard HTTP errors when authentication fails or if the user tries to access a forbidden resource or perform an unauthorized activity. + +## 5.2 gNMI server +The gNMI server should return standard gRPC errors when authentication fails. + +## 5.3 CLI +Authentication errors will be handled by user detection on Unix sockets. However, the CLI must gracefully handle authorization failures from the REST server. While the CLI will render all of the available commands to a user, the user will actually only be able to execute a subset of them. This limitation is a result of the design decision to centralize RBAC in Translib. Nevertheless, the CLI must inform the user when they attempt to execute an unauthorized command. + + +## 5.4 Translib +Translib will authorize the user and when the authorization fails will return appropriate error string to the REST/gNMI server. Translib will also log an audit message with the username and the command that was attempted. + +If a user authenticates but is not part of one of the pre-defined roles, they will not be allowed to do anything at all on the system. + +# 6 Serviceability and Debug +All operations performed by NBIs (CLI commands, REST/gNMI operations) should be logged/audited with usernames attached to the given operation(s) performed. + +Initially, users who are remotely authenticated will share a common role-specific username, so there will be a limitation here. + +# 7 Warm Boot Support +N/A + +# 8 Scalability +See previous section 1.1.3: Scalability Requirements + +# 9 Unit Test + +### Table 3: Test Cases +| **Test Case** | **Description** | +|:--------------------------|:-------------------------------------| +| REST with password | Authenticate to REST server with username/password and perform some operations | +| REST with token | Perform subsequent operations with token, ensure username/password are not re-prompted | +| REST authorized RBAC | Perform authorized operations as both `Admin` and `Operator` via REST | +| REST unauthorized RBAC | Attempt unauthorized operations as both `Admin` and `Operator` via REST | +| CLI with password | SSH to the system with username/password and execute some commands | +| CLI with RSA | SSH to the system with pubkey and execute some commands | +| CLI authorized RBAC | SSH to the system and perform authorized commands | +| CLI unauthorized RBAC | SSH to the system and perform unauthorized commands | +| RBAC no-group | Create a user and assign them to a non-predefined group; make sure they can't perform any operations | +| gNMI authentication | Test the same authentication methods as REST, but for gNMI instead | +| gNMI authorization | Test the same authorization as REST, but for gNMI instead | +| Create custom role | Create a custom role on the system with individual features | +| Delete custom role | Delete custom role from system | +| REST with custom role authorized | Perform authorized operations with custom role user via REST | +| REST with custom role unauthorized | Perform unauthorized operations with custom role user via REST | +| gNMI with custom role authorized | Perform authorized operations with custom role user via gNMI | +| gNMI with custom role unauthorized | Perform unauthorized operations with custom role user via gNMI | diff --git a/doc/aaa/ldap.md b/doc/aaa/ldap.md new file mode 100644 index 0000000000..a3e8df30df --- /dev/null +++ b/doc/aaa/ldap.md @@ -0,0 +1,2632 @@ +# LDAP Name Service + +## High Level Design Document +#### Rev 0.14 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Overview](#overview) + * [Revision](#revision) + * [Requirements](#requirements) + * [Functional Description](#functional-description) + * [LDAP NSS](#ldap-nss) + * [LDAP Authentication](#ldap-authentication) + * [LDAP Authorization](#ldap-authorization) + * [LDAP Logon Authorization](#ldap-logon-authorization) + * [LDAP Sudoers Authorization](#ldap-sudoers-authorization) + * [Session Setup](#session-setup) + * [Design](#design) + * [Block Diagram of PAM and NSS Configuration](#block-diagram-of-pam-and-nss-configuration) + * [Sequence Diagram of TACPLUS or RADIUS Authentication and LDAP Name Service](#sequence-diagram-of-tacplus-or-radius-authentication-and-ldap-name-service) + * [ConfigDB Schema](#configdb-schema) + * [OC YANG](#oc-yang) + * [Klish](#klish) + * [Click](#click) + * [Known Issues and Notes](#known-issues-and-notes) + * [Use Case](#use-case) + * [Flow Diagrams](#flow-diagrams) + * [Error Handling](#error-handling) + * [Serviceability and Debug](#serviceability-and-debug) + * [Warm Boot Support](#warm-boot-support) + * [Scalability](#scalability) + * [Unit Test](#unit-test) + * [Prerequisites](#prerequisites) + * [UT Authentication LDAP](#ut-authentication-ldap) + * [UT Authentication TACPLUS with Name Service LDAP](#ut-authentication-tacplus-with-name-service-ldap) + * [UT Authentication RADIUS with Name Service LDAP](#ut-authentication-radius-with-name-service-ldap) + * [UT Authentication RADIUS with Logon Authorization LDAP and Name Service LDAP](#ut-authentication-radius-with-logon-authorization-ldap-and-name-service-ldap) + * [UT Authentication RADIUS with Name Service LDAP and sudo Authorization LDAP](#ut-authentication-radius-with-name-service-ldap-and-sudo-authorization-ldap) + * [UT Authentication Non Global NSS PAM SUDO options](#ut-authentication-non-global-nss-pam-sudo-options) + * [References](#references) + * [nss_ldap](#nss_ldap) + * [pam_ldap](#pam_ldap) + * [sudo_ldap](#sudo_ldap) + * [Linux-PAM](#Linux-PAM) + * [NSS Reference](#NSS-Reference) + * [LDAP client configuration file](#LDAP-client-configuration-file) + * [TACACS+ Authentication](#TACPLUS-Authentication) + * [RADIUS Management User Authentication](#RADIUS-Management-User-Authentication) + +# List of Tables + +# Revision + +| Rev | Date | Author | Change Description | +|:---:|:------------:|:------------------:|------------------------------------------------------------| +| 0.1 | 03/20/2020 | Arun Barboza | Initial version | +| 0.2 | 03/23/2020 | Arun Barboza | After Internal Review | +| 0.3 | 04/03/2020 | Arun Barboza | First External Review | +| 0.4 | 04/29/2020 | Arun Barboza | Detail Config DB Table schema | +| 0.5 | 04/30/2020 | Arun Barboza | Updated Unit Test Details | +| 0.6 | 05/04/2020 | Arun Barboza | Fixed typos, omissions, examples | +| 0.7 | 05/06/2020 | Arun Barboza | Updates for Future Release | +| 0.8 | 05/07/2020 | Arun Barboza | Annotations in LDAP table | +| 0.9 | 05/11/2020 | Arun Barboza | LDAP New Mgmt if- Balachandar M. | +| | | | Update for VRF, SRC IP - Suresh R. | +| 0.10| 05/17/2020 | Arun Barboza | KLISH show aaa|ldap output draft. | +| | | | Added missing retry to ldap host. | +| 0.11| 05/17/2020 | Arun Barboza | Update the UT with KLISH examples. | +| 0.12| 05/20/2020 | Arun Barboza | IIFR, clarifications, OCYang, Maps | +| 0.13| 06/22/2020 | Arun Barboza | project-arlo review updates | +| 0.14| 31/08/2021 | Asha Behera | Adding Holdoff timer to delay and consolidate config update| + +# Overview + +Lightweight Directory Access Protocol (LDAP) allows a client to access +information stored in distributed directory services. + +Name Service information typically includes users, hosts, groups. This +information is historically stored in flat files or other information +services (like a directory service). + +This document describes the high level design of LDAP Name service feature +in SONiC. + +# Requirements + +- Obtain User Information from an LDAP server. + +- Enable LDAP to be used as an authentication service. + +- Enable LDAP to be used as an authorization service. + +- Allow an authentication service, authorization service, separate from + Name Service. + +# Functional Description + +## LDAP NSS + +LDAP can be used as an option in the Name Services Switch(NSS) configuration. +The NSS configuration enables various programming APIs to use other sources +than the default files (e.g., Use LDAP directory information instead of +/etc/passwd for user and group information). User information includes uid, +gid, and home directory. + +## LDAP Authentication + +The LDAP Pluggable Authentication Module (PAM) can be used to authenticate +a CLI (SSH, or console) user to a Linux device like SONiC. + +## LDAP Authorization + +### LDAP Logon Authorization + +The LDAP PAM can be used to authorize a user at login. (e.g., ensure the +user logging on is a member of a group aka group-based logon authorization). +This group can represent any membership (e.g., group of users who are +permitted to logon to a host). In PAM, authorization task is done by account +management modules. + +### LDAP Sudoers Authorization + +The sudoers policy plugin determines a user's sudo privileges. This policy +is driven by the /etc/sudoers file. LDAP can be used to drive this policy +by being a source in the NSS configuration. (e.g. can a user be allowed to +execute the click config command?) + +## Session Setup + +When a user logs into a SONiC device, there would need to be some session +environment (e.g. home directory). This can also be provided by PAM session +modules. The pam_open_session(3) mentions tasks such as creating or +mounting a home directory. + +As of writing, the KLISH certificate to authenticate with the REST server +may not be necessary if the move to UNIX domain sockets from TCP sockets +takes place. + +In order to perform RBAC (Role Based Access Control), role(s) may be +needed for the user. The PAM session module could retrieve the same and +store it in the redis ConfigDB UserTable. Some options to retrieve roles. + +- Infer from the user's group membership. A possible way would be to use membership in a group to indicate the role membership (say by using a map). +- Have a role attribute in the user's LDAP entry. This might require an LDAP schema change so could be less desirable. + +The session modules could be made extensible by passing configuration options. + + +# Design + +Publicly available pre-built debian packages for libpam-ldap, libnss-ldap, +and sudo-ldap, will be planned on being used for providing the backend. +Configuration will take place through the Host Config Daemon using +configuration stored in ConfigDB. The ConfigDB tables can be set using +REST/gNMI, KLISH or click. + + +## Block Diagram of PAM and NSS Configuration + + +``` + +-------+ +---------+ + | SSH | | Console | + +---+---+ +----+----+ + | | + | (4) | + | | + | | + V V ++---------------------------------------------+ +| AUTHENTICATION, Name | +| AUTHORIZATION, Service | +| & SESSION SETUP | +| +-------------------+ +-------------------+ | +| | PAM Configuration | | NSS Configuration | | +| | Files | | Files | | +| +-------------------+ +-------------------+ | +| ^ ^ | +---------------------+ +| | | | | | +| | | | (3) | +------------+ | +| +---------------------+-----------+--------+--+ AAA Config | | +| | | +------------+ | +| | | | +| +--------------+ +------------+ | | Host Config Daemon | +| | PAM | | NSS | | +----------^----------+ +| | Libraries | | Libraries | | | +| +------------- + +------------+ | | ++---------------------------------------------+ (2) | + | + | + +---------+ +-------+--------+ + | | (1) | | + | CLI +------------------------------------> ConfigDB | + | | | | + +---------+ +----------------+ + +``` + +1. The CLI is used to update the ConfigDB with the LDAP and AAA configuration. +2. The Host Config Daemon(HCD) reads data from ConfigDB. +3. The HCD configures the PAM & NSS configuration files. (The AAA config + module in the HCD is responsible for modifying the configuration files.) +4. All new CLI sessions (SSH, Console), now use the new PAM & NSS + configurations to obtain name service information, authenticate, and + authorize users. + +## Sequence Diagram of TACPLUS or RADIUS Authentication and LDAP Name Service + +Following is the sequence of events during TACACS+/RADIUS authenticated login +for an SSH client, when LDAP is used for name service: + +``` +SSH Client SONiC Device TACACS+ Server + SSH Server NSS LDAP Plugin or RADIUS Server +---------------------------------------------------------------------- + + | | | | + | | | | + | Login(1) | | | + +---------------->| | | + | | getpwnam_r()(2) | | + | +---------------->| Retrieve | + | | |(3) data from LDAP | + | | user info.(4) | or name service | + | |<----------------+ cache daemon | + | | | | + | (5) | | | + | pam_auth_rqst. | | | + | to PAM library | | | + | | | + | | TACACS+ Authentication START (6) | + | | or RADIUS Access-Request | + | +------------------------------------->| + | | | + | | Authentication REPLY - PASS (7) | + | | or RADIUS Access-Accept | + | |<-------------------------------------+ + | | | + | | | | + | Success(8) | | | + |<----------------+ | | + | | | | +``` + +1. A user tries to login through a SSH client. +2. A getpwnam_r() call is made to obtain user information. +3. If the user is not found in the local /etc/passwd file, the NSS + LDAP plugin retreives the data from the LDAP server (or name service + cache daemon). If not found, login fails. +4. User information is returned to the SSH server. +5. The SSH server makes a PAM Authentication request to the PAM library. +6. The PAM configuration causes a Authentication-Start to TACACS+ server. + (or RADIUS Access-Request to RADIUS server). +7. TACACS+ returns Authentication-Reply (PASS) (or RADIUS returns Access- + Accept) -- (Goto 8.), + If Authentication-Reply (FAIL)/Access-Reject, the user authentication has + failed. End. +8. The user has authenticated successfully. + +Instead of TACACS+, RADIUS (or LDAP) could be used as the authentication +service as well. + +## ConfigDB Schema + + +There will be New Config DB Table schema for holding LDAP Configuration information. + +- LDAP (keys: 'global' [|'nss' | 'sudo' | 'pam']) + +- LDAP_SERVER (keys: .... ) + +- LDAP_MAP (keys[0]: ATTRIBUTE, + OBJECTCLASS, + DEFAULT_ATTRIBUTE_VALUE, + OVERRIDE_ATTRIBUTE_VALUE + keys[1]: )) + +An existing table will be modified to allow for LDAP Authentication, +Authorization, Name Service, and Session Setup. + +- AAA (new keys: 'authorization', 'nss' [, 'open_session']?) + + ? The 'open_session' key will be reserved for Implementation In a Future + Release(IIFR) + + +``` + #--------------------------------------------------# + # # + # LDAP LDAP_SERVER # + # # + #--------------------------------------------------# + # # + # ____ nss .... server_1 # + # | .... # + # global -----+--- sudo .... # + # |___ .... # + # pam .... server_n # + # # + #--------------------------------------------------# + + #--------------------------------------------------# + # # + # LDAP_MAP_xyz # + # # + #--------------------------------------------------# + + #--------------------------------------------------# + # # + # AAA # + # # + #--------------------------------------------------# + # # + # authentication # + # # + # authorization # + # # + # nss # + # # + # [open_session]? # + # # + # ? For Implementation In a Future Release (IIFR) # + # # + #--------------------------------------------------# + +``` + +### LDAP Table Schema + +This table holds some global LDAP configuration values common to NSS, PAM, +and sudoers. (e.g. search base in LDAP) + +``` +; LDAP configuration attributes global to the device. Upto 4 rows can exist in +; the table. Any of the "global" row LDAP attributes can be overriden by the +; server use_type row settings in the same table. +; Only a few settings can be overriden in the per server Table (please see the +; next table) +; Key +global_key = "global" / "nss" / "sudo" / "pam"; + ; + ; Attributes supported in all use_types + ; unless noted Eg: + ; n --> "nss" only + ; s --> "sudo" only + ; p --> "pam" only + ; - --> "global" support only. + ; np --> "nss" and "pam" only + ; + ; (Units,Default[ - meaning])[([snp]*)] + ; Eg: (seconds,(0 - indefinite)) + ; means that + ; the units are in seconds, + ; the default when the attribute is + ; absent is 0, which means wait + ; indefinitely. + ; the attribute is supported on all + ; use_types. + +; Attributes +timelimit = 1*nDIGIT ; Search time limit (secs,(0 - indefinite)) +bind_timelimit = 1*nDIGIT ; Connect time limit (secs,(10)) +idle_timelimit = 1*nDIGIT ; NSS idle time limit (secs,(0 - indefinite))(n) +retry = 1*2DIGIT ; Retry (attempts,(0,none)) +port = 1*5DIGIT ; Port (number,(389)) +scope = 1*nVCHAR ; Search scope ("sub"|"one"|"base",("sub"))(np) +ldap_version = 1DIGIT ; LDAP protocol version (2|3,(3)) +base = 1*nVCHAR ; Search base DN +ssl = 1*nVCHAR ; Use TLS? ("on"|"off"|"start_tls",("off")) +binddn = 1*nVCHAR ; Bind DN (,( - bind anonymously)) +bindpw = 1*nVCHAR ; Bind credentials (,( - bind anonymously)) +pam_filter = 1*nVCHAR ; Filter for retrieving user info. (,)(p) +pam_login_attribute = 1*nVCHAR ; Attribute for retrieving user info. + ; (,("uid"))(p) +pam_group_dn = 1*nVCHAR ; DN of a group for login Authorization (,)(p) +pam_member_attribute = 1*nVCHAR ; Attribute for login Authorization (,)(p) +sudoers_base = 1*nVCHAR ; Sudo LDAP queries search base DN (,)(s) +nss_base_passwd = 1*nVCHAR ; ... (,)(np) +nss_base_group = 1*nVCHAR ; ... (,)(n) +nss_base_shadow = 1*nVCHAR ; ... (,)(n) +nss_base_netgroup = 1*nVCHAR ; ... (,)(n) +nss_base_sudoers = 1*nVCHAR ; ... (,)(n) +nss_initgroups_ignoreusers = 1*nVCHAR ; NOT_FOUND for initgroups(3) (,)(n) +src_ip = IPAddress ; source IPv4|IPv6 addr for LDAP client (,)(-) +vrf = 1*nVCHAR ; vrf_name (,)(-) + +``` + +### LDAP_SERVER Table Schema + +This table holds per server configuration (e.g. connection retries) +``` +; LDAP per server configuration on the device. +; Key +server_key = NameOrIPAddress; LDAP server's DNS name or IPv4|6 addr +; Attributes +port = 1*5DIGIT ; per server Port +use_type = 1*nVCHAR ; "all" / "nss" / "sudo" / "pam" ("all") +priority = 1*2DIGIT ; specify LDAP server's priority (1) +ssl = 1*nVCHAR ; Use TLS? ("on"|"off"|"start_tls",("off")) +retry = 1*2DIGIT ; Retry (attempts,(0,none)) + +``` + +### LDAP_MAP Table Schema + +This table holds the maps for LDAP configuration. + +- LDAP_MAP_ATTRIBUTE +- LDAP_MAP_OBJECTCLASS +- LDAP_MAP_DEFAULT_ATTRIBUTE_VALUE +- LDAP_MAP_OVERRIDE_ATTRIBUTE_VALUE + +``` +; LDAP_MAP +; Key +map_name = "ATTRIBUTE"/ + "OBJECTCLASS"/ + "DEFAULT_ATTRIBUTE_VALUE"/ + "LDAP_MAP_OVERRIDE_ATTRIBUTE_VALUE" ; Map Name +from_key = 1*nVCHAR ; Map key +; Attributes +to = 1*nVCHAR ; Map value +``` + +### AAA Table Schema + +This is an existing table which will hold additional keys for configuring +NSS, and Authorization. For Authentication a key exists already. + +``` +; Key +aaa_key = "authentication" / "authorization" / "nss" ; AAA type +; Attributes +; For "authentication" / "authorization" Key only +login = LIST(1*32VCHAR) ; "local"/"ldap"/"radius"/"tacacs+" ; PAM modules +failthrough = "True" / "False" ; failthrough mechanism for pam modules +debug = "True" / "False" ; Debugging (Developer) +trace = "True" / "False" ; Packet Trace (Developer) +; For "nss" Key only +passwd = "login" / "ldap" / "radius" / "tacacs+" +group = "login" / "local" / "ldap" +shadow = "login" / "local" / "ldap" +netgroup = "local" / "ldap" +sudoers = "ldap" +; For "nss" Key: "login" ==> Name Service db is based on authentication login +``` + +## OC YANG + +OC-Yang extensions need to be written for LDAP for the following: + +- LDAP global configurations. +- LDAP per server configuration. +- LDAP Maps. +- AAA login authorization. +- AAA nss configuration. + +The OC-Yang tree would need to be extended, and modified in the following +areas as follows: [ All new leaves ] +``` +module: openconfig-system + +--rw system +... + +--rw aaa +... + | +--rw authentication +... + | | +--rw config + | | | +--rw authentication-method* (This is existing leaf-list that + needs to be extended for "ldap" + option) +... + | +--rw oc-sys-ext:authorization +... + | | +--rw oc-aaa-ldap-ext:login (New container: LDAP logon authorization) + | | +--rw oc-aaa-ldap-ext:config + | | | +--rw oc-aaa-ldap-ext:authorization-method* union +... + | +--rw oc-aaa-ldap-ext:name-service (New container) + | +--rw oc-aaa-ldap-ext:passwd + | +--rw config + | +--rw oc-aaa-ldap-ext:name-service-method* +... + | +--rw oc-aaa-ldap-ext:shadow + | +--rw config + | +--rw oc-aaa-ldap-ext:name-service-method* +... + | +--rw oc-aaa-ldap-ext:group + | +--rw config + | +--rw oc-aaa-ldap-ext:name-service-method* +... + | +--rw oc-aaa-ldap-ext:netgroup + | +--rw config + | +--rw oc-aaa-ldap-ext:name-service-method* +... + | +--rw oc-aaa-ldap-ext:sudoers + | +--rw config + | +--rw oc-aaa-ldap-ext:name-service-method* +... + | +--rw server-groups + | +--rw server-group* [name] (new LDAP_ALL group) + (new LDAP_SUDO|PAM|NSS groups) +... + | +--rw servers + | +--rw server* [address] + | | | + | | | +... + | | | + | | +--rw oc-aaa-ldap-ext:ldap (New container) + | | +--rw oc-aaa-ldap-ext:config + | | | +--rw oc-aaa-ldap-ext:port? oc-inet:port-number + | | | +--rw oc-aaa-ldap-ext:use-type? enumeration + | | | +--rw oc-aaa-ldap-ext:priority? uint8 + | | | +--rw oc-aaa-ldap-ext:ssl? ldap-ssl-type + | | | +--rw oc-aaa-ldap-ext:retransmit-attempts? uint8 + | | +--ro oc-aaa-ldap-ext:state + | | +--ro oc-aaa-ldap-ext:port? oc-inet:port-number + | | +--ro oc-aaa-ldap-ext:use-type? enumeration + | | +--ro oc-aaa-ldap-ext:priority? uint8 + | | +--ro oc-aaa-ldap-ext:ssl? ldap-ssl-type + | | +--ro oc-aaa-ldap-ext:retransmit-attempts? uint8 + | +--rw oc-aaa-ldap-ext:ldap (New Container) + | +--rw oc-aaa-ldap-ext:config + | | +--rw oc-aaa-ldap-ext:source-interface? -> /oc-if:interfaces/interface/name + | | +--rw oc-aaa-ldap-ext:vrf-name? -> /oc-ni:network-instances/network-instance/name + | | +--rw oc-aaa-ldap-ext:search-time-limit? uint32 + | | +--rw oc-aaa-ldap-ext:bind-time-limit? uint32 + | | +--rw oc-aaa-ldap-ext:retransmit-attempts? uint8 + | | +--rw oc-aaa-ldap-ext:port? oc-inet:port-number + | | +--rw oc-aaa-ldap-ext:version? uint8 + | | +--rw oc-aaa-ldap-ext:base? string + | | +--rw oc-aaa-ldap-ext:ssl? ldap-ssl-type + | | +--rw oc-aaa-ldap-ext:bind-dn? string + | | +--rw oc-aaa-ldap-ext:bind-pw? string + | | +--rw oc-aaa-ldap-ext:source-address? oc-inet:ip-address + | | +--rw oc-aaa-ldap-ext:vrf-name? string + | | +--rw oc-aaa-ldap-ext:idle-time-limit? uint32 + | | +--rw oc-aaa-ldap-ext:nss-base-group? string + | | +--rw oc-aaa-ldap-ext:nss-base-shadow? string + | | +--rw oc-aaa-ldap-ext:nss-base-netgroup? string + | | +--rw oc-aaa-ldap-ext:nss-base-sudoers? string + | | +--rw oc-aaa-ldap-ext:nss-initgroups-ignoreusers? string + | | +--rw oc-aaa-ldap-ext:pam-filter? string + | | +--rw oc-aaa-ldap-ext:pam-login-attribute? string + | | +--rw oc-aaa-ldap-ext:pam-group-dn? string + | | +--rw oc-aaa-ldap-ext:pam-member-attribute? string + | | +--rw oc-aaa-ldap-ext:sudoers-base? string + | | +--rw oc-aaa-ldap-ext:scope? enumeration + | | +--rw oc-aaa-ldap-ext:nss-base-passwd? string +... + | | +--rw oc-aaa-ldap-ext:maps + | | +--rw oc-aaa-ldap-ext:map* [name from] + | | +--rw oc-aaa-ldap-ext:name -> ../config/name + | | +--rw oc-aaa-ldap-ext:from -> ../config/from + | | +--rw oc-aaa-ldap-ext:config + | | | +--rw oc-aaa-ldap-ext:name? enumeration + | | | +--rw oc-aaa-ldap-ext:from? string + | | | +--rw oc-aaa-ldap-ext:to? string + | | +--ro oc-aaa-ldap-ext:state + | | +--ro oc-aaa-ldap-ext:name? enumeration + | | +--ro oc-aaa-ldap-ext:from? string + | | +--ro oc-aaa-ldap-ext:to? string + +... + + +``` + +## REST/gNMI + +REST, and gNMI support would be needed by writing transformer annotation, +overloads, and SONiC Yang. + +## KLISH + +KLISH commands (XML, actioners, renderers) needs to be written for + +- AAA authentication ldap option. + +``` +sonic(config)# [no] aaa authentication login default local \ + [ | group { radius | tacacs+ | ldap } ] +sonic(config)# [no] aaa authentication login default \ + group { radius | tacacs+ | ldap } [ local ] +sonic(config)# [no] aaa authentication login default +``` + +- AAA authorization login ldap option.. + +``` +sonic(config)# [no] aaa authorization login default { group ldap | local } +``` + +- AAA name-service. + +``` +sonic(config)# [no] aaa name-service passwd { group ldap | login | local } +sonic(config)# [no] aaa name-service shadow { group ldap | login | local } +sonic(config)# [no] aaa name-service group { group ldap | login | local } +sonic(config)# [no] aaa name-service netgroup { group ldap | local } +sonic(config)# [no] aaa name-service sudoers { group ldap | local } +``` + +- LDAP global|nss|pam|sudo server authentication, authorization, name-service parameters. + + +``` +ldap-server timelimit <0 - 65535> +- The default value is 0 seconds. + +ldap-server bind-timelimit <0 - 65535> +- The default value is 10 seconds. + +ldap-server idle-timelimit <0 - 65535> +- The default value is 0 seconds. + +ldap-server retry <0 - 10> +- Default value is 0. + +ldap-server port <0 - 65535> +- Default value is 389. + +ldap-server scope [sub|one|base] +- Default value is "sub". + +ldap-server ldap-version [2|3] +- Default value is 3. + +ldap-server base + +ldap-server ssl [on|off|start_tls] +-Default value is "off" + +ldap-server binddn + +ldap-server bindpw + +ldap-server pam-filter + +ldap-server pam-login-attribute + -Default value is "uid". + +ldap-server pam-group-dn + +ldap-server pam-member-attribute + +ldap-server sudoers-base + +ldap-server nss-base-passwd + +ldap-server nss-base-group + +ldap-server nss-base-shadow + +ldap-server nss-base-netgroup + +ldap-server nss-base-sudoers + +ldap-server nss-initgroups-ignoreusers + +ldap-server source-ip + +ldap-server vrf + +==================================================================== + +LDAP server NSS commands: + +ldap-server nss timelimit <0 - 65535> +- The default value is 0 seconds. + +ldap-server nss bind-timelimit <0 - 65535> +- The default value is 10 seconds. + +ldap-server nss idle-timelimit <0 - 65535> +- The default value is 0 seconds. + +ldap-server nss retry <0 - 10> +- Default value is 0. + +ldap-server nss port <0 - 65535> +- Default value is 389. + +ldap-server nss scope [sub|one|base] +- Default value is "sub". + +ldap-server nss ldap-version [2|3] +- Default value is 3. + +ldap-server nss base + +ldap-server nss ssl [on|off|start_tls] +-Default value is "off" + +ldap-server nss binddn + +ldap-server nss bindpw + +ldap-server nss nss-base-passwd + +ldap-server nss nss-base-group + +ldap-server nss nss-base-shadow + +ldap-server nss nss-base-netgroup + +ldap-server nss nss-base-sudoers + +ldap-server nss nss-initgroups-ignoreusers + +======================================================================= + +LDAP server PAM commands: + +ldap-server pam timelimit <0 - 65535> +- The default value is 0 seconds. + +ldap-server pam bind-timelimit <0 - 65535> +- The default value is 10 seconds. + +ldap-server pam retry <0 - 10> +- Default value is 0. + +ldap-server pam port <0 - 65535> +- Default value is 389. + +ldap-server pam scope [sub|one|base] +- Default value is "sub". + +ldap-server pam ldap-version [2|3] +- Default value is 3. + +ldap-server pam base + +ldap-server pam ssl [on|off|start_tls] +-Default value is "off" + +ldap-server pam binddn + +ldap-server pam bindpw + +ldap-server pam pam-filter + +ldap-server pam pam-login-attribute + -Default value is "uid". + +ldap-server pam pam-group-dn + +ldap-server pam pam-member-attribute + +ldap-server pam nss-base-passwd + + +====================================================================== + +LDAP server sudo commands: + +ldap-server sudo timelimit <0 - 65535> +- The default value is 0 seconds. + +ldap-server sudo bind-timelimit <0 - 65535> +- The default value is 10 seconds. + +ldap-server sudo retry <0 - 10> +- Default value is 0. + +ldap-server sudo port <0 - 65535> +- Default value is 389. + +ldap-server sudo ldap-version [2|3] +- Default value is 3. + +ldap-server sudo base + +ldap-server sudo ssl [on|off|start_tls] +-Default value is "off" + +ldap-server sudo binddn + +ldap-server sudo bindpw + +ldap-server sudo sudoers-base + + +=============================================================================== + +LDAP server configuration commands: + +ldap-server host use-type [all|nss|sudo|pam] | port <1-665535> | priority <1 - 99> ssl [on|off|start_tls] + +- use-type [all|nss|sudo|pam] - Default value is "all" +- priority <1 - 99> - Default value is 1 +- ssl [on|off|start_tls] - Default value is "off" +- retry <1 - 10> - Default value is 0 + +=============================================================================== + +LDAP server MAP commands: + +ldap-server map attribute to + +Eg: ldap-server map attribute uid to "sAMAccountName" + ldap-server map attribute shadowLastChange to "pwdLastSet" + +ldap-server map objectclass to +ldap-server map default-attribute-value to +ldap-server map override-attribute-value to +``` + + +- Show commands. + +``` +sonic# show ldap-server +--------------------------------------------------------- +LDAP Global Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +-------------------------------------------------------------------------------- +HOST USE-TYPE PRIORITY SSL RETRY +-------------------------------------------------------------------------------- +1.1.1.1 - 1 - - +sonic# + +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : True +login-method : ldap, local +sonic# + +``` + +## Click + +``` +$ show aaa +$ show ldap +``` + + +Similar to KLISH, but config, and show utilities. + +Note: + +1. AAA has a holdoff timer of 3 seconds to decrease the frequency of time intensive operation + of updating AAA configuration files. Hence, AAA configurations will be updated in the respective + configuration files after a minimum of 3 seconds after UI configurations. + +## Known Issues and Notes + +### Known Issues + +- The publicly available pre-built debian packages for libpam-ldap, libnss-ldap, +and sudo-ldap, do not have source-ip or Vrf support. The same applies for the openldap-client library. + +- authorization-sudoers exposes Linux/UNIX functionality, thus making it + apparent that SONiC is Linux/UNIX based. + +- The existing aaa authentication login-method KLISH cli, needs to be + future proofed for allowing for separate console and ssh login methods. + +### Implementation Notes + +#### Authentication Login: local,ldap ; Failthrough: False + +The LDAP user information is present in the *local* database (because with LDAP +authentication, *ldap* will be used as a *nss* source). Thus, *local* +authentication will be attempted for LDAP users, which will likely fail. +The recommendation is to enable *failthrough*. + +#### Authentication Login: ldap,local ; Failthrough: False + +For a local user that is not present in the LDAP, the authentication against +the *ldap* pam module will return a PAM_USER_UNKNOWN error, which is not +considered an authentication failure, thus authentication will take place +with the *local* (unix) pam module. + +#### Per LDAP Server Configuration ; Failthrough: True + +The OSS *pam_ldap* module can have only a single instance. Configuration +cannot be supported on a per individual server basis. An authentication failure +at a single LDAP server results in failure being returned for the entire +PAM stack. Please see [Linux-PAM](#Linux-PAM), under *config=* -- +"Configuring multiple instances of pam_ldap for the same service with +different configuration files is not supported, because the configuration +information is cached." + +### Implementation In a Future Release + +Some part of the design mentioned in the document is for Implementation In a +Future Release and have been marked as such with ? (IIFR) annotations. + +# Use Case + +## LDAP Authentication and LDAP Name Service + +LDAP provides the name services user, group information. +User's session is setup to create home directory locally, if it does not exist. +Users are authenticated through LDAP servers. + + +``` + + +------------+ +-----------+ +-----------------------+ + | | | | | | + | SSH Client |<-->| SONiC DUT |<----->| LDAP Name Service & | + | | | | | Authentication Server | + | | | | | | + +------------+ +-----------+ +-----------------------+ + +``` + +``` + +sonic(config)# ldap-server base "dc=example,dc=com" +sonic(config)# ldap-server ssl start_tls +sonic(config)# ldap-server 10.59.1.9 + +; Use defaults for locating the bases of passwd, shadow, and group entries. + +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group ldap local + +; Use defaults for name-service based on login authentication + +``` + + +## TACACS+ Authentication, LDAP Authorization, and LDAP Name Service + +LDAP provides the name service user, group information. +User's session is setup to create home directory locally, if it does not exist. +Users are authenticated through TACACS+ servers. +Users are authorized through LDAP servers for logon authorization only. +User's click interface is sudoers authorized through LDAP servers. (i.e + the rules to check whether the user is allowed to execute the sudo + command (config, show, sonic-clear) are obtained from LDAP servers. + This does not apply to KLISH interface, since the authorization for + that comes from the role assigned to the user). + +``` + + +----------------------------+ + | | + | LDAP Name Service, and | + +-->| LDAP sudoers Authorization | + | | Servers | + | | | + +------------+ +-----------+ | +----------------------------+ + | | | | | + | | | | | +----------------------------+ + | | | | | | | + | SSH Client |<-->| SONiC DUT |<--+-->| TACACS+ Authentication | + | | | | | | Servers | + | | | | | +----------------------------+ + | | | | | + | | | | | + +------------+ +-----------+ | +----------------------------+ + | | | + +-->| LDAP Login Authorization | + | Servers | + +----------------------------+ + +``` + +``` + + +; LDAP Name Service parameters +sonic(config)# ldap-server base "dc=example,dc=com" +sonic(config)# aaa name-service passwd group ldap +sonic(config)# aaa name-service shadow group ldap +sonic(config)# aaa name-service group group ldap + +; Use this server for name-service and sudo-ldap authorization only, with one +; reconnect attempt +sonic(config)# ldap-server rsa.loc.example.net use-type nss,sudo ldap-retry 1 + + +; Logon Authorization from LDAP +; Use this server for login authorization only +sonic(config)# ldap-server 10.59.1.9 use-type pam + +sonic(config)# ldap-server pam-groupdn "cn=t1.loc.example.com,ou=hostaccess,dc=example,dc=com" +sonic(config)# ldap-server pam-member-attribute "uniqueMember" +sonic(config)# aaa authorization login default group ldap + +; sudo-ldap authorization +sonic(config)# ldap-server sudoers-base "dc=example,dc=com" +sonic(config)# aaa name-service sudoers group ldap + +; Authentication from TACACS+ +sonic(config)# tacacs-server 10.59.1.15 +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group tacacs+ local + +``` + +# Flow Diagrams + +Please see the diagrams in section [Design](#design) + +# Error Handling + +PAM and NSS modules return errors as per [Linux-PAM](#Linux-PAM), and +[NSS](#NSS-Reference) respectively. + +# Serviceability and Debug + +The LDAP PAM and NSS modules can be debugged using the "debug" option +of AAA. (i.e. config aaa debug enable) + +# Warm Boot Support + +N/A + +# Scalability + +A maximum of 8 LDAP servers can be configured. + +# Unit Test + +## Prerequisites + +``` + LDAP Server with User, Group information. + TACACS+/RADIUS Server with User passwords configure. +``` + +### LDAP data + +For sudo authorization (*sudoRole* objectclass), the LDAP server schema +needs to be updated from DUT file /usr/share/doc/sudo-ldap/schema.OpenLDAP. +Please see next section for some tips. + + +``` + +$ ldapsearch -x -b "dc=sji,dc=broadcom,dc=net" +# extended LDIF +# +# LDAPv3 +# base with scope subtree +# filter: (objectclass=*) +# requesting: ALL +# + +# sji.broadcom.net +dn: dc=sji,dc=broadcom,dc=net +objectClass: top +objectClass: dcObject +objectClass: organization +o: sji.broadcom.net +dc: sji + +# admin, sji.broadcom.net +dn: cn=admin,dc=sji,dc=broadcom,dc=net +objectClass: simpleSecurityObject +objectClass: organizationalRole +cn: admin +description: LDAP administrator + +# Group, sji.broadcom.net +dn: ou=Group,dc=sji,dc=broadcom,dc=net +objectClass: organizationalUnit +ou: Group + +# People, sji.broadcom.net +dn: ou=People,dc=sji,dc=broadcom,dc=net +objectClass: organizationalUnit +ou: People + +# Sudoers, sji.broadcom.net +dn: ou=Sudoers,dc=sji,dc=broadcom,dc=net +objectClass: organizationalUnit +ou: Sudoers + +# sudo, Group, sji.broadcom.net +dn: cn=sudo,ou=Group,dc=sji,dc=broadcom,dc=net +objectClass: top +objectClass: posixGroup +gidNumber: 27 +memberUid: arun +memberUid: ldapuser +cn: sudo + +# docker, Group, sji.broadcom.net +dn: cn=docker,ou=Group,dc=sji,dc=broadcom,dc=net +objectClass: top +objectClass: posixGroup +gidNumber: 999 +memberUid: arun +memberUid: ldapuser +memberUid: raduser1 +cn: docker + +# arun, People, sji.broadcom.net +dn: cn=arun,ou=People,dc=sji,dc=broadcom,dc=net +objectClass: top +objectClass: account +objectClass: posixAccount +objectClass: shadowAccount +cn: arun +uid: arun +uidNumber: 1200 +gidNumber: 1000 +homeDirectory: /home/arun +loginShell: /bin/bash + +# ldapuser, People, sji.broadcom.net +dn: cn=ldapuser,ou=People,dc=sji,dc=broadcom,dc=net +objectClass: top +objectClass: account +objectClass: posixAccount +objectClass: shadowAccount +cn: ldapuser +uid: ldapuser +uidNumber: 1201 +gidNumber: 1000 +homeDirectory: /home/ldapuser +loginShell: /bin/bash + +# raduser1, People, sji.broadcom.net +dn: cn=raduser1,ou=People,dc=sji,dc=broadcom,dc=net +objectClass: top +objectClass: account +objectClass: posixAccount +objectClass: shadowAccount +cn: raduser1 +uid: raduser1 +uidNumber: 1202 +gidNumber: 100 +homeDirectory: /home/raduser1 +loginShell: /bin/bash + +# role1, Sudoers, sji.broadcom.net +dn: cn=role1,ou=Sudoers,dc=sji,dc=broadcom,dc=net +objectClass: sudoRole +objectClass: top +cn: role1 +sudoUser: raduser1 +sudoHost: ALL +sudoCommand: ALL +sudoRunAsUser: ALL +sudoRunAsGroup: ALL + +# search result +search: 2 +result: 0 Success + +# numResponses: 12 +# numEntries: 11 +$ + +``` + +### LDAP sudo-ldap.schema addition + +For 2.4.42+dfsg-2ubuntu3.7 OpenLDAP server (slapd), procedure like the +below was used. Your mileage may vary. + +``` + +$ scp admin@10.59.143.65:/usr/share/doc/sudo-ldap/schema.OpenLDAP /etc/ldap/schema/sudo-ldap.schema +admin@10.59.143.65's password: +schema.OpenLDAP 100% 2410 2.4KB/s 00:00 +$ + +$ touch /tmp/ldif/conf +$ echo 'include /etc/ldap/schema/sudo-ldap.schema' > /tmp/ldif/conf +$ slaptest -f /tmp/ldif/conf -F /tmp/ldif +config file testing succeeded +$ + +$ vi /tmp/ldif/cn\=config/cn\=schema/cn\=\{0}sudo-ldap.ldif + +[ Remove the header and trailer lines so now it looks something like this ] + +$ cat /tmp/ldif/cn\=config/cn\=schema/cn\=\{0}sudo-ldap.ldif +dn: cn=sudo-ldap,cn=schema,cn=config +objectClass: olcSchemaConfig +cn: sudo-ldap +olcAttributeTypes: {0}( 1.3.6.1.4.1.15953.9.1.1 NAME 'sudoUser' DESC 'User(s + ) who may run sudo' EQUALITY caseExactIA5Match SUBSTR caseExactIA5Substrin + gsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 ) +olcAttributeTypes: {1}( 1.3.6.1.4.1.15953.9.1.2 NAME 'sudoHost' DESC 'Host(s + ) who may run sudo' EQUALITY caseExactIA5Match SUBSTR caseExactIA5Substring + sMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 ) +olcAttributeTypes: {2}( 1.3.6.1.4.1.15953.9.1.3 NAME 'sudoCommand' DESC 'Com + mand(s) to be executed by sudo' EQUALITY caseExactIA5Match SYNTAX 1.3.6.1.4 + .1.1466.115.121.1.26 ) +olcAttributeTypes: {3}( 1.3.6.1.4.1.15953.9.1.4 NAME 'sudoRunAs' DESC 'User( + s) impersonated by sudo (deprecated)' EQUALITY caseExactIA5Match SYNTAX 1.3 + .6.1.4.1.1466.115.121.1.26 ) +olcAttributeTypes: {4}( 1.3.6.1.4.1.15953.9.1.5 NAME 'sudoOption' DESC 'Opti + ons(s) followed by sudo' EQUALITY caseExactIA5Match SYNTAX 1.3.6.1.4.1.1466 + .115.121.1.26 ) +olcAttributeTypes: {5}( 1.3.6.1.4.1.15953.9.1.6 NAME 'sudoRunAsUser' DESC 'U + ser(s) impersonated by sudo' EQUALITY caseExactIA5Match SYNTAX 1.3.6.1.4.1. + 1466.115.121.1.26 ) +olcAttributeTypes: {6}( 1.3.6.1.4.1.15953.9.1.7 NAME 'sudoRunAsGroup' DESC ' + Group(s) impersonated by sudo' EQUALITY caseExactIA5Match SYNTAX 1.3.6.1.4. + 1.1466.115.121.1.26 ) +olcAttributeTypes: {7}( 1.3.6.1.4.1.15953.9.1.8 NAME 'sudoNotBefore' DESC 'S + tart of time interval for which the entry is valid' EQUALITY generalizedTim + eMatch ORDERING generalizedTimeOrderingMatch SYNTAX 1.3.6.1.4.1.1466.115.12 + 1.1.24 ) +olcAttributeTypes: {8}( 1.3.6.1.4.1.15953.9.1.9 NAME 'sudoNotAfter' DESC 'En + d of time interval for which the entry is valid' EQUALITY generalizedTimeMa + tch ORDERING generalizedTimeOrderingMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1 + .24 ) +olcAttributeTypes: {9}( 1.3.6.1.4.1.15953.9.1.10 NAME 'sudoOrder' DESC 'an i + nteger to order the sudoRole entries' EQUALITY integerMatch ORDERING intege + rOrderingMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 ) +olcObjectClasses: {0}( 1.3.6.1.4.1.15953.9.2.1 NAME 'sudoRole' DESC 'Sudoer + Entries' SUP top STRUCTURAL MUST cn MAY ( sudoUser $ sudoHost $ sudoCommand + $ sudoRunAs $ sudoRunAsUser $ sudoRunAsGroup $ sudoOption $ sudoOrder $ su + doNotBefore $ sudoNotAfter $ description ) ) +$ + + +$ cp /tmp/ldif/cn\=config/cn\=schema/cn\=\{0}sudo-ldap.ldif /etc/ldap/schema/sudo-ldap.ldif + +$ ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/sudo-ldap.ldifSASL/EXTERNAL authentication started +SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth +SASL SSF: 0 +adding new entry "cn=sudo-ldap,cn=schema,cn=config" + +$ + +``` + +### TACACS+ Server user + +``` +... +user = ldapuser { + default service = permit + chap = cleartext "ldapuser" + pap = cleartext "ldapuser" + service = exec { + priv-lvl=15 + } +} +... + +``` + +### RADIUS Server user + +``` +... +raduser1 Cleartext-Password := "password" + Management-Privilege-Level = 1 +... +ldapuser Cleartext-Password := ldapuser + Brocade-Auth-Role = admin, + Management-Privilege-Level = 15 +... + +``` + +## UT Authentication LDAP + +### Config DB: Before + +``` + +127.0.0.1:6379[4]> keys *LDAP* +(empty list or set) +127.0.0.1:6379[4]> keys *AAA* +(empty list or set) +127.0.0.1:6379[4]> + +``` + +### Click: show aaa, show ldap: Before + +``` + +admin@L10:~/show$ show aaa +AAA authentication login local (default) +AAA authentication failthrough False (default) +AAA nss passwd login (default) +AAA nss shadow login (default) +AAA nss group login (default) +AAA nss netgroup local (default) +AAA authorization login local (default) + +admin@L10:~/show$ + +admin@L10:~/show$ show ldap +LDAP global timelimit 0 (default) +LDAP global retry 0 (default) +LDAP global ldap_version 3 (default) +LDAP global scope sub (default) +LDAP global idle_timelimit 0 (default) +LDAP global ssl off (default) +LDAP global use_type all (default) +LDAP global port 389 (default) +LDAP global bind_timelimit 10 (default) + +admin@L10:~/show$ + +``` + +### Click CLI + +``` + +sudo config ldap add 10.59.143.229 +sudo config ldap base dc=sji,dc=broadcom,dc=net +sudo config aaa authentication login ldap local +sudo config aaa authentication failthrough enable + +``` + +### KLISH: show aaa, show ldap-server: Before + +``` +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : False +login-method : local +sonic# +sonic# show ldap-server +sonic# + +``` + +### KLISH CLI + +``` + +sonic(config)# ldap-server host 10.59.143.229 +sonic(config)# ldap-server base dc=sji,dc=broadcom,dc=net +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group ldap local + +``` + +### Config DB: After + +``` + +127.0.0.1:6379[4]> keys *LDAP* +1) "LDAP|global" +2) "LDAP_SERVER|10.59.143.229" +127.0.0.1:6379[4]> keys *AAA* +1) "AAA|authentication" +127.0.0.1:6379[4]> hgetall LDAP|global +1) "base" +2) "dc=sji,dc=broadcom,dc=net" +127.0.0.1:6379[4]> hgetall LDAP_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "retry" +4) "0" +5) "use_type" +6) "all" +127.0.0.1:6379[4]> hgetall AAA|authentication +1) "login" +2) "ldap,local" +3) "failthrough" +4) "True" +127.0.0.1:6379[4]> + +``` + +### Click: show aaa, show ldap: After + +``` + +admin@L10:~/show$ id ldapuser +uid=1201(ldapuser) gid=1000(admin) groups=1000(admin),27(sudo),999(docker) +admin@L10:~/show$ show ldap +LDAP global timelimit 0 (default) +LDAP global retry 0 (default) +LDAP global ldap_version 3 (default) +LDAP global scope sub (default) +LDAP global idle_timelimit 0 (default) +LDAP global ssl off (default) +LDAP global base dc=sji,dc=broadcom,dc=net +LDAP global use_type all (default) +LDAP global port 389 (default) +LDAP global bind_timelimit 10 (default) + +LDAP_SERVER address 10.59.143.229 + priority 1 + retry 0 + use_type all + +admin@L10:~/show$ show aaa +AAA authentication login ldap,local +AAA authentication failthrough True +AAA nss passwd login (default) +AAA nss shadow login (default) +AAA nss group login (default) +AAA nss netgroup local (default) +AAA authorization login local (default) + +admin@L10:~/show$ + +``` + +### KLISH: show aaa, show ldap-server: After + +``` +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : True +login-method : ldap, local +sonic# +sonic# show ldap-server +--------------------------------------------------------- +LDAP Global Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +-------------------------------------------------------------------------------- +HOST USE-TYPE PRIORITY SSL RETRY +-------------------------------------------------------------------------------- +10.59.143.229 - 1 - - +sonic# + +``` + +### Verify ldapuser can login with LDAP password. + +``` + +$ ssh ldapuser@10.59.143.109 +ldapuser@10.59.143.109's password: +Creating directory '/home/ldapuser'. +Linux L10 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 +... +ldapuser@L10:~$ + +``` + +## UT Authentication TACPLUS with Name Service LDAP + +### Click CLI + +``` + +sudo config tacacs add 10.59.143.229 +sudo config tacacs passkey testing123 +sudo config aaa authentication failthrough enable +sudo config aaa authentication login tacacs+ local + +sudo config ldap add 10.59.143.229 +sudo config ldap base dc=sji,dc=broadcom,dc=net +sudo config aaa nss passwd ldap +sudo config aaa nss shadow ldap +sudo config aaa nss group ldap + +``` + +### KLISH CLI + +``` + +sonic(config)# tacacs-server host 10.59.143.229 +sonic(config)# tacacs-server key testing123 +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group tacacs+ local + +sonic(config)# ldap-server host 10.59.143.229 +sonic(config)# ldap-server base dc=sji,dc=broadcom,dc=net +sonic(config)# aaa name-service passwd group ldap +sonic(config)# aaa name-service shadow group ldap +sonic(config)# aaa name-service group group ldap + +``` + +### Config DB: After + +``` + +127.0.0.1:6379[4]> keys *AAA* +1) "AAA|nss" +2) "AAA|authentication" +127.0.0.1:6379[4]> keys *TACPLUS* +1) "TACPLUS_SERVER|10.59.143.229" +2) "TACPLUS|global" +127.0.0.1:6379[4]> keys *LDAP* +1) "LDAP|global" +2) "LDAP_SERVER|10.59.143.229" +127.0.0.1:6379[4]> hgetall AAA|nss +1) "passwd" +2) "ldap" +3) "shadow" +4) "ldap" +5) "group" +6) "ldap" +127.0.0.1:6379[4]> hgetall AAA|authentication +1) "failthrough" +2) "True" +3) "login" +4) "tacacs+,local" +127.0.0.1:6379[4]> hgetall TACPLUS|global +1) "passkey" +2) "testing123" +127.0.0.1:6379[4]> hgetall TACPLUS_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "tcp_port" +4) "49" +127.0.0.1:6379[4]> hgetall LDAP|global +1) "base" +2) "dc=sji,dc=broadcom,dc=net" +127.0.0.1:6379[4]> hgetall LDAP_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "retry" +4) "0" +5) "use_type" +6) "all" +127.0.0.1:6379[4]> + +``` + +### Click: show aaa, show tacacs, show ldap: After + +``` + +admin@L10:~$ show aaa +AAA authentication login tacacs+,local +AAA authentication failthrough True +AAA nss group ldap +AAA nss passwd ldap +AAA nss shadow ldap +AAA nss netgroup local (default) +AAA authorization login local (default) + +admin@L10:~$ show tacacs +TACPLUS global auth_type pap (default) +TACPLUS global timeout 5 (default) +TACPLUS global passkey testing123 + +TACPLUS_SERVER address 10.59.143.229 + priority 1 + tcp_port 49 + +admin@L10:~$ show ldap +LDAP global timelimit 0 (default) +LDAP global retry 0 (default) +LDAP global ldap_version 3 (default) +LDAP global scope sub (default) +LDAP global idle_timelimit 0 (default) +LDAP global ssl off (default) +LDAP global base dc=sji,dc=broadcom,dc=net +LDAP global use_type all (default) +LDAP global port 389 (default) +LDAP global bind_timelimit 10 (default) + +LDAP_SERVER address 10.59.143.229 + priority 1 + retry 0 + use_type all + +admin@L10:~$ + +``` + +### KLISH: show aaa, show tacacs-server, show ldap-server: After + +``` +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : True +login-method : tacacs+, local +--------------------------------------------------------- +AAA Name-Service Information +--------------------------------------------------------- +passwd : ldap +shadow : ldap +group : ldap +sonic# +sonic# show tacacs-server global +--------------------------------------------------------- +TACACS Global Configuration +--------------------------------------------------------- +key : testing123 +sonic# +sonic# show tacacs-server host +-------------------------------------------------------------------------------- +HOST AUTH-TYPE KEY PORT PRIORITY TIMEOUT +-------------------------------------------------------------------------------- +10.59.143.229 pap 49 1 5 +sonic# +sonic# show ldap-server +--------------------------------------------------------- +LDAP Global Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +-------------------------------------------------------------------------------- +HOST USE-TYPE PRIORITY SSL RETRY +-------------------------------------------------------------------------------- +10.59.143.229 - 1 - - +sonic# + +``` + +### Verify user can login using TACACS+ password with LDAP Name Service + +``` + +$ ssh ldapuser@10.59.143.109 +ldapuser@10.59.143.109's password: +Creating directory '/home/ldapuser'. +Linux L10 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 +... +ldapuser@L10:~$ + +``` + +## UT Authentication RADIUS with Name Service LDAP + +### Click CLI + +``` + +sudo config radius add 10.59.143.229 +sudo config radius passkey sharedsecret +sudo config aaa authentication failthrough enable +sudo config aaa authentication login radius local + +sudo config ldap add 10.59.143.229 +sudo config ldap base dc=sji,dc=broadcom,dc=net +sudo config aaa nss passwd ldap +sudo config aaa nss shadow ldap +sudo config aaa nss group ldap + +``` + +### KLISH CLI + +``` + +sonic(config)# radius-server host 10.59.143.229 +sonic(config)# radius-server key sharedsecret +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group radius local + +sonic(config)# ldap-server host 10.59.143.229 +sonic(config)# ldap-server base dc=sji,dc=broadcom,dc=net +sonic(config)# aaa name-service passwd group ldap +sonic(config)# aaa name-service shadow group ldap +sonic(config)# aaa name-service group group ldap + +``` + +### Config DB: After + +``` + +127.0.0.1:6379[4]> keys *AAA* +1) "AAA|nss" +2) "AAA|authentication" +127.0.0.1:6379[4]> keys *RADIUS* +1) "RADIUS_SERVER|10.59.143.229" +2) "RADIUS|global" +127.0.0.1:6379[4]> keys *LDAP* +1) "LDAP|global" +2) "LDAP_SERVER|10.59.143.229" +127.0.0.1:6379[4]> hgetall AAA|nss +1) "passwd" +2) "ldap" +3) "shadow" +4) "ldap" +5) "group" +6) "ldap" +127.0.0.1:6379[4]> hgetall AAA|authentication +1) "failthrough" +2) "True" +3) "login" +4) "radius,local" +127.0.0.1:6379[4]> hgetall RADIUS|global +1) "passkey" +2) "sharedsecret" +127.0.0.1:6379[4]> hgetall RADIUS_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "auth_port" +4) "1812" +127.0.0.1:6379[4]> hgetall LDAP|global +1) "base" +2) "dc=sji,dc=broadcom,dc=net" +127.0.0.1:6379[4]> hgetall LDAP_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "retry" +4) "0" +5) "use_type" +6) "all" +127.0.0.1:6379[4]> + +``` + +### show aaa, show radius, show ldap: After + +``` + +admin@L10:~$ show aaa +AAA authentication login radius,local +AAA authentication failthrough True +AAA nss group ldap +AAA nss passwd ldap +AAA nss shadow ldap +AAA nss netgroup local (default) +AAA authorization login local (default) + +admin@L10:~$ show radius +RADIUS global auth_type pap (default) +RADIUS global retransmit 3 (default) +RADIUS global timeout 5 (default) +RADIUS global passkey sharedsecret + +RADIUS_SERVER address 10.59.143.229 + priority 1 + auth_port 1812 + +admin@L10:~$ + +admin@L10:~$ show ldap +LDAP global timelimit 0 (default) +LDAP global retry 0 (default) +LDAP global ldap_version 3 (default) +LDAP global scope sub (default) +LDAP global idle_timelimit 0 (default) +LDAP global ssl off (default) +LDAP global base dc=sji,dc=broadcom,dc=net +LDAP global use_type all (default) +LDAP global port 389 (default) +LDAP global bind_timelimit 10 (default) + +LDAP_SERVER address 10.59.143.229 + priority 1 + retry 0 + use_type all + +admin@L10:~$ + +``` + +### KLISH: show aaa, show radius-server, show ldap-server: After + +``` +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : True +login-method : radius, local +--------------------------------------------------------- +AAA Name-Service Information +--------------------------------------------------------- +passwd : ldap +shadow : ldap +group : ldap +sonic# +sonic# show radius-server +--------------------------------------------------------- +RADIUS Global Configuration +--------------------------------------------------------- +key : sharedsecret +-------------------------------------------------------------------------------- +HOST AUTH-TYPE KEY AUTH-PORT PRIORITY TIMEOUT RTSMT VRF +-------------------------------------------------------------------------------- +10.59.143.229 - - 1812 1 - - - +sonic# +sonic# show ldap-server +--------------------------------------------------------- +LDAP Global Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +-------------------------------------------------------------------------------- +HOST USE-TYPE PRIORITY SSL RETRY +-------------------------------------------------------------------------------- +10.59.143.229 - 1 - - +sonic# + +``` + +### Verify user can login using RADIUS password with LDAP Name Service + +``` + +$ ssh ldapuser@10.59.143.109 +ldapuser@10.59.143.109's password: +Creating directory '/home/ldapuser'. +Linux L10 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 +... +ldapuser@L10:~$ + +``` + +## UT Authentication RADIUS with Logon Authorization LDAP and Name Service LDAP + +### Click CLI + +``` + +sudo config radius add 10.59.143.229 +sudo config radius passkey sharedsecret +sudo config aaa authentication failthrough enable +sudo config aaa authentication login radius local + +sudo config ldap add 10.59.143.229 +sudo config ldap base dc=sji,dc=broadcom,dc=net +sudo config aaa nss passwd ldap +sudo config aaa nss shadow ldap +sudo config aaa nss group ldap + +sudo config ldap pam_groupdn cn=sudo,ou=Group,dc=sji,dc=broadcom,dc=net +sudo config ldap pam_member_attribute memberUid +sudo config aaa authorization login ldap + +``` + +### KLISH CLI + +``` + +sonic(config)# radius-server host 10.59.143.229 +sonic(config)# radius-server key sharedsecret +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group radius local + +sonic(config)# ldap-server host 10.59.143.229 +sonic(config)# ldap-server base dc=sji,dc=broadcom,dc=net +sonic(config)# aaa name-service passwd group ldap +sonic(config)# aaa name-service shadow group ldap +sonic(config)# aaa name-service group group ldap + +sonic(config)# ldap-server pam-groupdn cn=sudo,ou=Group,dc=sji,dc=broadcom,dc=net +sonic(config)# ldap-server pam-member-attribute memberUid +sonic(config)# aaa authorization login default group ldap + +``` + +### Config DB: After + +``` + +127.0.0.1:6379[4]> keys *AAA* +1) "AAA|authorization" +2) "AAA|nss" +3) "AAA|authentication" +127.0.0.1:6379[4]> keys *RADIUS* +1) "RADIUS_SERVER|10.59.143.229" +2) "RADIUS|global" +127.0.0.1:6379[4]> keys *LDAP* +1) "LDAP|global" +2) "LDAP_SERVER|10.59.143.229" +127.0.0.1:6379[4]> hgetall AAA|nss +1) "passwd" +2) "ldap" +3) "shadow" +4) "ldap" +5) "group" +6) "ldap" +127.0.0.1:6379[4]> hgetall AAA|authentication +1) "failthrough" +2) "True" +3) "login" +4) "radius,local" +127.0.0.1:6379[4]> hgetall AAA|authorization +1) "login" +2) "ldap" +127.0.0.1:6379[4]> hgetall RADIUS|global +1) "passkey" +2) "sharedsecret" +127.0.0.1:6379[4]> hgetall RADIUS_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "auth_port" +4) "1812" +127.0.0.1:6379[4]> hgetall LDAP|global +1) "base" +2) "dc=sji,dc=broadcom,dc=net" +3) "pam_groupdn" +4) "cn=sudo,ou=Group,dc=sji,dc=broadcom,dc=net" +5) "pam_member_attribute" +6) "memberUid" +127.0.0.1:6379[4]> + +``` + +### show aaa, show radius, show ldap: After + +``` + +admin@L10:~$ show aaa +AAA authentication login radius,local +AAA authentication failthrough True +AAA nss group ldap +AAA nss passwd ldap +AAA nss shadow ldap +AAA nss netgroup local (default) +AAA authorization login ldap + +admin@L10:~$ + +admin@L10:~$ show radius +RADIUS global auth_type pap (default) +RADIUS global retransmit 3 (default) +RADIUS global timeout 5 (default) +RADIUS global passkey sharedsecret + +RADIUS_SERVER address 10.59.143.229 + priority 1 + auth_port 1812 + +admin@L10:~$ + +admin@L10:~$ show ldap +LDAP global retry 0 (default) +LDAP global ssl off (default) +LDAP global idle_timelimit 0 (default) +LDAP global bind_timelimit 10 (default) +LDAP global timelimit 0 (default) +LDAP global pam_member_attribute memberUid +LDAP global pam_groupdn cn=sudo,ou=Group,dc=sji,dc=broadcom,dc=net +LDAP global ldap_version 3 (default) +LDAP global scope sub (default) +LDAP global base dc=sji,dc=broadcom,dc=net +LDAP global use_type all (default) +LDAP global port 389 (default) + +LDAP_SERVER address 10.59.143.229 + priority 1 + retry 0 + use_type all + +admin@L10:~$ + +``` + +### KLISH: show aaa, show radius-server, show ldap-server: After + +``` +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : True +login-method : radius, local +--------------------------------------------------------- +AAA Authorization Information +--------------------------------------------------------- +login : ldap +--------------------------------------------------------- +AAA Name-Service Information +--------------------------------------------------------- +passwd : ldap +shadow : ldap +group : ldap +sonic# +sonic# show radius-server +--------------------------------------------------------- +RADIUS Global Configuration +--------------------------------------------------------- +key : sharedsecret +-------------------------------------------------------------------------------- +HOST AUTH-TYPE KEY AUTH-PORT PRIORITY TIMEOUT RTSMT VRF +-------------------------------------------------------------------------------- +10.59.143.229 - - 1812 1 - - - +sonic# +sonic# show ldap-server +--------------------------------------------------------- +LDAP Global Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +pam_member_attribute : memberUid +pam_groupdn : cn=sudo,ou=Group,dc=sji,dc=broadcom,dc=net +-------------------------------------------------------------------------------- +HOST USE-TYPE PRIORITY SSL RETRY +-------------------------------------------------------------------------------- +10.59.143.229 - 1 - - +sonic# + +``` + +### Verify user can login using RADIUS password with LDAP Name Service + +``` + +$ ssh ldapuser@10.59.143.109 +ldapuser@10.59.143.109's password: +Creating directory '/home/ldapuser'. +Linux L10 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 +... +ldapuser@L10:~$ + +``` + + +## UT Authentication RADIUS with Name Service LDAP and sudo Authorization LDAP + +### Click CLI + +``` + +sudo config radius add 10.59.143.229 +sudo config radius passkey sharedsecret +sudo config aaa authentication failthrough enable +sudo config aaa authentication login radius local + +sudo config ldap add 10.59.143.229 +sudo config ldap base dc=sji,dc=broadcom,dc=net +sudo config aaa nss passwd ldap +sudo config aaa nss shadow ldap +sudo config aaa nss group ldap + +sudo config aaa nss sudoers ldap +sudo config ldap sudoers_base ou=Sudoers,dc=sji,dc=broadcom,dc=net + + +``` + +### KLISH CLI + +``` + +sonic(config)# radius-server host 10.59.143.229 +sonic(config)# radius-server key sharedsecret +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group radius local + +sonic(config)# ldap-server host 10.59.143.229 +sonic(config)# ldap-server base dc=sji,dc=broadcom,dc=net +sonic(config)# aaa name-service passwd group ldap +sonic(config)# aaa name-service shadow group ldap +sonic(config)# aaa name-service group group ldap + +sonic(config)# aaa name-service sudoers group ldap +sonic(config)# ldap-server sudoers-base ou=Sudoers,dc=sji,dc=broadcom,dc=net + +``` + +### Config DB: After + +``` + +127.0.0.1:6379[4]> keys *AAA* +1) "AAA|nss" +2) "AAA|authentication" +127.0.0.1:6379[4]> keys *RADIUS* +1) "RADIUS_SERVER|10.59.143.229" +2) "RADIUS|global" +127.0.0.1:6379[4]> keys *LDAP* +1) "LDAP|global" +2) "LDAP_SERVER|10.59.143.229" +127.0.0.1:6379[4]> hgetall AAA|nss +1) "passwd" +2) "ldap" +3) "shadow" +4) "ldap" +5) "group" +6) "ldap" +7) "sudoers" +8) "ldap" +127.0.0.1:6379[4]> hgetall AAA|authentication +1) "failthrough" +2) "True" +3) "login" +4) "radius,local" +127.0.0.1:6379[4]> hgetall RADIUS|global +1) "passkey" +2) "sharedsecret" +127.0.0.1:6379[4]> hgetall RADIUS_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "auth_port" +4) "1812" +127.0.0.1:6379[4]> hgetall LDAP|global +1) "base" +2) "dc=sji,dc=broadcom,dc=net" +3) "sudoers_base" +4) "ou=Sudoers,dc=sji,dc=broadcom,dc=net" +127.0.0.1:6379[4]> hgetall LDAP_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "retry" +4) "0" +5) "use_type" +6) "all" +127.0.0.1:6379[4]> + +``` + +### show aaa, show radius, show ldap: After + +``` + +admin@L10:~$ show aaa +AAA authentication login radius,local +AAA authentication failthrough True +AAA nss group ldap +AAA nss passwd ldap +AAA nss sudoers ldap +AAA nss netgroup local (default) +AAA nss shadow ldap +AAA authorization login local (default) + +admin@L10:~$ + +admin@L10:~$ show radius +RADIUS global auth_type pap (default) +RADIUS global retransmit 3 (default) +RADIUS global timeout 5 (default) +RADIUS global passkey sharedsecret + +RADIUS_SERVER address 10.59.143.229 + priority 1 + auth_port 1812 + +admin@L10:~$ + +admin@L10:~$ show ldap +LDAP global retry 0 (default) +LDAP global ssl off (default) +LDAP global sudoers_base ou=Sudoers,dc=sji,dc=broadcom,dc=net +LDAP global idle_timelimit 0 (default) +LDAP global bind_timelimit 10 (default) +LDAP global timelimit 0 (default) +LDAP global ldap_version 3 (default) +LDAP global scope sub (default) +LDAP global base dc=sji,dc=broadcom,dc=net +LDAP global use_type all (default) +LDAP global port 389 (default) + +LDAP_SERVER address 10.59.143.229 + priority 1 + retry 0 + use_type all + +admin@L10:~$ + +``` + +### KLISH: show aaa, show radius-server, show ldap-server: After + +``` +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : True +login-method : radius, local +--------------------------------------------------------- +AAA Name-Service Information +--------------------------------------------------------- +passwd : ldap +shadow : ldap +group : ldap +sudoers : ldap +sonic# +sonic# show radius-server +--------------------------------------------------------- +RADIUS Global Configuration +--------------------------------------------------------- +key : sharedsecret +-------------------------------------------------------------------------------- +HOST AUTH-TYPE KEY AUTH-PORT PRIORITY TIMEOUT RTSMT VRF +-------------------------------------------------------------------------------- +10.59.143.229 - - 1812 1 - - - +sonic# +sonic# show ldap-server +--------------------------------------------------------- +LDAP Global Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +sudoers-base : ou=Sudoers,dc=sji,dc=broadcom,dc=net +-------------------------------------------------------------------------------- +HOST USE-TYPE PRIORITY SSL RETRY +-------------------------------------------------------------------------------- +10.59.143.229 - 1 - - +sonic# + +``` + +### Verify user can login using RADIUS password, LDAP Name Service, and sudo + +``` + +$ ssh raduser1@10.59.143.109 +raduser1@10.59.143.109's password: +Linux L10 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 +... +raduser1@L10:~$ id +uid=1202(raduser1) gid=100(users) groups=100(users),999(docker) +raduser1@L10:~$ sudo -i +[sudo] password for raduser1: +root@L10:~# id +uid=0(root) gid=0(root) groups=0(root) +root@L10:~# exit +logout +raduser1@L10:~$ + +``` + +## UT Authentication Non Global NSS PAM SUDO options + +### Click CLI + +``` + +sudo config radius add 10.59.143.229 +sudo config radius passkey sharedsecret +sudo config aaa authentication failthrough enable +sudo config aaa authentication login radius local + +sudo config ldap add 10.59.143.229 +sudo config ldap base dc=sji,dc=broadcom,dc=net + +sudo config ldap nss base dc=sji,dc=broadcom,dc=net +sudo config aaa nss passwd ldap +sudo config aaa nss shadow ldap +sudo config aaa nss group ldap + +sudo config aaa nss sudoers ldap +sudo config ldap sudo sudoers_base ou=Sudoers,dc=sji,dc=broadcom,dc=net + +sudo config ldap pam pam_groupdn cn=docker,ou=Group,dc=sji,dc=broadcom,dc=net +sudo config ldap pam pam_member_attribute memberUid +sudo config aaa authorization login ldap + +``` + +### KLISH CLI + +``` + +sonic(config)# radius-server host 10.59.143.229 +sonic(config)# radius-server key sharedsecret +sonic(config)# aaa authentication failthrough enable +sonic(config)# aaa authentication login default group radius local + +sonic(config)# ldap-server host 10.59.143.229 +sonic(config)# ldap-server base dc=sji,dc=broadcom,dc=net +sonic(config)# ldap-server nss base dc=sji,dc=broadcom,dc=net +sonic(config)# aaa name-service passwd group ldap +sonic(config)# aaa name-service shadow group ldap +sonic(config)# aaa name-service group group ldap + +sonic(config)# aaa name-service sudoers group ldap +sonic(config)# ldap-server sudo sudoers-base ou=Sudoers,dc=sji,dc=broadcom,dc=net + +sonic(config)# ldap-server pam pam-groupdn cn=docker,ou=Group,dc=sji,dc=broadcom,dc=net +sonic(config)# ldap-server pam pam-member-attribute memberUid +sonic(config)# aaa authorization login default group ldap + +``` + +### Config DB: After + +``` + +127.0.0.1:6379[4]> keys *AAA* +1) "AAA|authorization" +2) "AAA|nss" +3) "AAA|authentication" +127.0.0.1:6379[4]> keys *RADIUS* +1) "RADIUS_SERVER|10.59.143.229" +2) "RADIUS|global" +127.0.0.1:6379[4]> keys *LDAP* +1) "LDAP|global" +2) "LDAP|nss" +3) "LDAP|pam" +4) "LDAP|sudo" +5) "LDAP_SERVER|10.59.143.229" +127.0.0.1:6379[4]> hgetall AAA|nss +1) "passwd" +2) "ldap" +3) "shadow" +4) "ldap" +5) "group" +6) "ldap" +5) "sudoers" +6) "ldap" +127.0.0.1:6379[4]> hgetall AAA|authentication +1) "failthrough" +2) "True" +3) "login" +4) "radius,local" +127.0.0.1:6379[4]> hgetall AAA|authorization +1) "login" +2) "ldap" +127.0.0.1:6379[4]> hgetall RADIUS|global +1) "passkey" +2) "sharedsecret" +127.0.0.1:6379[4]> hgetall RADIUS_SERVER|10.59.143.229 +1) "priority" +2) "1" +3) "auth_port" +4) "1812" +127.0.0.1:6379[4]> hgetall LDAP|global +1) "base" +2) "dc=sji,dc=broadcom,dc=net" +127.0.0.1:6379[4]> hgetall LDAP|nss +1) "base" +2) "dc=sji,dc=broadcom,dc=net" +127.0.0.1:6379[4]> hgetall LDAP|pam +1) "pam_groupdn" +2) "cn=docker,ou=Group,dc=sji,dc=broadcom,dc=net" +3) "pam_member_attribute" +4) "memberUid" +127.0.0.1:6379[4]> hgetall LDAP|sudo +1) "sudoers_base" +2) "ou=Sudoers,dc=sji,dc=broadcom,dc=net" +127.0.0.1:6379[4]> + +``` + +### show aaa, show radius, show ldap: After + +``` + +admin@L10:~$ show aaa +AAA authentication login radius,local +AAA authentication failthrough True +AAA nss group ldap +AAA nss passwd ldap +AAA nss shadow ldap +AAA nss sudoers ldap +AAA nss netgroup local (default) +AAA authorization login ldap + +admin@L10:~$ + +admin@L10:~$ show radius +RADIUS global auth_type pap (default) +RADIUS global retransmit 3 (default) +RADIUS global timeout 5 (default) +RADIUS global passkey sharedsecret + +RADIUS_SERVER address 10.59.143.229 + priority 1 + auth_port 1812 + +admin@L10:~$ + +admin@L10:~$ show ldap +LDAP global retry 0 (default) +LDAP global ssl off (default) +LDAP global idle_timelimit 0 (default) +LDAP global bind_timelimit 10 (default) +LDAP global timelimit 0 (default) +LDAP global ldap_version 3 (default) +LDAP global scope sub (default) +LDAP global base dc=sji,dc=broadcom,dc=net +LDAP global use_type all (default) +LDAP global port 389 (default) +LDAP nss base dc=sji,dc=broadcom,dc=net +LDAP pam pam_member_attribute memberUid +LDAP pam pam_groupdn cn=docker,ou=Group,dc=sji,dc=broadcom,dc=net +LDAP sudo sudoers_base ou=Sudoers,dc=sji,dc=broadcom,dc=net + +LDAP_SERVER address 10.59.143.229 + priority 1 + retry 0 + use_type all + +admin@L10:~$ + +``` + +### KLISH: show aaa, show radius-server, show ldap-server: After + +``` +sonic# show aaa +--------------------------------------------------------- +AAA Authentication Information +--------------------------------------------------------- +failthrough : True +login-method : radius, local +--------------------------------------------------------- +AAA Authorization Information +--------------------------------------------------------- +login : ldap +--------------------------------------------------------- +AAA Name-Service Information +--------------------------------------------------------- +passwd : ldap +shadow : ldap +group : ldap +sudoers : ldap +sonic# +sonic# show radius-server +--------------------------------------------------------- +RADIUS Global Configuration +--------------------------------------------------------- +key : sharedsecret +-------------------------------------------------------------------------------- +HOST AUTH-TYPE KEY AUTH-PORT PRIORITY TIMEOUT RTSMT VRF +-------------------------------------------------------------------------------- +10.59.143.229 - - 1812 1 - - - +sonic# +sonic# show ldap-server +--------------------------------------------------------- +LDAP Global Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +--------------------------------------------------------- +LDAP NSS Configuration +--------------------------------------------------------- +binddn : dc=sji,dc=broadcom,dc=net +--------------------------------------------------------- +LDAP PAM Configuration +--------------------------------------------------------- +pam_member_attribute : memberUid +pam_groupdn : cn=docker,ou=Group,dc=sji,dc=broadcom,dc=net +--------------------------------------------------------- +LDAP sudo Configuration +--------------------------------------------------------- +sudoers_base : ou=Sudoers,dc=sji,dc=broadcom,dc=net +-------------------------------------------------------------------------------- +HOST USE-TYPE PRIORITY SSL RETRY +-------------------------------------------------------------------------------- +10.59.143.229 - 1 - - +sonic# + +``` + +### Verify user can login using RADIUS password with LDAP Name Service LDAP logon authorization and sudo authorization + +``` + +$ ssh raduser1@10.59.143.109 +raduser1@10.59.143.109's password: +Linux L10 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 +... +raduser1@L10:~$ id +uid=1202(raduser1) gid=100(users) groups=100(users),999(docker) +raduser1@L10:~$ sudo -i +[sudo] password for raduser1: +root@L10:~# id +uid=0(root) gid=0(root) groups=0(root) +root@L10:~# exit +logout +raduser1@L10:~$ + +``` + +## UT Authentication LDAP AD ? For Implementation In a Future Release (IIFR) + +### Click CLI + +``` + +sudo config ldap add 10.75.16.174 +sudo config ldap base 'OU=San Jose,OU=California,OU=US,OU=Users,OU=Accounts,DC=Broadcom,DC=net' +sudo config ldap binddn 'binddn CN=Search(035)Account,OU=San Jose,OU=California,OU=US,OU=Users,OU=Accounts,DC=Broadcom,DC=net' +sudo config ldap bindpw searchPW +sudo config ldap pam_login_attribute sAMAccountName +sudo config ldap pam_filter objectclass=User + +sudo config ldap map add attribute uid sAMAccountName shadowLastChange pwdLastSet uniqueMember member +sudo config ldap map add objectclass posixAccount user shadowAccount user posixGroup group + + +sudo config aaa authentication login ldap local +sudo config aaa authentication failthrough enable + +``` + +### Config DB: After + +``` + +127.0.0.1:6379[4]> keys *LDAP* +1) "LDAP_MAP|OBJECTCLASS|posixGroup" +2) "LDAP_MAP|ATTRIBUTE|shadowLastChange" +3) "LDAP_MAP|ATTRIBUTE|uid" +4) "LDAP_MAP|ATTRIBUTE|uniqueMember" +5) "LDAP_MAP|OBJECTCLASS|posixAccount" +6) "LDAP_MAP|OBJECTCLASS|shadowAccount" +7) "LDAP_SERVER|10.75.16.174" +8) "LDAP|global" +127.0.0.1:6379[4]> keys *AAA* +1) "AAA|authentication" +127.0.0.1:6379[4]> hgetall LDAP_SERVER|10.75.16.174 +1) "priority" +2) "1" +3) "retry" +4) "0" +5) "use_type" +6) "all" +127.0.0.1:6379[4]> hgetall LDAP|global + 1) "pam_login_attribute" + 2) "sAMAccountName" + 3) "pam_filter" + 4) "objectclass=User" + 5) "binddn" + 6) "binddn CN=Search(035)Account,OU=San Jose,OU=California,OU=US,OU=Users,OU=Accounts,DC=Broadcom,DC=net" + 7) "bindpw" + 8) "searchPW" + 9) "base" +10) "OU=San Jose,OU=California,OU=US,OU=Users,OU=Accounts,DC=Broadcom,DC=net" +127.0.0.1:6379[4]> hgetall LDAP_MAP|OBJECTCLASS|posixGroup +1) "to" +2) "group" +127.0.0.1:6379[4]> hgetall LDAP_MAP|ATTRIBUTE|shadowLastChange +1) "to" +2) "pwdLastSet" +127.0.0.1:6379[4]> hgetall LDAP_MAP|ATTRIBUTE|uid +1) "to" +2) "sAMAccountName" +127.0.0.1:6379[4]> hgetall LDAP_MAP|ATTRIBUTE|uniqueMember +1) "to" +2) "member" +127.0.0.1:6379[4]> hgetall LDAP_MAP|OBJECTCLASS|posixAccount +1) "to" +2) "user" +127.0.0.1:6379[4]> hgetall LDAP_MAP|OBJECTCLASS|shadowAccount +1) "to" +2) "user" +127.0.0.1:6379[4]> + +``` + +### show aaa, show ldap: After + +``` + +admin@L10:~$ id aduser +uid=95431(aduser) gid=95431 groups=95431 +admin@L10:~$ +admin@L10:~$ show aaa +AAA authentication login ldap,local +AAA authentication failthrough True +AAA nss passwd login (default) +AAA nss shadow login (default) +AAA nss group login (default) +AAA nss netgroup local (default) +AAA authorization login local (default) + +admin@L10:~$ show ldap +LDAP global binddn CN=Search(035)Account,OU=San Jose,OU=California,OU=US,OU=Users,OU=Accounts,DC=Broadcom,DC=net +LDAP global retry 0 (default) +LDAP global ssl off (default) +LDAP global idle_timelimit 0 (default) +LDAP global bind_timelimit 10 (default) +LDAP global timelimit 0 (default) +LDAP global base OU=San Jose,OU=California,OU=US,OU=Users,OU=Accounts,DC=Broadcom,DC=net +LDAP global bindpw searchPW +LDAP global ldap_version 3 (default) +LDAP global pam_filter objectclass=User +LDAP global scope sub (default) +LDAP global pam_login_attribute sAMAccountName +LDAP global use_type all (default) +LDAP global port 389 (default) + +LDAP_MAP_ATTRIBUTE + uniqueMember member + shadowLastChange pwdLastSet + uid sAMAccountName + +LDAP_MAP_OBJECTCLASS + shadowAccount user + posixGroup group + posixAccount user + +LDAP_SERVER address 10.75.16.174 + priority 1 + retry 0 + use_type all + +admin@L10:~$ + +``` + +### Verify AD user can login with LDAP password. + +``` +$ ssh aduser@10.59.143.205 +aduser@10.59.143.205's password: +Creating directory '/home/aduser'. +Linux L10 4.9.0-11-2-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 +... +aduser@L10:~$ + +``` + +# References + +## nss_ldap + +https://manpages.debian.org/testing/libnss-ldap/libnss-ldap.conf.5.en.html + +## pam_ldap + +https://manpages.debian.org/testing/libpam-ldapd/pam_ldap.8.en.html + +## sudo_ldap + +https://www.sudo.ws/man/1.8.17/sudoers.ldap.man.html + +## Linux PAM + +http://man7.org/linux/man-pages/man3/pam.3.html + +## NSS Reference + +http://man7.org/linux/man-pages/man5/nsswitch.conf.5.html + +## LDAP client configuration file + +https://linux.die.net/man/5/ldap.conf + +## TACPLUS Authentication + +https://github.com/Azure/SONiC/blob/master/doc/aaa/TACACS%2B%20Authentication.md + +## RADIUS Management User Authentication + +https://github.com/Azure/SONiC/blob/c82c8b67c31a63b1b438a143e2ce1a92b2b580fb/doc/aaa/radius_authentication.md + + diff --git a/doc/audit-log/Audit_Log_HLD.md b/doc/audit-log/Audit_Log_HLD.md new file mode 100644 index 0000000000..13d1956c10 --- /dev/null +++ b/doc/audit-log/Audit_Log_HLD.md @@ -0,0 +1,393 @@ +# Feature Name +Audit Log. +# High Level Design Document +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 06/16/2020 | Srinadh Penugonda | Initial version | + +# About this Manual +This document provides general information about the Audit Log feature implementation in SONiC. +# Scope +This document describes the high level design of Audit Log feature. + +# 1 Feature Overview + +Audit log provides a way to monitor several security relevant information on the system with an insight on security health of the system. + +## 1.1 Requirements + +- The log messages will be stored as a separate file under /var/log on the switch itself and will follow similar format to syslog messages. No support for remote logging or remote destination. + +- The audit log should contain the following messages: + 1. Log messages corresponding to login and logout through SSH and Console. The message content & format is from open source modules and is fixed. + 2. Log messages corresponding to when user tries config & show command via KLISH, gNMI and REST. The command message will not have command hierarchy information. + + Audit log result for REST & gNMI will be in simple flat list; message will not be in key/value format. + +- Log rotate is used to enforce size as 10M for audit log and goes through four rotations before being removed. + +- Audit log files should be included in tech-support bundle. + +### 1.1.2 Configuration and Management Requirements +- There need to be a show command to display the contents of audit log. + 1. show audit-log all - lists all messages from audit.log and rotated, uncompressed audit logs - namely, audit.log.1. + The command will not attempt to display compressed and rotated audit log files. + + The *all* filter will fetch all audit log messages and potentially have performance implications and high response time. + + 2. show audit-log - lists recent 20 messages from audit.log + +- There need to be a clear command to clear contents of audit logs. + +## 1.2 Design Overview +### 1.2.1 Basic Approach +The feature implements the framework for audit log. As of now, the audit log comprises of messages corresponding to +* login and logout related events through SSH and console +* requests from north bound interfaces: CLI, REST and gNMI + +Syslog rules are used to filter out these messages and populate audit log. + +The login linux package is updated to trigger syslog message for login through console. + +Any subsequent requirements to add new audit messages need to add a new syslog rule or augment existing syslog rules +as mentioned in this [section](#311-syslog-rules) + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Audit Log collects certain types of system activity to facilitate incident investigation. The system activities includes +login/logout events through ssh/console. The other correpsonds to requests from north bound interfaces - CLI, REST and gNMI. + +# 3 Design +## 3.1 Overview +Specified audit messages are stored in /var/log/audit.log. + +Syslog rules with the help of various syslog properties like *syslogtag*, *syslogfacility*, *programname* are used to filter +these messages and redirect them to /var/log/audit.log. + +SSHD triggers messages for login and logout through ssh. For login and logout through console, linux login package need to be +modified to trigger a syslog message. + +Log rotate would manage audit log with size restrictions of 1M. + +### 3.1.1 Syslog Rules +SSHD generates successful login/logout messages with *authpriv* facility. Messages corresponding to invalid credentials triggered +with *auth* facility. + +``` +if $syslogtag contains 'sshd' then { + if $syslogfacility-text == 'authpriv' then { + action(type="omfile" file="/var/log/audit.log") + } + if $syslogfacility-text == 'auth' then { + action(type="omfile" file="/var/log/audit.log") + } +} +``` +Management framework log messages are sent with *local4* facility. +``` +if $syslogfacility-text == 'local4' then { + action(type="omfile" file="/var/log/audit.log") +} +``` +Some of the console login/logout messages comes through *systemd*. +``` +if $programname == 'systemd' then { + action(type="omfile" file="/var/log/audit.log") +} +``` + +The login linux package triggers messages corresponding to successful and failed login attempts with *programname* as **login**. +``` +if $programname == 'login' then { + action(type="omfile" file="/var/log/audit.log") +} +``` + +### 3.1.2 Linux login package +This package is modified to trigger a syslog message after user logs in. The build system is updated to build login target. + +### 3.1.3 Commands +'show audit-log' displays contents of audit.log and audit.log.1. +By default the command displays a brief snapshot of audit log by displaying around latest twenty messages. +With *all* option, all of the audit.log and audit.log.1 is displayed. + +Through REST, the path is /restconf/operations/sonic-auditlog:get-auditlog +For gNMI, the path is sonic-auditlog:get-auditlog + +``` +sonic# show audit-log + Display all of audit log + | Pipe through a command + +``` +Through REST, the path is /restconf/operations/sonic-auditlog:clear-auditlog +For gNMI, the path is /sonic-auditlog:clear-auditlog + +``` +sonic# clear audit-log + + +``` + +### 3.1.4 Log rotate +Log rotate is configured to enforce size limitation of 10M for audit log. There will be four rotations. Rsyslog will be restated +after each rotation. + +### 3.1.5 show tech-support +The existing show tech-support already packages all the files under /var/log into tech-support package. + +Currently, the following log files from /var/log are packaged with tech support: +audit.log, auth.log, cron.log, daemon.log, messages, syslog, stpd.log, teamd.log, telemetry.log, udldd.log, user.log and zebra.log + +## 3.2 User Interface +### 3.2.1 Data Models +``` +module sonic-auditlog { + namespace "http://github.com/Azure/sonic-auditlog"; + prefix auditlog; + yang-version 1.1; + + organization + "SONiC"; + + contact + "SONiC"; + + description + "SONiC yang for RPC based audit log operations."; + + revision 2020-05-29 { + description + "Initial revision."; + } + + rpc get-auditlog { + description "RPC for getting audit log."; + + input { + leaf content-type { + type string { + pattern "all"; + } + description "Indicates if user wants to get all or latest twenty lines of audit log"; + } + } + + output { + list audit-content { + leaf content { + type string; + description "Audit message"; + } + } + } + } + + rpc clear-auditlog { + description "RPC for clearing audit log."; + } +} + +``` +### 3.2.2 CLI +#### 3.2.2.1 Clear Commands +Clearing audit log is implemented as RPC and is achieved with the help of HostQuery module - clearaudit.py. +On receiving the request from container, this module will remove audit.log and restarts rsyslog. + +``` +sonic# clear audit-log +sonic# +``` + +#### 3.2.2.2 Show Commands +To display contents of audit log, show command is implemented and it is RPC. +To achieve better performance, audit log is mounted onto mgmt-framework container. +The command by default displays a brief snapshot of audit log by showing last twenty lines. +The command has an 'all' option to display all of the audit.log and audit.log.1. + +``` +sonic# show audit-log + Display all of audit log + | Pipe through a command + +``` + +### 3.2.3 REST API Support +#### 3.2.3.1 Show Audit Log +Path: /restconf/operations/sonic-auditlog:get-auditlog + +Input: "all" or none + +#### 3.2.3.2 Clear Audit Log +Path: /restconf/operations/sonic-auditlog:clear-auditlog + +### 3.2.4 GNMI API Support +#### 3.2.4.1 Show Audit Log +Path: /sonic-auditlog:get-auditlog + +Input: "all" or none + +#### 3.2.4.2 Clear Audit Log +Path: /sonic-auditlog:clear-auditlog + +# 4 Serviceability and Debug + +- **Successful login** + +Provides user name, IP from where user is logging in from. + +*SSH* + +Jun 2 22:47:08.619590 sonic INFO sshd[13990]: Accepted password for **admin** from **10.14.8.140** port 49074 ssh2 +Jun 2 22:47:08.711691 sonic INFO sshd[13990]: pam_unix(sshd:session): **session opened** for user **admin** by (uid=0) + +The first message displays that authentication was successful for the user 'admin'. The IP of the host from where host is trying to login is displayed along with its port. + +Timestamp at which the login occurred is displayed. + +The subsequent log indicates that session has been opened for user 'admin'. + +*Console* + +Jun 2 22:48:47.939333 sonic INFO login[30983]: Accepted password for **admin** on terminal=**'/dev/ttyS0'** +Jun 2 22:48:48.056522 sonic INFO login[30983]: pam_unix(login:session): **session opened** for user **admin** by LOGIN(uid=0) + +- **Successful logout** + +Provides user name, IP from where user is logging out from. + +*SSH* + +Jun 2 22:49:33.855434 sonic INFO sshd[14073]: Received disconnect from **10.14.8.140** port 49074:11: **disconnected by user** +Jun 2 22:49:33.966591 sonic INFO sshd[13990]: pam_unix(sshd:session): **session closed** for user **admin** + +The first message indicates that user has disconnected the session. The IP and port of the host from which user disconnected the session is displayed as well. + +The second message indicates that session for user 'admin' has been closed. + +Timestamp at which the login occurred is displayed. + +*Console* + +Jun 2 22:50:47.510380 sonic INFO login[30983]: pam_unix(login:session): **session closed** for user **admin** +Jun 2 22:50:47.665222 sonic INFO systemd[1]: Stopped Serial Getty on **ttyS0** + +- **Login with invalid username** + +Provides invalid user name and IP address of the host invalid user is trying to log in from. + +*SSH* + +Jun 2 22:51:53.619712 sonic INFO sshd[31688]: **Invalid user** **adminxxx** from **10.14.8.140** port 49090 + +The message indicates that there was an attempt to connect over ssh with an invalid user name. + +The user name is displayed along with the host IP and port from which user tried to login. + +*Console* + +Jun 2 22:52:49.080189 sonic NOTICE login[27921]: pam_unix(login:auth): **authentication failure**; logname=LOGIN uid=0 euid=0 **tty=/dev/ttyS0** ruser= rhost= +Jun 2 22:52:52.055801 sonic NOTICE login[27921]: FAILED LOGIN (1) on '/dev/ttyS0' FOR 'UNKNOWN', Authentication failure + +- **Login with invalid password** + +Informs that specified user trying to log in with an invalid password and informs IP of the host user is logging in from. + +*SSH* + +Jun 2 22:53:39.571670 sonic NOTICE sshd[6296]: pam_unix(sshd:auth): **authentication failure**; logname= uid=0 euid=0 tty=ssh ruser= rhost=**10.14.8.140** user=**admin** +Jun 2 22:53:41.163938 sonic INFO sshd[6296]: Failed password for admin from 10.14.8.140 port 49094 ssh2 + +The message indicates that user 'admin' entered wrong password that resulted in authentication failure. + +The host IP and port from which user tried the login is displayed in the message. + +*Console* + +Jun 2 22:54:54.938982 sonic NOTICE login[6927]: pam_unix(login:auth): **authentication failure**; logname=LOGIN uid=0 euid=0 **tty=/dev/ttyS0** ruser= rhost= user=**admin** +Jun 2 22:54:57.568058 sonic NOTICE login[6927]: FAILED LOGIN (1) on '/dev/ttyS0' FOR 'admin', Authentication failure + +- **Session Timeout** + +*SSH* +Jun 10 19:26:32.887528 sonic INFO sshd[24578]: **Timeout, client not responding.** + +Jun 10 19:26:33.025597 sonic INFO sshd[24481]: pam_unix(sshd:session): **session closed** for user **admin** + +*Console* + +Jun 10 20:02:33.904878 sonic INFO systemd[1]: **Stopped Serial Getty** on **ttyS0**. + +- **Set Request** + +Informs type of request, URI, user name, command string and status of command execution. + +*CLI* + +Jun 2 22:57:09.060819 sonic INFO mgmt-framework#clish: User "**admin**" command "**tacacs-server key mykey**" status - **success** + +The message displays the command string; its status and name of the user that executed the command. + +*REST/gNMI* + +Jun 12 19:33:40.728039 sonic INFO mgmt-framework#/usr/sbin/rest_server[711]: [REST-5] User "**admin@10.14.125.28**:55937" request "**PATCH** **/restconf/data/openconfig-system:system/aaa/server-groups/server-group=TACACS/config/openconfig-system-ext:secret-key**" status - **204** + +- **Delete Request** + +Informs type of request, URI, user name, command string and status of command execution. + +*CLI* + +May 27 21:19:32.141471 sonic INFO mgmt-framework#clish: User "**admin**" command "**no tacacs-server timeout**" status - **success** + +The message displays the command string; its status and name of the user that executed the command. + +*REST/gNMI* + +Jun 12 19:35:03.326971 sonic INFO mgmt-framework#/usr/sbin/rest_server[711]: [REST-6] User "**admin@10.14.125.28**:55937" request "**DELETE /restconf/data/openconfig-system:system/aaa/server-groups/server-group=TACACS/config/openconfig-system-ext:secret-key**" status - **204** + +- **Get Request** + +Informs type of request, URI, user name, command string and status of command execution. + +*CLI* + +Jun 2 22:55:55.171404 sonic INFO mgmt-framework#clish: User "**admin**" command "**show tacacs-server global**" status - **success** + +The message displays the command string; its status and name of the user that executed the command. + +*REST/gNMI* + +Jun 12 19:36:04.059130 sonic INFO mgmt-framework#/usr/sbin/rest_server[711]: [REST-7] User "**admin@10.14.125.28**:55937" request "**GET /restconf/data/openconfig-system:system/aaa/server-groups/server-group=TACACS/config/openconfig-system-ext:secret-key**" status - **200** + +# 5 Unit Test +- show audit-log : Should display around last twenty audit messages +- show audit-log all : Should display audit.log and audit.log.1 (if it exists) +- clear audit-log : clears audit-log +- Login through ssh, verify that show audit-log displays messages corresponding to login +- Logout through ssh, verify that show audit-log displays messages corresponding to logout +- Login through console, verify that show audit-log displays messages corresponding to login +- Logout through console, verify that show audit-log displays messages corresponding to logout +- Login with invalid password through ssh, verify that show audit-log displays messages corresponding to login +- Login with invalid password through console, verify that show audit-log displays messages corresponding to login +- Login with invalid username through ssh, verify that show audit-log displays messages corresponding to login +- Login with invalid username through console, verify that show audit-log displays messages corresponding to login +- timeout for an existing ssh session, verify that show audit-log displays messages corresponding to session clearing out +- timeout for an existing console session, verify that show audit-log displays messages corresponding to session clearing out +- set request through CLI, REST, gNMI, verify show audit-log displays user, command string (for CLI) / path (for REST/gNMI) and status +- get request through CLI, REST, gNMI, verify show audit-log displays user, command string (for CLI) / path (for REST/gNMI) and status +- delete request through CLI, REST, gNMI, verify show audit-log displays user, command string (for CLI) / path (for REST/gNMI) and status +- Verify that show tech-support has audit log packaged diff --git a/doc/event-mgmt-framework/event-alarm-framework-alarm-lifecycle.png b/doc/event-mgmt-framework/event-alarm-framework-alarm-lifecycle.png new file mode 100644 index 0000000000..9773603e4b Binary files /dev/null and b/doc/event-mgmt-framework/event-alarm-framework-alarm-lifecycle.png differ diff --git a/doc/event-mgmt-framework/event-alarm-framework-blockdiag.png b/doc/event-mgmt-framework/event-alarm-framework-blockdiag.png new file mode 100644 index 0000000000..d735f9b1b3 Binary files /dev/null and b/doc/event-mgmt-framework/event-alarm-framework-blockdiag.png differ diff --git a/doc/event-mgmt-framework/event-alarm-framework-seqdiag.png b/doc/event-mgmt-framework/event-alarm-framework-seqdiag.png new file mode 100644 index 0000000000..5473b2c63a Binary files /dev/null and b/doc/event-mgmt-framework/event-alarm-framework-seqdiag.png differ diff --git a/doc/event-mgmt-framework/event-alarm-framework.md b/doc/event-mgmt-framework/event-alarm-framework.md new file mode 100644 index 0000000000..fee86b7425 --- /dev/null +++ b/doc/event-mgmt-framework/event-alarm-framework.md @@ -0,0 +1,1265 @@ +# Feature Name +Event and Alarm Framework +# High Level Design Document +#### Rev 0.2 + +# Table of Contents + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + * [1.1.1 Functional Requirements](#111-functional-requirements) + * [1.2 Design Overview](#12-design-overview) + * [1.2.1 Basic Approach](#121-basic-approach) + * [1.2.2 Container](#122-container) + * [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-user-cases) + * [2.2 Functional Description](#22-functional-description) + * [3 Design](#3-description) + * [3.1 Overview](#31-overview) + * [3.1.1 Event Producers](#311-event-producers) + * [3.1.1.2 Development Process](#3112-development-process) + * [3.1.2 Event Consumer](#312-event-consumer) + * [3.1.2.1 Severity](#3121-severity) + * [3.1.2.2 Sequence-ID](#3122-sequence-id) + * [3.1.3 Alarm Consumer](#313-alarm-consumer) + * [3.1.4 Event Receivers](#314-event-receivers) + * [3.1.4.1 syslog](#3141-syslog) + * [3.1.4.2 REST](#3142-rest) + * [3.1.4.3 gNMI](#3143-gnmi) + * [3.1.4.4 System LED](#3144-system-led) + * [3.1.4.5 Event/Alarm flooding](#3145-event/alarm-flooding) + * [3.1.4.6 Eventd continuous restart](#3146-event-continuous-restart) + * [3.1.5 Event Profile](#315-event-profile) + * [3.1.6 CLI](#316-cli) + * [3.1.7 Event History Table and Current Alarm Table](#317-event-history-table-and-current-alarm-table) + * [3.1.8 Pull Model](#318-pull-model) + * [3.1.9 Supporting third party containers](#319-supporting-third-party-containers) + * [3.2 DB Changes](#32-db-changes) + * [3.2.1 EVENT DB](#321-event-db) + * [3.3 User Interface](#33-user-interface) + * [3.3.1 Data Models](#331-data-models) + * [3.3.2 CLI](#332-cli) + * [3.3.2.1 Exec Commands](#3321-exec-commands) + * [3.3.2.2 Configuration Commands](#3322-configuration-commands) + * [3.3.2.3 Show Commands](#3323-show-commands) + * [3.3.3 REST API Support](#333-rest-api-support) + * [4 Flow Diagrams](#4-flow-diagrams) + * [5 Warm Boot Support](#5-warm-boot-support) + * [5.1 Application warm boot](#51-application-warm-boot) + * [5.2 eventd warm boot](#52-eventd-warm-boot) + * [6 Scalability](#6-scalability) + * [7 Showtech Support](#7-showtech-support) + * [8 Unit Test](#8-unit-test) + + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|----------------------------------- | +| 0.1 | 03/20/2021 | Srinadh Penugonda | Initial Version | +| 0.2 | 04/30/2021 | Srinadh Penugonda | Updated with comments from HLD review | + +# About this Manual +This document provides general information on the implementation and functionality of Event and Alarm Framework in SONiC. + +Note: Wherever CLI is specified, it is the CLISH cli that is referred - SONiC native (CLICK) CLI is not updated for this feature. + +# Scope +This document describes the high-level design of Event and Alarm Framework. +It is not in the scope of the framework to update ANY of the applications to raise events and alarms. + +# 1 Feature Overview + +The Event and Alarm Framework feature provides a centralized framework for applications in SONiC to raise notifications and store them for NBIs to listen and fetch to monitor the device. + +Events and Alarms are means to indicate a change in the state of the system that operator may be interested in. +Such a change has an important metric called *severity* to indicate how critical it is to the health of the system. + +* Events + + Events are "one shot" notifications to indicate an abnormal/important situation. + + User logging in, authentication failure, configuration changed notification are all examples of events. + +* Alarms + + Alarms are notifications raised for conditions that could be cleared by correcting or removal of such conditions. + + Out of memory, temperature crossing a threshold, and so on, are examples of conditions when the alarms are raised. + Such conditions are dynamic: a faulty software/hardware component encounters the above such condition and **may** come out of that situation when the condition is resolved. + + Events are sent as the condition progresses through being raised and cleared in addition to operator acknowledging/unacknowledging it. + So, these events have a field called *action*: RAISE, CLEAR or ACKNOWLEDGE/UNACKNOWLEDGE. + + Each of such events for an alarm is characterized by "action" in addition to "severity". + + An application *raises* an alarm when it encounters a faulty condition by sending an event with action: *RAISE*. + After the application recovers from the condition, that alarm is *cleared* by sending an event with action: *CLEAR*. + An operator could *ACKNOWLEDGE/UNACKNOWLEDGE* an alarm. This indicates that the operator is aware of the faulty condition. + + The set of alarms and their severities are an indication to health of various applications of the system and System LED can be deduced from alarms. + An acknowledged alarm means that operator is aware of the condition so, acknowledged alarm will be taken out of consideration. + +Both events and alarms get recorded in a new DB called EVENT DB in a new redis instance. + +1. Event History Table + + All events get recorded in the event history table, by name, "EVENT". EVENT table contains history of all events generated by the system. + This table is persisted across system restarts of any kind, including restore to factory defaults and SW upgrades and downgrades. + +2. Current Alarm Table + + All events with an action field of *RAISE* get recorded in a table, by name, "ALARM" in addition to getting recorded in Event History Table ( only events corresponding to an alarm has action field ). + When an application that raised the alarm clears it ( by sending an event with action *CLEAR* ), the alarm record is removed from ALARM table. + An user acknowledging a particular alarm will NOT remove that alarm record from this table; only when application clears it, the alarm is removed from ALARM table. + + In effect, ALARM table contains outstanding alarms that need to be cleared by those applications who raised them. + This table is NOT persisted and its contents are cleared with a reload. + +In summary, the framework provides both current and historical event status of software and physical entities of the system through ALARM and EVENT tables. + +In addition to the above tables, the framework maintains various statisitcs. + +1. Event Statistics Table + + Statistics on number of events and alarms are maintained in EVENT_STATS table. + +2. Alarm Statistics Table + + Statistics on number of alarms per severity are maintained in ALARM_STATS table. + When application raises an alarm, the counter corresponding to that alarm's severity is increased by 1. + When the alarm is cleared or acknowledged, the corresponding severity counter will be reduced by 1. + This table categorizes "active" alarms per severity. + +As mentioned above, each event has an important characteristic: severity. SONiC uses following severities for events and alarms as defined in opeconfig alarm yang. + +- CRITICAL : Requires immediate action. An critical event may trigger if one or more hardware components fail, or one or more hardware components exceed temperature thresholds. + ( maps to log-alert ) +- MAJOR : Requires escalation or notification. For example, a major alarm may trigger if an interface failure occurs, such as a port channel being down. + ( maps to log-critical ) +- MINOR : If left unchecked, might cause system service interruption or performance degradation. An alarm with minor severity requires monitoring or maintenance. + ( maps to log-error ) +- WARNING : It may or may not result in an error condition. + ( maps to log-warning ) +- INFORMATIONAL : Does not impact performance. NOT applicable to alarms. + ( maps to log-notice ) + +The following describes how an alarm transforms and how various tables are updated. +![Alarm Life Cycle](event-alarm-framework-alarm-lifecycle.png) + +By default every event will have a severity assigned by the component. The framework provides Event Profiles to customize severity of an event and also disable an event. + +Template for event profile is as below: +``` +{ + "events":[ + { + "name" : , + "severity" : , + "enable" : , + "message" : + } + ] +} +``` +Event Profiles only contains declarations of events and their characteristics. There has to be an application to raise these events using eventnotify API. + +The framework maintains default event profile at /etc/evprofile/default.json. +Operator can download default event profile to a remote host. +This downloaded file can be modified by changing the severity or enable flag of event(s). +This modified file can then be uploaded to the device to /etc/evprofile/. +Operator can select any of these custom event profiles to change default properties of events. +The selected profile is persistent across reboots and will be in effect until operator selects either default or another custom profile. + +In addition to storing events in DB, framework forwards log messages corresponding to all the events to syslog. +Syslog message displays the type (ALARM or EVENT), action (RAISE, CLEAR, ACKNOWLEDGE or UNACKNOWLEDGE) - when the message corresponds to an event of an alarm, name of the event and detailed message. + +gNMI clients can subscribe to receive events as they are raised. Subscribing through REST is being evaluated. + +CLI and REST/gNMI clients can query either table with filters - based on severity, delta based on timestamp, sequence-id etc., + +Application owners need to identify various conditions that would be of interest to the operator and use the framework to raise events/alarms. + +## 1.1 Requirements + + +### 1.1.1 Functional Requirements + +| ID | Requirement | Comment | +| :--- | :---- | :--- | +| 1 | Provide API via library for apps to publish events | | +| 2 | Provide API via library for apps to publish alarms | | +| 3 | Event Infra to write formatted syslog messages corresponding to all events to Syslog. | | +| 4 | Event Infra to persist all events and alarms in DB. | | +| 5 | Event Infra to read Event profile ( severity and enable/disable flag ) from a json file. | | +| 6 | Event Infra to read Event table parameters (size and # of days) from a config file. | | +| 7 | NBI interface (gNMI and REST) and CLI | | +| 7.1 | Events | | +| 7.1.1 | Openconfig interface to pull event information. | | +| 7.1.2 | Openconfig interface to pull event summary information. | | +| | Event summary information to contain cumulative counters for: | | +| | - Raised-count (events) | | +| 7.1.3 | Openconfig interface to pull events using following filters | | +| | - ALL ( pull all events) | | +| | - Severity. | | +| | - Recent records (eg., last 5 minutes, one hour, one day). | | +| | - Records between two timestamps, one timestamp and end, and beginning and a timestamp. | | +| | - All records between two Sequence Numbers (incl begin and end) | | +| 7.2 | Alarms | | +| 7.2.1 | Openconfig interface to pull alarm information. | | +| 7.2.2 | Openconfig interface to pull alarm summary information. | | +| | Counters for Total, Critical, Major, Minor, Warning, Acknowledged | | +| 7.2.3 | Openconfig interface to pull alarms using following filters | | +| | - All (pull all events) | | +| | - Severity. | | +| | - Recent alarms (eg., last 5 minutes, one hour, one day). | | +| | - Records between two timestamps, one timestamp and end, and beginning and a timestamp. | | +| | - All records between two Sequence Numbers (incl end and begin) | | +| 7.2.4 | Openconfig interface to acknowledge an alarm. | | +| 8 | CLI commands | | +| 8.1 | show alarm [ detail \| summary \| severity \| timestamp \| recent <5min\|1hr\|1day> \| sequence-number \| all] | | +| 8.2 | show event [ detail \| summary \| severity \| timestamp \| recent <5min\|1hr\|1day> \| sequence-number ] | | +| 8.3 | show event profile | | +| 8.4 | alarm acknowledge | | +| 8.5 | logging server [ log \| event ] | default is 'log' | +| 8.6 | event profile [ default \| name-of-file ] | | +| 9 | gNMI subscription | | +| 9.1 | Subscribe to openconfig Event container and Alarm container. All events and alarms published to gNMI subscribed clients. | | +| 10 | Clear all events | | +| 11 | Any change in open source should be aligned and upstream. | | + +## 1.2 Design Overview + +![Block Diagram](event-alarm-framework-blockdiag.png) + +### 1.2.1 Basic Approach +The feature involves new development. +Applications act as producers by writing to a table with the help of event notify library. +Eventd reads new record in the table and processes it: +It saves the entry in event history table; if the event has an action and if it is *RAISE*, record gets added to alarm table, severity counter in ALARM_STATS is increased. +If the received event action is *CLEAR*, record in the ALARM table is removed and severity counter in ALARM_STATS of that alarm is reduced by 1. +If eventd receives an event with action *ACKNOWLEDGE* from mgmt-framework, severity counter in ALARM_STATS is reduced by 1. +If eventd receives an event with action *UNACKNOWLEDGE* from mgmt-framework, severity counter in ALARM_STATS is increased by 1. +Eventd then informs logging API to format the log message and send the message to syslog. + +Any application like pmon can subscribe to tables like ALARM_STATS to act accordingly. + +### 1.2.2 Container +A new container by name, eventd, is created to hold event consumer logic. + +# 2 Functionality +## 2.1 Target Deployment Use Cases + +The framework assigns an unique sequence number to each of the events sent by applications. + +In addition, the framework provides the following key management services: + +- Push model: Event/Alarm information to remote syslog hosts and subscribed gNMI clients +- Pull model: Event/Alarm information from CLI, REST/gNMI interfaces +- Ability to change severity of events, turn off a particular event +- Ability to acknowledge an alarm + +## 2.2 Functional Description +Event Management Framework allows applications to store "state" of the system for user to query through various north bound interfaces. + +# 3 Design +## 3.1 Overview +There are three players in the event framework. Producers, which raises events; a consumer to receive and process them as they are raised and a set of receivers one for each NBI type. + +Applications act as producers of events. + +Event consumer class in eventd container receives and processes the received event. +Event consumer manages received events, updates event history table, current alarm table, event_stats table and alarm_stats tables and invokes logging API, which constructs message and sends it over to syslog. + +Operator can chose to change properties of events with the help of event profile. Default +event profile is available at */etc/evprofile/default.json*. User can download the default event profile, +modify and upload it back to the switch to apply it. + +Through event profile, user can change severity of any event and also can enable/disable a event. + +Through CLI, REST or gNMI, event history table and current alarm table can be retrieved using various filters. + +### 3.1.1 Event Producers +Application that need to raise an event, need to use event notifiy API ( LOG_EVENT ). +This API is part of *libeventnotify* library that applications need to link. + +For one-shot events, applications need to provide event-id (name of the event), source, dynamic message, and event action set to NOTIFY. + +For alarms, applications need to provide event-id (name of the event), source, dynamic message, and event action (RAISE_ALARM / CLEAR_ALARM / ACK_ALARM /UNACK_ALARM). +The ACK_ALARM/UNACK_ALARM action types are used only by mgmt-framework to provide the functionality to acknowledge/unacknowledge the alarms through NBI. + +Eventd maintains a json file of events and alarms at sonic-eventd/etc/evprofile/default.json. This is the default event profile that gets installed on the device at /etc/evprofile/default.json. +Developers of new events or alarms need to update this file by declaring name and other characteristics - severity, enable flag and static message that gets appended with dynamic message. + +``` +{ + "__README__" : "This is default map of events that eventd uses. Developer can modify this file and send + SIGINT to eventd to make it read and use the updated file. Alternatively developer can test + the new event by adding it to a custom event profile and use 'event profile ' command + to apply that profile without sending SIGINT to eventd. Developer need to commit default.json file + with the new event after testing it out. + Supported severities are: CRITICAL, MAJOR, MINOR, WARNING and INFORMATIONAL. + Supported enable flag values are: true and false.", + "events":[ + { + "name" : "CUSTOM_EVPROFILE_CHANGE", + "severity" : "INFORMATIONAL", + "enable" : "true", + "message" : "Custom Event Profile is applied." + }, + { + "name": "TEMPERATURE_EXCEEDED", + "severity": "CRITICAL", + "enable": "true" + "message" : "Temperature threshold is 75 degrees." + } + ] +} +``` +The format of event notify API is: + +definition: +``` + LOG_EVENT(name, source, action, MSG, ...) +``` +- name is name of the event +- source is the object that is generating this event +- action is either NOTIFY, RAISE_ALARM, CLEAR_ALARM, ACK_ALARM or UNACK_ALARM + +Usage: +For one-shot events: +``` + LOG_EVENT(CUSTOM_EVPROFILE_CHANGE, profile_name.c_str(), NOTIFY, "New event profile is %s", profile_name.c_str()); +``` + +For alarms: +``` + if (temperature >= THRESHOLD) { + LOG_EVENT(TEMPERATURE_EXCEEDED, sensor_name_p, RAISE_ALARM, "Temperature for sensor %s is %d degrees", sensor_name_p, current_temp); + } else { + LOG_EVENT(TEMPERATURE_EXCEEDED, sensor_name_p, CLEAR_ALARM, "Temperature for the sensor %s is %d degrees ", sensor_name_p, current_temp); + } +``` +#### 3.1.1.2 Development Process + +Here is a typical developement process to link eventnotify library to a component and be able to send new events/alarms: + +a. Update buildimage/rules/*app*.mk + + Add $(LIBEVENTNOTIFY_DEV) to compile dependency. + + Add $(LIBEVENTNOTIFY) to runtime dependency. + +``` + Ex: For rules/tam.mk, + + $(SONIC_TAM)_DEPENDS += $(LIBEVENTNOTIFY_DEV) + $(SONIC_TAM)_RDEPENDS += $(LIBEVENTNOTIFY) +``` + +b. Update Makefile.am of the app to link to event notify library. +``` + Ex: To let tammgr use event notify API, update src/sonic-tam/tammgr/Makefile.am as below: + + tammgrd_LDADD += -leventnotify +``` +c. Declare the name of new event/alarm along with severity, enable flag and static message in sonic-eventd/etc/evprofile/default.json + +d. In the source file where event is to be raised, include eventnotify.h and invoke LOG_EVENT with action as NOTIFY/RAISE_ALARM/CLEAR_ALARM (ACK_ALARM/UNACK_ALARM are used by mgmt-framework to allow users to acknowledge/unacknowledge alarms). + +The event notifier takes the event properties, packs a field value tuple and writes to a table, by name, EVENTPUBSUB. + +The EVENTPUBSUB table uses event-id and a sequence-id generated locally by event notifier as the key so that there wont be any conflicts across multiple applications trying to write to this table. + +### 3.1.2 Event Consumer +The event consumer is a class in sonic-eventd container that processes the incoming record. + +On intitialization, event consumer reads */etc/evprofile/default.json* and builds an internal map of events, called *static_event_map*. +It then verifies if there was a custom event profile configured and merges its contents to static_event_map built from default event profile. +It then reads from EVENTPUBSUB table. This table contains records that are published by applications and waiting to be read by eventd. +Whenever there is a new record, event consumer reads the record, processes and deletes it. + +On reading the field value tuple, using the event-id in the record, event consumer fetches static information from *static_event_map*. +As mentioned above, static information contains severity, static message and event enable flag. +If the enable flag is set to false, event consumer ignores the event by logging a debug message. +If the flag is set to true, it continues to process the event as follows: +- Generate new sequence-id for the event +- Write the event to Event History Table +- It verifies if the event corresponds to an alarm - by checking the *action* field. If so, alarm consumer API is invoked for the event for further processing. + - If action is RAISE_ALARM, add the record to ALARM table + - If action is CLEAR_ALARM, remove the entry from ALARM table + - If action is ACK_ALARM, update *acknowledged* flag of the corresponding raised entry to true in ALARM table and stores timestamp to *acknowledge_time*. + - If action is UNACK_ALARM, update *acknowledged* flag of the corresponding raised entry to false in ALARM table and stores timestamp to *acknowledge_time*. + - Event and Alarm Statistics tables are updated +- Invoke logging API to send a formatted message to syslog + +#### 3.1.2.1 Severity +Supported event severities: CRITICAL, MAJOR, MINOR, WARNING and INFORMATIONAL as defined opeconfig alarm yang. +The corresponding syslog severities are: log-alert, log-crit, log-error, log-warning and log-notice respectively. +Severity INFORMATIONAL is not applicable to alarms. + +#### 3.1.2.2 Sequence-ID +Every new event should have a unique sequential ID. The sequence-id is persistent and continues to grow until 2 exp 64. + +### 3.1.3 Alarm Consumer +The alarm consume method on receiving the event record, verifies the event action. If it is RAISE_ALARM, it adds the record to Current Alarm Table. +The counter in ALARM_STATS corresponding to the severity of the incoming alarm is increased by 1. + +Eventd maintains a lookup map of *sequence-id* and pair of *event-id* and *resource* fields. +An entry for the newly received event is added to this look up map. + +- If the action is CLEAR_ALARM, it removes the previous record of the raised alarm using above lookup map. + The counter in ALARM_STATS corresponding to the severity of the updated alarm is reduced by 1. + +- If the action is ACK_ALARM, alarm consumer finds the raised record of the alarm in the ALARM table using the above lookup map and updates *acknowledged* flag to true. The *acknowledge-time* is updated with the timestamp of ack event. + ALARM_STATS is updated by reducing the corresponding severity counter by 1. + +- If the action is UNACK_ALARM, alarm consumer finds the raised record of the alarm in the ALARM table using the above lookup map and updates *acknowledged* flag to false. The *acknowledge-time* is updated with the timestamp of unack event. + ALARM_STATS is updated by increasing the corresponding severity counter by 1. + +pmon can use ALARM_STATS to update system LED based on severities of outstanding alarms: +``` + Red if any outstanding critical/major alarms, else Yellow if any minor/warning alarms, else Green. +``` +An outstanding alarm is an alarm that is either not cleared or not acknowledged by the user yet. + +The following illustrates how ALARM table is updated as alarms goes through their life cycle and how can an application use it. +Example here is pmon using ALARM_STATS table to control system LED. + +| alarm | severity | acknowledged | +|:-----:|:----------:|:-------------:| +| | | | +| | | | + +Alarm table is empty. All counters in ALARM_STATS is 0. System LED is Green. + +| alarm | severity | acknowledged | +|:-----:|:----------:|:------------:| +| ALM-1 | CRITICAL | | +| ALM-2 | MINOR | | + +Alarm table now has two alarms. One with *CRITICAL* and other with *MINOR*. ALARM_STATS is updated as: Critical as 1 and Minor as 1. As There is atleast one alarm with *critical/major* severity, system LED is Red. + +| alarm | severity | acknowledged | +|:-----:|:----------:|:------------:| +| ALM-2 | MINOR | | + +The *CRITICAL* alarm is cleared by the application, so alarm consumer removes it from ALARM table, ALARM_STATS is updated as: Critical as 0 and Minor as 1. As there is at least one *minor/warning* alarms in the table, system LED is Amber. + +| alarm | severity | acknowledged | +|:-----:|:----------:|:------------:| +| ALM-2 | MINOR | | +| ALM-9 | MAJOR | | + +Now there is an alarm with *MAJOR* severity. ALARM_STATS now reads as: Major as 1 and Minor as 1. So, system LED is Red. + +| alarm | severity | acknowledged | +|:-----:|:----------:|:------------:| +| ALM-2 | MINOR | | +| ALM-9 | MAJOR | true | + +The *MAJOR* alarm is acknowledged by user, alarm consumer sets *acknolwedged* flag to true and reduces Major counter in ALARM_STATS by 1, ALARM_STATS now reads as: Major 0 and Minor 1. This way, acknowledged major alarm has no effect on system LED. There are no other *CRITICAL/MAJOR* alarms. There however, exists an alarm with *MINOR/WARNING* severity. System LED is Amber. + +| alarm | severity | acknowledged | +|:-----:|:----------:|:------------:| +| ALM-2 | MINOR | true | +| ALM-9 | MAJOR | true | + +The *MINOR* alarm is also acknowledged by user. ALARM_STATS reads: Major as 0, Minor as 0. So it is also taken out of consideration for system LED. System LED is Green. + +| alarm | severity | acknowledged | +|:-----:|:----------:|:------------:| +| ALM-2 | MINOR | true | +| ALM-9 | MAJOR | false | + +The *MAJOR* alarm is also unacknowledged by user. ALARM_STATS reads: Major as 1, Minor as 0. So it is now considered for system LED. System LED becomes Red. + +### 3.1.4 Event Receivers +Supported NBIs are: syslog, REST and gNMI. + +#### 3.1.4.1 syslog +Logging API contains logic to take the event record, augment it with any static information, format the message and +send it to syslog. +``` + if (ev_act.empty()) { + const char LOG_FORMAT[] = "[%s], %%%s %s. %s"; + // event Type + // Event Name + // Static Desc + // Dynamic Desc + + // raise a syslog message + syslog(LOG_MAKEPRI(ev_sev, SYSLOG_FACILITY), LOG_FORMAT, + ev_type.c_str(), + ev_id.c_str(), ev_msg.c_str(), ev_static_msg.c_str()); + } else { + const char LOG_FORMAT[] = "[%s] (%s), %%%s %s. %s"; + // event Type + // event action + // Event Name + // Static Desc + // Dynamic Desc + // raise a syslog message + syslog(LOG_MAKEPRI(ev_sev, SYSLOG_FACILITY), LOG_FORMAT, + ev_type.c_str(), ev_act.c_str(), + ev_id.c_str(), ev_msg.c_str(), ev_static_msg.c_str()); + } +``` +An example of syslog message generated for an event raised when user selects a custom event profile. +``` +May 19 21:22:07.122786 2021 sonic WARNING eventd#eventd[2419]: [EVENT], %CUSTOM_EVPROFILE_CHANGE :- handle_custom_evprofile: Custom Event Profile myprofile.json is applied.. Custom Event Profile is selected by user. +``` +Syslog message for an alarm raised by a sensor: +``` +May 19 21:42:14.373410 2021 sonic ALERT eventd#eventd[2453]: [ALARM] (RAISE), %TEMPERATURE_EXCEEDED :- temperatureCrossedThreshold: Current temperature of sensor/2 is 76 degrees. Temperature threshold is 75 degrees. +``` +Syslog message when alarm is clared is as follows: +``` +May 19 21:46:34.373693 2021 sonic ALERT eventd#eventd[2453]: [ALARM] (CLEAR), %TEMPERATURE_EXCEEDED :- temperatureCrossedThreshold: Current temperature of sensor/2 is 70 degrees. Temperature threshold is 75 degrees. +``` +Syslog message when alarm with id=4 is acknowledged is as follows: +``` +May 19 21:48:05.870530 2021 sonic ALERT eventd#eventd[2453]: [ALARM] (ACKNOWLEDGE), Alarm id 4 ACKNOWLEDGE. +``` + +Syslog message when alarm with id=4 is unacknowledged is as follows: +``` +May 19 21:53:24.490545 2021 sonic ALERT eventd#eventd[2453]: [ALARM] (UNACKNOWLEDGE), Alarm id 4 UNACKNOWLEDGE. +``` +Operator can configure specifc syslog host to receive either syslog messages corresponding to events or general log messages. +Through CLI, operator can chose 'logging server [log|event]' command. +When operator configures a host with 'event' type, it receives *only* log messages corresponding to events. +Support for VRF/source-interface/UDP port are all are applicable for 'event' type. + +#### 3.1.4.2 REST +Subcribing through REST to receive event notifications is currently being evaluated. + +#### 3.1.4.3 gNMI +gNMI clients can subscribe to receive event notifications. Subscribed gNMI clients receive event fields as in the DB and +there is no customization of these fileds similar to syslog messages. + +TODO: add definitions of protobuf spec + +#### 3.1.4.4 System LED +The original requirement was to change LED based on severities of the events. But on most of the platforms the system/power/fan LEDs are managed by the BMC. +BMC (baseboard management controller) is an embedded system that manages various platform elements like fan, PSU, temperature sensors. +There is an API that can be invoked to control LED, but not all platforms will support that API if they are fully controlled by the BMC. +So, on certain platforms, system LED could not represent events on the system. + +Another issue is: Currently pmon controls LED, and as eventd now tries to change the very same LED, which leads to conflicts. +A mechanism must exist for one of these to be master, which, in this case, is pmon. + +The proposed solution is to have pmon use ALAMR_STATS counters in conjunction with existing logic to update system LED. + +#### 3.1.4.5 Event/Alarm flooding +There are scenarios when system enters a loop of a fault condition that makes application trigger events continuously. To avoid such +instances flood the EVENT or ALARM tables, eventd maintains a cache of last event/alarm. Every new event/alarm is compared against this cache entry +to make sure it is not a flood. If it is found to be same event/alarm, the newly raised entry will be silently discarded. + +#### 3.1.4.6 Eventd continuous restart +Under the scenarios when eventd runs into an issue and restarts continuously, applications might keep writing to the eventpubsub table. As consumer - eventd - is not able to remove events from the pusbsub table, eventpusbub table could grow forever as applications keep rising events/alarms. +One way to fix is to have the system monitor daemon to periodically (very high polling interval) to check the number of keys in the table and if it exceeds a number, delete all the entries. When system monitor daemon does this, it logs a syslog message. + +### 3.1.5 Event Profile +The Event profile contains mapping between event-id and severity of the event, enable flag. +Through event profile, operator can change severity of a particular event. And can also enable/disable +a particular event. + +The default profile exists at */etc/evprofile/default.json* +By default, every event is enabled. +The severity of event is decided by developer while adding the event. +``` +{ + "__README__" : "This is default map of events that eventd uses. Developer can modify this file and send + SIGINT to eventd to make it read and use the updated file. Alternatively developer can test + the new event by adding it to a custom event profile and use 'event profile ' command + to apply that profile without sending SIGINT to eventd. Developer need to commit default.json file + with the new event after testing it out. + Supported severities are: CRITICAL, MAJOR, MINOR, WARNING and INFORMATIONAL. + Supported enable flag values are: true and false.", + "events":[ + { + "name" : "CUSTOM_EVPROFILE_CHANGE", + "severity" : "INFORMATIONAL", + "enable" : "true", + "message" : "Custom Event Profile is applied." + }, + { + "name": "TEMPERATURE_EXCEEDED", + "severity": "CRITICAL", + "enable": "true" + "message" : "Temperature threshold is 75 degrees." + } + ] +} +``` +User can download the default event profile to a remote host. User can modify characteristics of +some/all events in the profile and can upload it back to the switch and place the file at /etc/evprofile/. + +The uploaded profile will be called custom event profile. + +An example of custom event profile is as below. +With this particular custom event profile, user wants to +- change severity of CUSTOM_EVPROFILE_CHANGE event (severity changed from INFORMATIONAL to MAJOR) +- suppress the TEMPERATURE_EXCEEDED alarm (enable flag is changed from true to false) +- introduce new alarm by name DUMMY_ALARM (there should be an application to raise/clear this new alarm). +``` +{ + "events": [ + { + "name" : "CUSTOM_EVPROFILE_CHANGE", + "severity" : "MAJOR", + "enable" : "true", + }, + { + "name": "TEMPERATURE_EXCEEDED", + "severity": "CRITICAL", + "enable": "false" + }, + { + "name" : "DUMMY_ALARM", + "severity" : "WARNING", + "enable" : "true", + } + ] +} +``` + +User can have multiple custom profiles and can select any of the profiles under /etc/evprofile/ using 'event profile' command. + +The framework will sanity check the user selected profile and merges it map of events *static_event_map* maintained by eventd. + +After a successful sanity check, the framework generates an event indicating that a new profile is in effect. + +If there are any outstanding alarms in the current alarm table, the framework removes those records for which enable is set to false in the new profile. +Severity counters in ALARM_STATS are reduced accordingly. + +Eventd starts using the merged map of characteristics for the all the newly generated events. A CUSTOM_EVPROFILE_CHANGE event is generated. + +The event profile is upgrade and downgrade compatible by accepting only those attributes that are *known* to eventd. +All the other attributes will remain to their default values. + +Sanity check rejects the profile if attributes contains values that are not known to eventd. + +Config Migration hooks will be used to persist custom profiles across an upgrade. + +The profile can also be applied through ztp. + +### 3.1.6 CLI +The show CLI require many filters with range specifiers. +Various filters are supported using RPC. + +e.g. +``` +rpc getEventBySeqeuenceId{ +input { + from sequence-id; + to sequence-id; + } +output { + list event-table-entries; +} +``` + +The rpc callback needs to access DB with the given set of sequence ids. + +The gNMI server (gnoi_client.go, gnoi.go, sonic_proto, transl_utils.go) need to be extended to support the RPC to support similar operations for gNMI. + +### 3.1.7 Event History Table and Current Alarm Table +The Event History Table (EVENT) and Current Alarm List Table (ALARM) stored in EVENT_DB. +The size of Event History Table is 40k records or 30 days worth of events which ever hits earlier. +A manifest file will be created with parameters to specify the number and number of days limits for +eventd to read and enforce them. + +``` +root@sonic:/etc# cat eventd.json +{ + "config" : { + "no-of-records": 40000, + "no-of-days": 30 + } +} +``` +'no-of-records' indicates maximum number of records EVENT table can hold. The range is 1-40000. +'no-of-days' indicates maximum number of days an event can exist in the EVENT table. The range is 1-30. + +When either of the limit is reached, the framework wraps around the table by discarding older records. + +User can send SIGINT to eventd process to force read and apply the manifest limits. + +An example of an event in EVENT table. +``` +EVENT Table: +============================== + +Key : id + +id : Unique sequential ID generated by the system for every event {uint64} +type-id : Name of the event generated {string} +text : Dynamic message describing the cause for the event {string} +time-created : Time stamp at which the event is generated {uint64} +action : Indicates action of the event; for one-shot events, it is empty. For alarms it could be raise, clear or acknowledge {enum} +resource : Object which generated the event {string} +severity : Severity of the event {string} + +127.0.0.1:6379[6]> hgetall "EVENT|1" + 1) "text" + 2) ":- handle_custom_evprofile: Custom Event Profile x.json is applied." + 3) "type-id" + 4) "CUSTOM_EVPROFILE_CHANGE" + 5) "id" + 6) "1" + 7) "time-created" + 8) "1621459327118629520" + 9) "resource" +10) "/etc/evprofile/x.json" +11) "severity" +12) "WARNING" +127.0.0.1:6379[6]> +``` + +Schema for EVENT_STATS table is as follows: +``` +EVENT_STATS Table: +============================== + +Key : id + +id : key {state} +events : Total events raised {uint64} +raised : Total alarms raised {uint64} +cleared : Total alarms cleared {uint64} +acked : Total alarms acknowledged {uint64} + +127.0.0.1:6379[6]> hgetall "EVENT_STATS|state" +1) "events" +2) "1" +3) "raised" +4) "0" +5) "cleared" +6) "0" +7) "acked" +8) "0" +127.0.0.1:6379[6]> +``` +Current Alarm Table will not have any limits as it only contains the snapshot of the alarms during the current run. + +Contents of an alarm record. In this case, the alarm was raised temperature crossed a threshold. +``` +ALARM Table: +============================== + +Key : id + +id : Unique sequential ID generated by the system for every event {uint64} +type-id : Name of the event generated {string} +text : Dynamic message describing the cause for the event {string} +time-created : Time stamp at which the event is generated {uint64} +acknowledged : Indicates if alarm has been acknowledged {boolean} +resource : Object which generated the event {string} +severity : Severity of the event {string} +acknowledged : Indicates when alarm has been acknowledged/unacknowledged {uint64} + +127.0.0.1:6379[6]> hgetall "ALARM|2" + 1) "type-id" + 2) "TEMPERATURE_EXCEEDED" + 3) "text" + 4) "temperatureCrossedThreshold: Current temperature for sensor/2 is 76 degrees" + 5) "action" + 6) "RAISE" + 7) "resource" + 8) "sensor/2" + 9) "time-created" +10) "1621460371062299951" +11) "severity" +12) "CRITICAL" +13) "id" +14) "2" +15) "acknowledged" +16) "false" +127.0.0.1:6379[6] +``` + +Schema for ALARM_STATS table is as below. When an alarm of particular severity is cleared, +the corresponding severity counter is decremented. +``` +ALARM_STATS Table: +============================== + +Key : id + +id : key {state} +alarms : Number of active alarms {uint64} +critical : Number of alarms of severity 'critical' {uint64} +major : Number of alarms of severity 'major' {uint64} +minor : Number of alarms of severity 'minor' {uint64} +warning : Number of alarms of severity 'warning' {uint64} +informational : Number of alarms of severity 'informational' {uint64} + +127.0.0.1:6379[6]> hgetall "ALARM_STATS|state" + 1) "alarms" + 2) "1" + 3) "critical" + 4) "1" + 5) "major" + 6) "0" + 7) "minor" + 8) "0" + 9) "warning" +10) "0" + +``` +### 3.1.8 Pull Model +All NBIs - CLI, REST and gNMI - can pull contents of current alarm table and event history table. +The following filters are supported: +- ALL ( pulls all alarms) +- Severity. +- Recent alarms (eg., last 5 minutes, one hour, one day). +- Records between two timestamps, one timestamp and end, and beginning and a timestamp. +- All records between two Sequence Numbers (incl end and begin) + +### 3.1.9 Supporting third party containers +To support third party components ( e.g. FRR, teamd, DHCP Relay, LLDPd, ntpd etc ) which can not be modified to raise events, the following options are considered +and are being evaluated. +1. Patch the components + Create a patch for these components by adding libeventnotify library and invoke the API. This however, requires these patches need to be maintained in the code forever. + +2. Listen to syslog messages + As many of these components raises syslog messages on an important event, a listener can be implemented to read incoming syslog messages and raise + events based on the message. + This however is heavy on performance due to the fact that listener has to parse each syslog message. Also listener need to maintain a map of messages to + event-id and need to be aware of resource and other specific details. It need to be aware of nuances of alarm raising/clearing if the component follows + any specific logic. + +Approach 1 is preferred. + +## 3.2 DB Changes +### 3.2.1 EVENT DB +A new instance, redis4, is created and EVENT DB uses the new instance. +The following tables uses Event DB. +Table EVENTPUBSUB is used for applications to write events and for eventd to access and process them. +Event History Table (EVENT) and Current Alarm Table (ALARM) are used to house events and alarms respectively. +To maintain various statistics of events, these two tables are used : EVENT_STATS and ALARM_STATS. + +EVPROFILE table is used by mgmt-framework to communicate name of the custom event profile when configured through NBI. +Eventd reads the file name from this table and merges it with its static_event_map. + +## 3.3 User Interface +### 3.3.1 Data Models + +The following is SONiC yang for events. +``` +module: sonic-event + +--rw sonic-event + +--rw EVENT + | +--rw EVENT_LIST* [id] + | +--rw id uint64 + | +--rw resource? string + | +--rw text? string + | +--rw time-created? timeticks64 + | +--rw type-id? string + | +--rw severity? severity-type + | +--rw action? action-type + +--rw EVENT_STATS + +--rw EVENT_STATS_LIST* [id] + +--rw id enumeration + +--rw events? uint64 + +--rw raised? uint64 + +--rw acked? uint64 + +--rw cleared? uint64 + + rpcs: + +---x show-events + +---w input + | +---w (option)? + | +--:(time) + | | +---w time + | | +---w begin? yang-types:date-and-time + | | +---w end? yang-types:date-and-time + | +--:(last-interval) + | | +---w interval? enumeration + | +--:(severity) + | | +---w severity? severity-type + | +--:(id) + | +---w id + | +---w begin? string + | +---w end? string + +--ro output + +--ro status? int32 + +--ro status-detail? string + +--ro EVENT + +--ro EVENT_LIST* [id] + +--ro id uint64 + +--ro resource? string + +--ro text? string + +--ro time-created? timeticks64 + +--ro type-id? string + +--ro severity? severity-type + +--ro action? action-type +``` + +The following is SONiC yang for alarms. +``` +module: sonic-alarm + +--rw sonic-alarm + +--rw ALARM + | +--rw ALARM_LIST* [id] + | +--rw id uint64 + | +--rw resource? string + | +--rw text? string + | +--rw time-created? event:timeticks64 + | +--rw type-id? string + | +--rw severity? event:severity-type + | +--rw acknowledged? boolean + | +--rw acknowledge-time? event:timeticks64 + +--rw ALARM_STATS + +--rw ALARM_STATS_LIST* [id] + +--rw id enumeration + +--rw alarms? uint64 + +--rw critical? uint64 + +--rw major? uint64 + +--rw minor? uint64 + +--rw warning? uint64 + +--rw acknowledged? uint64 + + rpcs: + +---x acknowledge-alarms + | +---w input + | | +---w id* string + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string + +---x unacknowledge-alarms + | +---w input + | | +---w id* string + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string + +---x show-alarms + +---w input + | +---w (option)? + | +--:(time) + | | +---w time + | | +---w begin? yang-types:date-and-time + | | +---w end? yang-types:date-and-time + | +--:(last-interval) + | | +---w interval? enumeration + | +--:(severity) + | | +---w severity? event:severity-type + | +--:(id) + | +---w id + | +---w begin? string + | +---w end? string + +--ro output + +--ro status? int32 + +--ro status-detail? string + +--ro ALARM + +--ro ALARM_LIST* [id] + +--ro id uint64 + +--ro resource? string + +--ro text? string + +--ro time-created? event:timeticks64 + +--ro type-id? string + +--ro severity? event:severity-type + +--ro acknowledged? boolean + +--ro acknowledge-time? event:timeticks64 +``` + +Following is for sonic yang to support event profiles. +``` +module: sonic-evprofile + + rpcs: + +---x get-evprofile + | +--ro output + | +--ro file-name? string + | +--ro file-list* string + +---x set-evprofile + +---w input + | +---w file-name? string + +--ro output + +--ro status? string +``` + +openconfig alarms yang is defined at [here](https://github.com/openconfig/public/blob/master/release/models/system/openconfig-alarms.yang) + +### 3.3.2 CLI +#### 3.3.2.1 Exec Commands +``` +sonic# alarm acknowledge +``` +An operator can acknolwedge a raised alarm. This indicates that the operator is aware of the fault condition and considers the condition not catastrophic. +Acknowledging an alarm updates alarm statistics and thereby applications like pmon can remove the particular alarm from status consideration. + +The alarm record in the ALARM table is marked with acknowledged field set to true. There is acknowledge-time field that indicates when that alarm is acknowledged. + +``` +sonic# alarm unacknowledge +``` +An operator can un-acknolwedge a previously acknowledged raised alarm. +Un-acknowledging an alarm updates alarm statistics and thereby applications like pmon can take the particular alarm into status consideration. + +The alarm record in the ALARM table is marked with acknowledged field set to false. +There is acknowledge-time field that indicates when that alarm is un-acknowledged. + +``` +sonic# event profile +``` +The command takes name of specified file, validates it for its syntax and values; merges it with its internal static map of events *static_event_map*. + +``` +sonic# clear event history +``` +This command clears all the records in the event history table. All the event stats are cleared. +The command will not affect alarm table or alarm statistics. +Eventd generates an event informing that event history table is cleared. + +#### 3.3.2.2 Configuration Commands +``` +sonic(config)# logging server [log|event] +``` +Note: The 'logging server' command is an existing, already supported command. +It is only enhanced to take either 'log' or 'event' to indicate either native syslog messages or syslog messages corresponding to events alone are sent to the remote host. +Support with VRF/source-interface and configuring remote-port are all backward comaptible and will be applicable to either 'log' or 'event' options. + +#### 3.3.2.3 Show Commands +``` +sonic# show event profile +-------------------------- +Active Event Profile +-------------------------- +myProfile.json +-------------------------- +Available Event Profiles +-------------------------- +default.json +myProfile.json +userProfile.json + +sonic# show event [ details | summary | severity | start end | recent <5min|60min|24hr> | id | from to ] + +'show event' commands would display all the records in EVENT table. + +sonic# show event +---------------------------------------------------------------------------------------------------------------------------- +Id Action Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +1 - WARNING CUSTOM_EVPROFILE_CHANGE 2021-05-19T21:38:27.455Z :- handle_custom_evprofile: Custom Event Profile x.json is applied. +2 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:39:31.622Z :- signalHandler: Raising simulated alarm +3 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:42:34.371Z :- signalHandler: Clearing simulated alarm +4 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:46:14.371Z :- signalHandler: Raising simulated alarm +5 ACKNOWLEDGE CRITICAL DUMMY_ALARM 2021-05-19T21:48:05.845Z Alarm id 4 ACKNOWLEDGE. +6 UNACKNOWLEDGE CRITICAL DUMMY_ALARM 2021-05-19T21:53:24.484Z Alarm id 4 UNACKNOWLEDGE. +7 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:55:54.977Z :- signalHandler: Clearing simulated alarm + +sonic# show event details +---------------------------------------------- +Event Details - 1 +---------------------------------------------- +Id: 1 +Action: - +Severity: WARNING +Type: CUSTOM_EVPROFILE_CHANGE +Timestamp 2021-05-19T21:38:27.455Z +Description: :- handle_custom_evprofile: Custom Event Profile x.json is applied. +Source: /etc/evprofile/x.json + +---------------------------------------------- +Event Details - 2 +---------------------------------------------- +Id: 2 +Action: RAISE +Severity: CRITICAL +Type: DUMMY_ALARM +Timestamp 2021-05-19T21:39:31.622Z +Description: :- signalHandler: Raising simulated alarm +Source: simulation + +---------------------------------------------- +Event Details - 3 +---------------------------------------------- +Id: 3 +Action: CLEAR +Severity: CRITICAL +Type: DUMMY_ALARM +Timestamp 2021-05-19T21:42:34.371Z +Description: :- signalHandler: Clearing simulated alarm +Source: simulation + +sonic# show event summary +Event summary +--------------------------------- +Total: 14 +Raised: 4 +Acknowledged: 1 +Cleared: 3 +---------------------------------- + +sonic# show event severity critical +---------------------------------------------------------------------------------------------------------------------------- +Id Action Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +2 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:39:31.622Z :- signalHandler: Raising simulated alarm +3 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:42:34.371Z :- signalHandler: Clearing simulated alarm +4 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:46:14.371Z :- signalHandler: Raising simulated alarm +5 ACKNOWLEDGE CRITICAL DUMMY_ALARM 2021-05-19T21:48:05.845Z Alarm id 4 ACKNOWLEDGE. +6 UNACKNOWLEDGE CRITICAL DUMMY_ALARM 2021-05-19T21:53:24.484Z Alarm id 4 UNACKNOWLEDGE. +7 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:55:54.977Z :- signalHandler: Clearing simulated alarm + +sonic# show event recent 24hr +---------------------------------------------------------------------------------------------------------------------------- +Id Action Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +2 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:39:31.622Z :- signalHandler: Raising simulated alarm +3 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:42:34.371Z :- signalHandler: Clearing simulated alarm +4 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:46:14.371Z :- signalHandler: Raising simulated alarm +5 ACKNOWLEDGE CRITICAL DUMMY_ALARM 2021-05-19T21:48:05.845Z Alarm id 4 ACKNOWLEDGE. +6 UNACKNOWLEDGE CRITICAL DUMMY_ALARM 2021-05-19T21:53:24.484Z Alarm id 4 UNACKNOWLEDGE. +7 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:55:54.977Z :- signalHandler: Clearing simulated alarm + +sonic# show event id 2 +---------------------------------------------- +Event Details - 2 +---------------------------------------------- +Id: 2 +Action: RAISE +Severity: CRITICAL +Type: DUMMY_ALARM +Timestamp 2021-05-19T21:39:31.622Z +Description: :- signalHandler: Raising simulated alarm +Source: simulation + +sonic# show event from 2 to 5 +---------------------------------------------------------------------------------------------------------------------------- +Id Action Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +2 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:39:31.622Z :- signalHandler: Raising simulated alarm +3 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:42:34.371Z :- signalHandler: Clearing simulated alarm +4 RAISE CRITICAL DUMMY_ALARM 2021-05-19T21:46:14.371Z :- signalHandler: Raising simulated alarm +5 ACKNOWLEDGE CRITICAL DUMMY_ALARM 2021-05-19T21:48:05.845Z Alarm id 4 ACKNOWLEDGE. + +sonic# show event start 2021-05-19T21:39:31.622Z end 2021-05-19T21:46:14.371Z +---------------------------------------------------------------------------------------------------------------------------- +Id Action Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +3 CLEAR CRITICAL DUMMY_ALARM 2021-05-19T21:42:34.371Z :- signalHandler: Clearing simulated alarm + +sonic# show alarm [ acknowledged | all | detail | summary | severity | id | start end | recent <5min|1hr|1day> | from to ] + +'show alarm' command would display all the *active* alarm records in ALARM table. Acknowledged alarms wont be shown here. + +sonic# show alarm +---------------------------------------------------------------------------------------------------------------------------- +Id Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +14 WARNING TEMPERATURE_EXCEEDED 2021-05-20T00:47:52.992Z :- temperatureCrossedThreshold: Current temperature of sensor/2 is 76 degrees +16 WARNING PSU_FAULT 2021-05-20T02:16:42.611Z :- /psu/2 has experienced a fault + +sonic# show alarm all +---------------------------------------------------------------------------------------------------------------------------- +Id Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +14 WARNING TEMPERATURE_EXCEEDED 2021-05-20T00:47:52.992Z :- temperatureCrossedThreshold: Current temperature of sensor/2 is 76 degrees +15 WARNING DUMMY_ALARM 2021-05-20T02:16:41.637Z :- signalHandler: Raising simulated alarm +16 WARNING PSU_FAULT 2021-05-20T02:16:42.611Z :- /psu/2 has experienced a fault + + +sonic# show alarm detail + +alarm details - 14 +------------------------------------------- +Id: 14 +Severity: CRITICAL +Source: /sensor/2 +Name: TEMPERATURE_EXCEEDED +Description: temperatureCrossedThreshold: Current temperature of sensor/2 is 76 degrees +Raise-time: Wed Feb 10 18:08:24 2021 +Ack-time: +New: true +Acknowledged: false + +sonic# show alarm from 14 to 16 +---------------------------------------------------------------------------------------------------------------------------- +Id Severity Name Timestamp Description +---------------------------------------------------------------------------------------------------------------------------- +14 WARNING TEMPERATURE_EXCEEDED 2021-05-20T00:47:52.992Z :- temperatureCrossedThreshold: Current temperature of sensor/2 is 76 degrees +15 WARNING DUMMY_ALARM 2021-05-20T02:16:41.637Z :- signalHandler: Raising simulated alarm +16 WARNING PSU_FAULT 2021-05-20T02:16:42.611Z :- /psu/2 has experienced a fault + +sonic# show alarm summary +Alarm summary +--------------------------------- +Total: 3 +Critical: 0 +Major: 0 +Minor: 0 +Warning: 3 +Acnowledged: 2 +---------------------------------- +``` + +### 3.3.3 REST API Support + +sonic REST links: +* /restconf/data/sonic-event:sonic-event/EVENT/EVENT_LIST +* /restconf/data/sonic-event:sonic-event/EVENT_STATS/EVENT_STATS_LIST +* /restconf/data/sonic-alarm:sonic-alarm/ALARM/ALARM_LIST +* /restconf/data/sonic-alarm:sonic-alarm/ALARM_STATS/ALARM_STATS_LIST +* /restconf/operations/sonic-evprofile:get-evprofile +* /restconf/operations/sonic-evprofile:set-evprofile +* /restconf/operations/sonic-alarm:acknowledge-alarms +* /restconf/operations/sonic-alarm:unacknowledge-alarms + +openconfig REST links: +* /restconf/data/openconfig-system:system/openconfig-events:events +* /restconf/data/openconfig-system:system/openconfig-events:event-stats +* /restconf/data/openconfig-system:system/alarms +* /restconf/data/openconfig-system:system/openconfig-alarms-ext:alarm-stats + +# 4 Flow Diagrams +![Sequence Diagram](event-alarm-framework-seqdiag.png) + +# 5 Warm Boot Support +## 5.1 Application warm boot +Applications confirming to the warm boot, should have stored their state and compare current values against previous values. +Such compliant application also "remembers" that it raised an event before for a specific condition. +They would +* not raise alarms/events for the same condition that it raised pre warm boot +* clear those alarms once current state of a particular condition is recovered (by comparing against the stored state). + +## 5.2 eventd warm boot +Records from applications are stored in a table, called EVENTPUBSUB. +Records that are being written will be queued when the consumer (eventd) is down. + +During normal operation, eventd reads, processes whenever a new record is added to the table. + +When eventd is restarted, events and alarms raised by applications will be waiting in a queue while eventd is coming up. +When eventd eventually comes back up, it reads those records in the queue. + +# 6 Scalability +In this feature, scalability applies to Event History Table (EVENT). As it is persistent and it records every event generated on the system, to protect +against it growing indefinitely, user can limit its size through a manifest file. +By default, the size of Event History Table is set to 40k events or events for 30 days - after which, older records are discarded to make way for new records. + +# 7 Showtech support +The techsupport bundle is upgraded to include output of "show event recent 60min” and “show alarm all”. +The first command displays all the events that were sent by applications for the last one hour. +The second command displays all the alarms that are waiting to be cleared by applications (this includes alarms that were acknowledged by operator as well). + +# 8 Unit Test +- Raise an event and verify the fields in EVENT table and EVENT_STATS table +- Raise an alarm and verify the fields in ALARM table and ALARM_STATS table +- Clear an alarm and verify that record is removed from ALARM and ALARM_STATS tables are udpated +- Ack an alarm and verify that acknowledged flag is set to true in ALARM table and acknowledge-time is set +- Un-Ack an alarm and verify that acknowledged flag is set to false in ALARM table and acknowledge-time is set +- Verify wrap around for EVENT table ( change manifest file to a lower range and trigger that many events ) +- Verify sequence-id for events is persistent by restarting +- Verify counters by raising various alarms with different severities +- Change severity of an event through custom event profile and verify it is logged at specified severity +- Change enable/disable of an event through custom event profile and verify it is suppressed +- Verify custom event profile with an invalid severity is rejected +- Verify custom event profile with an invalid enable/disable flag is rejected +- Verify custom event profile is persisted after a reboot +- Verify various show commands +- Verify 'logging-server event' command forwards only event log messages to the host diff --git a/doc/filemanagement/apps_SONiC_FileSystem_HLD.md b/doc/filemanagement/apps_SONiC_FileSystem_HLD.md new file mode 100644 index 0000000000..646b0263f3 --- /dev/null +++ b/doc/filemanagement/apps_SONiC_FileSystem_HLD.md @@ -0,0 +1,359 @@ +# SONiC Filesystem support design spec draft + +Table of Contents + + +- [SONiC Filesystem support design spec draft](##SONiC-Filesystem-support-design-spec-draft) + - [Scope](#Scope) + - [Document History](#Document-History) + - [Abbreviations](#Abbreviations) + - [SONiC Filesystem Feature Requirement](#SONiC-Filesystem-Feature-Requirement) + - [Functionality](#Functionality) + - [Target Deployment Use Cases](#Target-Deployment-Use-Cases) + - [Functional Description](#Functional-Description) + - [Dependencies](#Dependencies) + - [SONiC system diagram for Filesystem](#SONiC-system-diagram-for-filesystem) + - [The schema changes](#The-schema-changes) + - [Event flow diagram](#Event-flow-diagram) + - [Modules changes](#Modules-changes) + - [CLI](#CLI) + - [Other Linux utilities](#Other-Linux-utilities) + - [User scenarios](#User-scenarios) + - [Test plan](#Test-plan) + + +## Scope +This document describes the high-level design for the filesystem commands. + +## Document History +| Version | Date | Author | Description | +|---------|------------|--------------|--------------------------------------------------| +| v.01 | 08/04/2021 | Reemus Vincent, Syed Obaid Amin | Initial version | + + +## Abbreviations +| **Term** | **Definition** | +|----------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| URL | Uniform Resource Locator | + +## SONiC Filesystem Feature Requirement +1. Add copy command to copy files from/to remote servers +2. Add dir command to list the files in the local storage +3. Add delete command to remove file from the local storage + +### Functionality + +#### Target Deployment Use Cases + +The SONiC filesystem commands allow users to manage files in the SONiC Operating system. It provides several commands to help users manage configuration, support bundle, event-profile and core-dump files hence only folders containing these files and home folder are exposed to the users. + +![image info](images/router-with-servers.jpg "Figure 1: Filesystem Deployment use case") +__Figure 1: Filesystem Deployment use case__ + +#### Functional Description +With this feature support, users will be able to: + 1. Take backup of System Configurations - both running and startup to any remote server + 2. Restore System Configuration from any remote server + 3. Copy Coredump files to any remote server + 4. Copy Support Bundle Files to any remote server + 5. Copy Event Profile from any remote server or local storage + 6. Copy Log Files to any remote server or local storage + +## Dependencies +The Filesystem command depends on the following: + 1. D-Bus Service - reliable service communication channel between the Mgmt Framework Container and the Host Services (Host) + 2. "curl" utility for downloading files from remote server. + +## SONiC system diagram for Filesystem +![image info](images/SONIC-Host-Service.jpg "Figure 3: Figure 2: SONiC Host Service") +__Figure 2: SONiC Host Service__ + +## The schema changes +Not applicable + +## Event flow diagram +![image info](images/Filesystem-Eventflow.jpg "Figure 3: Figure 3: Event Flow Diagram") +__Figure 3: Event Flow Diagram__ + +## User Context +All the filesystem commands will have user context passed down from Xfmr to the Host Service. The "curl" utility will be run in the context of the user passed down to the Host Service. +The "home:" shortcut will be expanded to the appropriate home directory(/home/$USERNAME). The user will be able execute to dir/copy/delete filesystem commands on the the appropritate files/directories according to the user privilage and group. + +All the "admin" role users will be added to "adm" group by default. + +## Module changes +* sonic-mgmt-common + * YANG Model + * Transformer +* sonic-mgmt-framework + * CLI XML + * Actioner +* sonic-host-service + * Host Service + +## CLI + +### copy command +To copy a file from a source to a destination, use the copy . +The exact format of the source and destination URLs varies according to the file or directory location. +Users may enter either a alias keyword for a particular file or a alias keyword for a file system type. + +**Syntax:** +``` + copy + +``` +Common Keyword Aliases for local files + +| **Alias Name** | **Description** | +|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| startup-config | This alias represents the configuration file used during initialization (start up). | +| running-config | This alias represents the current running configuration file. | +| config: | This alias represents the system config directory. | +| home: | This alias represents the $HOME directory of the current user. | +| coredump: | This alias represents the Coredump directory. | +| supportbundle: | This alias represents the Support Bundle directory. | +| logs: | This alias represents the log directory. | +| event-profile: | This alias represents the Event profile directory. | + + +URL Prefixes for Remote Files + +| **Alias Name** | **Format** | **Description** | +|----------------|------------------------------------------------------------------------|-------------------------------------------------------------------------------| +| ftp: | ftp://[username [:password]@]{hostname\|host-ip}/directory/[filename] | Source or destination URL of an FTP server. | +| http: | http://[username [:password]@]{hostname\|host-ip}/directory/[filename] | Source or destination URL of an HTTP server. | +| https: | http://[username [:password]@]{hostname\|host-ip}/directory/[filename] | Source or destination URL of an HTTPS server. | +| scp: | scp://[username [:password]@]{hostname\|host-ip}/directory/[filename] | Source or destination URL of an Secure Shell(SSH) server. | + + +**Note:** +Either the source or the destination should be a local file. + +**Example** +``` +sonic# copy ftp://admin:admin@10.10.10.10/new-startup-config.cfg startup-config +``` + +### dir command +List the files stored in available directories. The Dir command will list the files in the Filename-sorted order like Linux. + +**Syntax:** +``` + dir [home: | coredump: | supportbundle: | logs: | config: | event-profile:] +``` + +### delete command +Remove or delete the file from the system. + +**Syntax:** +``` + delete [home:// | coredump:// | supportbundle:// | logs:// | config:// | event-profile://] +``` + +### Yang Model + +**Copy Command** +```txt + rpcs: + +---x copy + | +---w input + | | +---w source? filename-uri-type + | | +---w overwrite? boolean + | | +---w destination? filename-uri-type + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string +``` + +**Dir Command** +```txt + rpcs: + +--ro sonic-file-system-info + | +--ro system-folder* [name filename] + | | +--ro name folder-name + | | +--ro filename string + | | +--ro folder-match? string + | | +--ro file-size? uint32 + | | +--ro last-modified-date? yang:date-and-time +``` + +**Delete Command** +```txt + rpcs: + +---x delete-file + | +---w input + | | +---w filename? delete-filepath-type + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string +``` + +## Other Linux utilities +The filesystem will be using the "curl" utility to download and upload the files for remote files(ftp, http, scp & https). + +## User scenarios + +**User Scenario: 1** +```txt +! Backup Startup Configuration locally +sonic# copy startup-config home://config-backup-AUG0821 +!!! copied successfully +sonic# +``` + +**User Scenario: 2** +```txt +! Backup Running Configuration locally +sonic# copy running-config home://config-backup-AUG0821 +!!! copied successfully +sonic# +``` + +**User Scenario: 3** +```txt +! Backup Startup Configuration to a Remote FTP Server +sonic# copy startup-config ftp://admin:admin@10.10.10.10/startup-config-bkup +!!! copied successfully +sonic# +``` + +**User Scenario: 4** +```txt +! Backup Running Configuration to a Remote FTP Server +sonic# copy running-config ftp://admin:admin@10.10.10.10/config-bkup +!!! copied successfully +sonic# +``` + +**User Scenario: 5** +```txt +! Backup Running Configuration to a Remote SSH Server +sonic# copy running-config scp://admin:admin@10.10.10.10/config-bkup +!!! copied successfully +sonic# +``` + +**User Scenario: 6** +```txt +! Restore Running Configuration from a HTTP Server +sonic# copy http://inter128.acme.com/config-bkup running-config +!!! copied successfully +sonic# +``` + +**User Scenario: 7** +```txt +! Copy a Sonic Coredump file to FTP Server +sonic# copy coredump://sonic_dump_sonic_20210717_202813.tar.gz ftp://admin:admin@10.10.10.10/sonic_dump_sonic_20210717_202813.tar.gz +!!! copied successfully +sonic# +``` + +**User Scenario: 8** +```txt +! Display the contents of the config: +sonic# dir home:// +------------------------------------------------------------------------- +Date(Last Modified) Size(Bytes) Type Filename +------------------------------------------------------------------------- +2021-11-02 13:02 19144 - config_db.json +2021-11-02 13:01 30720 - running-1.tar +2021-11-02 13:02 30720 - running-2.tar +sonic# +``` + +**User Scenario: 9** +```txt +! Delete the config_backup.josn from home: +sonic# delete home://config_backup.josn +sonic# +``` + + +## Test plan +* copy Command Testcases + * source startup-config + * verify CLI: copy startup-config running-config + * verify CLI: copy startup-config home://...filename... + * verify CLI: copy startup-config http://...filename... + * verify CLI: copy startup-config ftp://...filename... + * verify CLI: copy startup-config scp://...filename... + * source running-config + * verify CLI: copy running-config startup-config + * verify CLI: copy running-config home://...filename... + * verify CLI: copy running-config http://...filename... + * verify CLI:copy running-config ftp://...filename... + * verify CLI:copy running-config scp://...filename... + * source home: + * verify CLI: copy home://...filename... startup-config + * verify CLI: copy home://...filename... running-config + * verify CLI: copy home://...filename-1... home://...filename-2... + * verify CLI: copy home://...filename... http://...filename... + * verify CLI: copy home://...filename... ftp://...filename... + * verify CLI: copy home://...filename... scp://...filename... + * source coredump: + * verify CLI: copy coredump://...filename... home://...filename... + * verify CLI: copy coredump://...filename... http://...filename... + * verify CLI: copy coredump://...filename... ftp://...filename... + * verify CLI: copy coredump://...filename... scp://...filename... + * source supportbundle: + * verify CLI: copy supportbundle://...filename... home://...filename... + * verify CLI: copy supportbundle://...filename... http://...filename... + * verify CLI: copy supportbundle://...filename... ftp://...filename... + * verify CLI: copy supportbundle://...filename... scp://...filename... + * source logs: + * verify CLI: copy logs://...filename... home://...filename... + * verify CLI: copy logs://...filename... http://...filename... + * verify CLI: copy logs://...filename... ftp://...filename... + * verify CLI: copy logs://...filename... scp://...filename... + * source event-profile: + * verify CLI: copy event-profile://...filename... home://...filename... + * verify CLI: copy event-profile://...filename... http://...filename... + * verify CLI: copy event-profile://...filename... ftp://...filename... + * verify CLI: copy event-profile://...filename... scp://...filename... + * source http: + * verify CLI: copy http://...filename... startup-config + * verify CLI: copy http://...filename... running-config + * verify CLI: copy http://...filename... home://...filename... + * verify CLI: copy http://...filename... event-profile://...filename... + * source ftp: + * verify CLI: copy ftp://...filename... startup-config + * verify CLI: copy ftp://...filename... running-config + * verify CLI: copy ftp://...filename... home://...filename... + * verify CLI: copy ftp://...filename... event-profile://...filename... + * source scp: + * verify CLI: copy scp://...filename... startup-config + * verify CLI: copy scp://...filename... running-config + * verify CLI: copy scp://...filename... home://...filename... + * verify CLI: copy scp://...filename... event-profile://...filename... + * Negative Testcases with source & destinations as Remote Servers + * Unreachable Remote Server + * Wrong Username for the Remote Server + * Wrong Password for the Remote Server + * Wrong Filename(not found) for the Remote Server + * Wrong Filename(not found) locally (home:, logs:, event-profile:) + * Managment VRF Testcases + * Test the Remote File Testcases with Management VRF configured + * Different Users + * Test the commands with admin users + * Test the commands with operator users + * Test the commands with Radius/TACACS users +* dir Command Testcases + * verify CLI: dir home: + * verify CLI: dir home://...dirname... + * verify CLI: dir config: + * verify CLI: dir logs: + * verify CLI: dir coredump: + * verify CLI: dir supportbundle: + * verify CLI: dir event-profile: +* delete Command Testcases + * verify CLI: delete home: + * verify CLI: delete home://...dirname... + * verify CLI: delete config: + * verify CLI: delete logs: + * verify CLI: delete coredump: + * verify CLI: delete supportbundle: + * verify CLI: delete event-profile: + + diff --git a/doc/filemanagement/images/Filesystem-Eventflow.jpg b/doc/filemanagement/images/Filesystem-Eventflow.jpg new file mode 100644 index 0000000000..1603d992bb Binary files /dev/null and b/doc/filemanagement/images/Filesystem-Eventflow.jpg differ diff --git a/doc/filemanagement/images/SONIC-Host-Service.jpg b/doc/filemanagement/images/SONIC-Host-Service.jpg new file mode 100644 index 0000000000..01d9b21326 Binary files /dev/null and b/doc/filemanagement/images/SONIC-Host-Service.jpg differ diff --git a/doc/filemanagement/images/router-with-servers.jpg b/doc/filemanagement/images/router-with-servers.jpg new file mode 100644 index 0000000000..a8472fd170 Binary files /dev/null and b/doc/filemanagement/images/router-with-servers.jpg differ diff --git a/doc/ifa-hld.md b/doc/ifa-hld.md new file mode 100644 index 0000000000..b4e97db320 --- /dev/null +++ b/doc/ifa-hld.md @@ -0,0 +1,284 @@ +# Feature Name +Inband Flow Analyzer. +# High Level Design Document +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/30/2019 | Srinadh Penugonda| Initial version | + +# About this Manual +This document provides general information about the Inband Flow Analyzer feature implementation in SONiC. +# Scope +This document describes the north bound interface and unit tests for Inband Flow Analyzer feature. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| IFA | Inband Flow Analyzer | +| TAM | Telemetry and Monitoring | +# 1 Feature Overview + +Inband Flow Analyzer is a flexible packet and flow monitoring inband telemtry solution. The feature allows configuration of IFA sessions that provide Inband telemetry over sampled traffic to collectors. + +It provides mechanism to monitor and analyze when packets enter/exit the network, the path packets and flows take through the network, the rate at which packets arrive at each hop and how log packets spend at each hop etc., Out of band management technhiques can not measure such details. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide management framework support to existing SONiC capabilities with respect to IFA. + +1.1 IFA feature is accomplished by configuring IFA session on various nodes that act as ingress, +intermediate and egress devices. Device role is per flow in a node and a single node can act as ingress +device for one flow and intermediate device for another flow. + +2.0.0.1 TAM device identifier to uniquely identify a device in network and insert the same in INT header. +2.0.0.2 ACL configuration to identify a flow and sample packets from that flow to insert IFA headers. +2.0.0.3 TAM collector configuration that can be attached to IFA flow on egress device to forward telemetry +data. +3.0 UI commands available to configure TAM device identifier, TAM collector and IFA configuration. +3.1 UI commands available to show TAM device identifier, TAM collector, IFA configuration +3.2 UI commands available to clear IFA configuration +4.0 The maximum number of IFA 􀃖ows are platform dependent. +4.1 Only one collector can be con􀃕gured in a device. +4.5 Some platforms may require provisioning to enable IFA. 'ifa -config -enable' command to be issued to +provision such platforms for IFA functionality. 'ifa -config -disable' command can be issued to disable +provisioning of IFA on such platforms. + +### 1.1.2 Configuration and Management Requirements +1. CLI configuration/show support +2. REST API support +3. gNMI support + +## 1.2 Design Overview +### 1.2.1 Basic Approach + +As there is no opeconfig/ietf yang file exists for this feature, it is decided to go with sonic yang. + +### 1.2.2 Container +The changes are in sonic-management-framework container. + +There will be additional files added. +1. XML file for the CLI +2. Python script to handle CLI request (actioner) +3. Jinja template to render CLI output (renderer) +4. YANG models + sonic-ifa.yang + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Whenever operator want to track packet latency/congestion metrics. +## 2.2 Functional Description +The management UI provides an user friendly interface to configure Inband Flow Analyzer. +# 3 Design +## 3.1 Overview +The packet action field in acl rule indicates the type of IFA device: ingress and egress. 'int_insert' makes the device as ingress IAF device and 'int_delete' as egress IFA device. Ingress IAF device makes a sample of a flow and tags them for analysis and data collection. Egress device is responsible for terminating IFA flow by summarizing the telemtry data of the entire path and sending it to the collector. + +The device identifer uniquely identifies the device in the network and inserts the ID in the IFA header. + +Collectors receive the telemetry data from egress devices. + +Flow configuration will contain sampling rate at which rate traffic will be sampled. + +IFA feature can be enabled or disabled. +## 3.6 User Interface +### 3.6.1 Data Models + +https://github.com/project-arlo/sonic-mgmt-framework/blob/master/models/yang/sonic/sonic-ifa.yang + +module: sonic-ifa + +--rw sonic-ifa + +--rw TAM_INT_IFA_FEATURE_TABLE + | +--rw TAM_INT_IFA_FEATURE_TABLE_LIST* [name] + | +--rw name enumeration + | +--rw enable? boolean + +--rw TAM_INT_IFA_FLOW_TABLE + +--rw TAM_INT_IFA_FLOW_TABLE_LIST* [name] + +--rw name string + +--rw acl-table-name string + +--rw acl-rule-name string + +--rw sampling-rate? uint16 + +--rw collector-name? string + +https://github.com/project-arlo/sonic-mgmt-framework/blob/master/models/yang/sonic/sonic-tam.yang + +module: sonic-tam + +--rw sonic-tam + +--rw TAM_DEVICE_TABLE + | +--rw TAM_DEVICE_TABLE_LIST* [name] + | +--rw name enumeration + | +--rw deviceid? uint16 + +--rw TAM_COLLECTOR_TABLE + +--rw TAM_COLLECTOR_TABLE_LIST* [name] + +--rw name string + +--rw ipaddress-type? enumeration + +--rw ipaddress? inet:ip-address + +--rw port? inet:port-number + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands +1. Command : confg tam device-id +Attribute : +The command is used to configure TAM device identifier. + +2. Command : config tam collector +Attribute(s) : {collector-name} ip-type ip-addr
port +The command is used to configure TAM collector and IFA report will be forwarded to the collector. + +3. Command : config tam int-ifa feature +Attribute : +The command is used to enable or disable the IFA feature. + +4. Command : config tam int-ifa flow +Attribute(s) : acl-rule acl-table { sampling-rate collector } +The command is used to specify flow criteria to match against incoming flow and tag with IFA data. When sampling rate is specified, one packet will be sampled out of its value. When collector is specified, IFA report will be forwarded to it. + +5. Command : config-tam no device-id +The command is used to clear user configured device identifier. Default device identifier is used. + +6. Command : config-tam no collector +Attribute : +The command is used to delete previously configured collector information. + +7. Command : config-tam-int-ifa no flow +Attribute : +The command is used to delete previously configure flow information. + +#### 3.6.2.2 Show Commands +1. Command : show tam device +The command is used to show TAM device identifier. + +2. Command : show tam collector +Attribute : { | all } +The command is used to show TAM collector information. + +3. Command : show tam int-ifa flow +Attribute(s) : { | all } +The command is used to display configured IFA flow information. + +4. Command : show tam int-ifa statistics +Attribute(s) : { | all } +The command is used to display statisitcs of IFA flow. + +5. Command : show tam int-ifa status +The command is used to display status of TAM. + +#### 3.6.2.3 Debug Commands +N/A + +### 3.6.3 REST API Support + +1. Get Device Information +sonic-tam:sonic-tam/TAM_DEVICE_TABLE + +2. Get Collector information +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE + +3. Get particular collector information +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE/TAM_COLLECTOR_TABLE_LIST={name} + +4. Get IFA feature information +sonic-ifa:sonic-ifa/TAM_INT_IFA_FEATURE_TABLE + +5. Get IFA flow information +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE + +6. Get particular IFA flow information +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE/TAM_INT_IFA_FLOW_TABLE_LIST={name} + +7. Set device identifier +sonic-tam:sonic-tam/TAM_DEVICE_TABLE/TAM_DEVICE_TABLE_LIST={device}/deviceid +{ + "sonic-tam:deviceid": 0 +} + +8. Set TAM collector +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE/TAM_COLLECTOR_TABLE_LIST +{ + "sonic-tam:TAM_COLLECTOR_TABLE_LIST": [ + { + "name": "string", + "ipaddress-type": "ipv4", + "v4addr": "string", + "v6addr": "string", + "port": 0 + } + ] +} + +9. Set TAM INT IFA feature +sonic-ifa:sonic-ifa/TAM_INT_IFA_FEATURE_TABLE/TAM_INT_IFA_FEATURE_TABLE_LIST={feature}/enable +{ + "sonic-ifa:enable": true +} + +10. Set TAM INT IFA flow +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE/TAM_INT_IFA_FLOW_TABLE_LIST={name} +{ + "sonic-ifa:TAM_INT_IFA_FLOW_TABLE_LIST": [ + { + "name": "string", + "acl-table-name": "string", + "acl-rule-name": "string", + "sampling-rate": 0, + "collector": "string" + } + ] +} + +11. Delete TAM device identifier +sonic-tam:sonic-tam/TAM_DEVICE_TABLE/TAM_DEVICE_TABLE_LIST={device}/deviceid + +12. Delete TAM collector +sonic-tam:sonic-tam/TAM_COLLECTOR_TABLE/TAM_COLLECTOR_TABLE_LIST={name} + +13. Delete IFA flow +sonic-ifa:sonic-ifa/TAM_INT_IFA_FLOW_TABLE/TAM_INT_IFA_FLOW_TABLE_LIST={name} + + +# 4 Flow Diagrams +N/A +# 5 Error Handling +N/A +# 6 Serviceability and Debug +TBD + +# 7 Warm Boot Support +N/A +# 8 Scalability +N/A +# 9 Unit Test +| Test Name | Test Description | +| :------ | :----- | +| Create TAM DEvice Identifier | Verify device-id is configured. Verify a value with more than five digits will be rejected | +| Delete TAM device Identifier | Verify when device-id is deleted, it defaults to default value of 0 | +| Create TAM collector | Verify TAM collector is configured ( ip address validation is does by application ) | +| Delete TAM collector | Verify TAM collector can be deleted | +| Enable TAM INT IFA Feature | Verify user can enable/disable IFA feature | +| Create IFA Flow | Verify IFA flow is configured. Verify configuration fails when user uses invalid acl table/rule. Verify configuration fails when user uses invalid collector name | +| Delete IFA flow | Verify IFA flow is deleted | +| Show TAM Device | Verify configured device identifier is displayed with the show command | +| Show TAM collector | Verify all collectors are displayed with 'all' keyword. Verify specified collector is displayed with the name. Verify command fails when an invalid flow name is used to display | +| Show TAM INT IFA flow | Verify all flow information is displayed when used with 'all'. Verify particular flow is displayed when name is supplied. Verify command fails when an non-existent flow name is given | +| show tam int ifa supported | Verify feature status is correctly displayed | + + +# 10 Internal Design Information +N/A diff --git a/doc/in-band-mgmt/SONiC_in_band_mgmt_via_mgmt_Vrf_HLD.md b/doc/in-band-mgmt/SONiC_in_band_mgmt_via_mgmt_Vrf_HLD.md new file mode 100644 index 0000000000..287d3be254 --- /dev/null +++ b/doc/in-band-mgmt/SONiC_in_band_mgmt_via_mgmt_Vrf_HLD.md @@ -0,0 +1,57 @@ + +# Introduction + +The scope of this document is to provide the requirements and a high-level design proposal for in-band management via mgmt VRF. + +# Requirements + +The following are the high level requirements for the in-band management via mgmt VRF. + +1. Bind/Unbind L3 interface (e.g Phy/Port-channel/VLAN..etc) into mgmt VRF +2. Create/Delete mgmt VRF in HW to trap in-band management traffic to CPU +3. Add/Delete IP configured on the L3 interface into HW for ip2Me action even if the L3 interface is part of mgmt VRF. + +# Design Proposal + +The design is intended to have a generic approach for in-band management via mgmt VRF feature. A user can set an attribute "in_band_mgmt_enabled" to the config_db for MGMT_VRF_CONFIG table entry. The default value if not specified would be "false" + +The schema change for in-band management is as below: + +``` +MGMT_VRF_CONFIG|vrf_global + "mgmtVrfEnabled" : "true" + "in_band_mgmt_enabled": "true" + +``` +``` +; Defines management VRF table schema + +key = MGMT_VRF_CONFIG|vrf_global ; Same as existing +; field +mgmtVrfEnabled = "true"/"false" ; Same as existing +in_band_mgmt_enabled = "true" / "false" ; Default "false" (Optional attribute), this field is active only when mgmtVrfEnabled is set to true. + +``` +``` +key = VLAN_INTERFACE|{{intf_name}} ; Any L3 interface table entry e.g INTERFACE, PORTCHANNEL_INTERFACE..etc. +; field +"vrf_name" = "mgmt" ; Existing field but accepts mgmt VRF name + +``` +# Flows + +The following diagrams capture the kernel and SAI configuration flows. + +## Mgmt VRF configuration flow + +![](in_band_mgmt_vrf_config_flow.png) + +## L3 inteface to mgmt VRF bind flow + +![](in_band_mgmt_vrf_intf_config_flow.png) + +# Additional Notes +1. The user has to decide whether eth0 and L3 interface (e.g mgmt VLAN) can co-exist in the mgmt VRF, if yes, the corresponding configurations have to be taken care. +2. The user has to take care of configuring ACL to provide higher priority to mgmt traffic trapped from mgmt VRF, to avoid any potential drop in the NPU because of the data traffic. +3. User has to decide whether to use same data port for data & mgmt traffic or only for mgmt traffic based on the use-case. +4. The user has to make sure STP configurations are done such a way no impact to mgmt traffic via mgmt VLAN. diff --git a/doc/in-band-mgmt/in_band_mgmt_vrf_config_flow.png b/doc/in-band-mgmt/in_band_mgmt_vrf_config_flow.png new file mode 100644 index 0000000000..d0442d5105 Binary files /dev/null and b/doc/in-band-mgmt/in_band_mgmt_vrf_config_flow.png differ diff --git a/doc/in-band-mgmt/in_band_mgmt_vrf_intf_config_flow.png b/doc/in-band-mgmt/in_band_mgmt_vrf_intf_config_flow.png new file mode 100644 index 0000000000..cda56e31a4 Binary files /dev/null and b/doc/in-band-mgmt/in_band_mgmt_vrf_intf_config_flow.png differ diff --git a/doc/mclag/MCLAG_Enhancements_HLD.md b/doc/mclag/MCLAG_Enhancements_HLD.md new file mode 100644 index 0000000000..84e9285e2c --- /dev/null +++ b/doc/mclag/MCLAG_Enhancements_HLD.md @@ -0,0 +1,1021 @@ +# MCLAG Enhancements +Rev 0.4 + +# Table of Contents +- **[List of Tables](#List-of-Tables)** +- **[Revision](#Revision)** +- **[About this Manual](#About-this-Manual)** +- **[Definition/Abbreviation](#Definition_Abbreviation)** + - [Table 1: Abbreviations](#Table-1-Abbreviations) +- **[1 Feature Overview](#1-Feature-Overview)** + - [1.1 Functional Requirements](#1_1-Functional-Requirements) + - [1.1.1 Static MAC support over MCLAG](#1_1_1-Static-MAC-support-over-MCLAG) + - [1.1.2 Configuration and Management Requirements](#1_1_2-Configuration-and-Management-Requirements) + - [1.1.3 Performance and Scalability Requirements](#1_1_3-Performance-and-Scalability-Requirements) + - [1.1.4 Warm Boot Requirements](#1_1_4-Warm-Boot-Requirements) + - [1.1.5 Unique IP for supporting L3 protocol over MCLAG VLAN interface Requirements](#1_1_5-Unique-IP-for-supporting-L3-protocol-over-MCLAG-VLAN-interface-Requirements) + - [1.1.6 Node reboot/down handling](#1_1_6-Node-reboot_down-handling) + - [1.1.7 MCLAG system MAC support requirements](#1_1_7-MCLAG-system-MAC-support-requirements) + - [1.1.8 MCLAG gateway MAC support requirements](#1_1_8-MCLAG-gateway-MAC-support-requirements) +- **[2 Functionality](#2-Functionality)** + - [2.1 Functional Description](#2_1-Functional-Description) + - [2.1.1 Dynamic configuration of MCLAG](#2_1_1-Dynamic-configuration-of-MCLAG) + - [2.1.2 Dynamic configuration of mclag_interface](#2_1_2-Dynamic-configuration-of-mclag_interface) + - [2.1.3 Keep-alive timer configuration](#2_1_3-Keep_alive-timer-configuration) + - [2.1.4 Session timeout configuration](#2_1_4-Session-timeout-configuration) + - [2.1.5 Static MAC support](#2_1_5-Static-MAC-support) + - [2.1.6 Aging disable for ICCP learned MAC addresses](#2_1_6-Aging-disable-for-ICCP-learned-MAC-addresses) + - [2.1.7 Performance improvements](#2_1_7-Performance-improvements) + - [2.1.8 Isolation group support](#2_1_8-Isolation-group-support) + - [2.1.9 Unique IP enhancements](#2_1_9-Unique-IP-enhancements) + - [2.1.10 Peer MCLAG node reboot/down handling.](#2_1_10-Peer-MCLAG-node-reboot_down-handling_) + - [2.1.11 MCLAG system MAC support](#2_1_11-MCLAG-system-MAC-support) + - [2.1.12 MCLAG gateway MAC support](#2_1_12-MCLAG-gateway-MAC-support) +- **[3 Design](#3-Design)** + - [3.1 Overview](#3_1-Overview) + - [3.2 DB Changes](#3_2-DB-Changes) + - [3.2.1 CONFIG DB](#3_2_1-CONFIG-DB) + - *[3.2.1.1 MCLAG Domain table](#3_2_1_1-MCLAG-Domain-table)* + - *[3.2.1.2 MCLAG Interface Table](#3_2_1_2-MCLAG-Interface-Table)* + - *[3.2.1.3 MCLAG UniqueIP Table](#3_2_1_3-MCLAG-UniqueIP-Table)* + - *[3.2.1.4 MCLAG Gateway MAC Table](#3_2_1_4-MCLAG-Gateway-MAC-Table)* + - [3.2.2 APP DB](#3_2_2-APP-DB) + - *[3.2.2.1 ISOLATION_GROUP_TABLE table](#3_2_2_1-ISOLATION_GROUP_TABLE-table)* + - *[3.2.2.2 APP_LAG_TABLE](#3_2_2_2-APP_LAG_TABLE)* + - *[3.2.2.3 APP_MCLAG_FDB_TABLE](#3_2_2_3-APP_MCLAG_FDB_TABLE)* + - [3.2.3 STATE DB](#3_2_3-STATE-DB) + - *[3.2.3.1 STATE_MCLAG_TABLE](#3_2_3_1-STATE_MCLAG_TABLE)* + - *[3.2.3.2 STATE_MCLAG_REMOTE_INTF_TABLE](#3_2_3_2-STATE_MCLAG_REMOTE_INTF_TABLE)* + - *[3.2.3.3 STATE_MCLAG_LOCAL_INTF_TABLE](#3_2_3_3-STATE_MCLAG_LOCAL_INTF_TABLE)* + - *[3.2.3.4 STATE_MCLAG_ARP_FDB_TABLE](#3_2_3_4-STATE_MCLAG_ARP_FDB_TABLE)* + - [3.3 ICCP Changes](#3_3-ICCP-Changes) + - [3.3.1 MCLAG interface flap handling](#3_3_1-MCLAG-interface-flap-handling) + - [3.3.2 MCLAG configuration change handling](#3_3_2-MCLAG-configuration-change-handling) + - [3.3.3 FDB handling changes](#3_3_3-FDB-handling-changes) + - *[3.3.3.1 Static MAC handling](#3_3_3_1-Static-MAC-handling)* + - *[3.3.3.2 Data structure and data type modifications](#3_3_3_2-Data-structure-and-data-type-modifications)* + - *[3.3.3.3 MAC sync optimizations](#3_3_3_3-MAC-sync-optimizations)* + - [3.3.4 Port Isolation changes](#3_3_4-Port-Isolation-changes) + - [3.3.5 MCLAG Domain State](#3_3_5-MCLAG-Domain-State) + - [3.3.6 Unique IP Changes](#3_3_6-Unique-IP-Changes) + - [3.3.7 Peer MCLAG node reboot/down handling](#3_3_7-Peer-MCLAG-node-reboot_down-handling) + - [3.3.8 MCLAG system MAC changes](#3_3_8-MCLAG-system-MAC-changes) + - [3.3.9 MCLAG Gateway MAC changes](#3_3_9-MCLAG-Gateway-MAC-changes) + - [3.4 Switch State Service Design](#3_4-Switch-State-Service-Design) + - [3.4.1 Orchestration Agent](#3_4_1-Orchestration-Agent) + - *[3.4.1.1 Isolation group orch agent](#3_4_1_1-Isolation-group-orch-agent)* + - *[3.4.1.2 PortsOrch Changes](#3_4_1_2-PortsOrch-Changes)* + - *[3.4.1.3 FdbOrch Changes](#3_4_1_3-FdbOrch-Changes)* + - *[3.4.1.4 MCLAG ARP MAC update to Kernel](#3_4_1_4-MCLAG-ARP-MAC-update-to-Kernel)* + - [3.5 SyncD](#3_5-SyncD) + - [3.6 MclagSyncD](#3_6-MclagSyncD) + - [3.6.1 FDB Changes](#3_6_1-FDB-Changes) + - *[3.6.1.1 Local MAC upstream using STATE_DB FDB_TABLE](#3_6_1_1-Local-MAC-upstream-using-STATE_DB-FDB_TABLE)* + - *[3.6.1.2 Removal of FDB table in MclagSyncd](#3_6_1_2-Removal-of-FDB-table-in-MclagSyncd)* + - *[3.6.1.3 Populating ICCP MAC updates to APP_DB MCLAG_FDB_TABLE](#3_6_1_3-Populating-ICCP-MAC-updates-to-APP_DB-MCLAG_FDB_TABLE)* + - *[3.6.1.4 Populating ICCP and MCLAG interface state information](#3_6_1_4-Populating-ICCP-and-MCLAG-interface-state-information)* + - [3.6.2 MCLAG config handling](#3_6_2-MCLAG-config-handling) + - [3.7 SAI](#3_7-SAI) + - [3.7.1 Port Isolation](#3_7_1-Port-Isolation) + - [3.7.2 Gateway MAC](#3_7_2-Gateway-MAC) + - [3.8 CLI](#3_8-CLI) + - [3.8.1 Click based CLI](#3_8_1-Click-based-CLI) + - *[3.8.1.1 Configuration commands](#3_8_1_1-Configuration-commands)* + - *[3.8.1.1.1 MCLAG Domain configuration](#3_8_1_1_1-MCLAG-Domain-configuration)* + - *[3.8.1.1.2 MCLAG interface configuration](#3_8_1_1_2-MCLAG-interface-configuration)* + - *[3.8.1.1.3 MCLAG unique IP configuration](#3_8_1_1_3-MCLAG-unique-IP-configuration)* + - *[3.8.1.1.4 MCLAG system MAC configuration](#3_8_1_1_4-MCLAG-system-MAC-configuration)* + - *[3.8.1.1.5 MCLAG gateway MAC configuration](#3_8_1_1_5-MCLAG-gateway-MAC-configuration)* + - [3.8.2 SONiC CLI](#3_8_2-SONiC-CLI) + - *[3.8.2.1 Configuration commands](#3_8_2_1-Configuration-commands)* + - *[3.8.2.1.1 MCLAG Domain configuration](#3_8_2_1_1-MCLAG-Domain-configuration)* + - *[3.8.2.1.2 MCLAG Interface configuration](#3_8_2_1_2-MCLAG-Interface-configuration)* + - *[3.8.2.1.3 MCLAG Gateway MAC configuration](#3_8_2_1_3-MCLAG-Gateway-MAC-configuration)* + - *[3.8.2.1.4 MCLAG Separate IP configuration](#3_8_2_1_4-MCLAG-Separate-IP-configuration)* + - *[3.8.2.2 Show commands](#3_8_2_2-Show-commands)* +- **[4 Flow Diagrams](#4-Flow-Diagrams)** + - [4.1 FDB Flow Diagrams](#4_1-FDB-Flow-Diagrams) + - [4.1.1 Local Static MAC upstream](#4_1_1-Local-Static-MAC-upstream) + - [4.1.2 Local Dynamic MAC update](#4_1_2-Local-Dynamic-MAC-update) + - [4.1.3 Remote MAC Processing](#4_1_3-Remote-MAC-Processing) + - [4.2 Port Isolation group processing](#4_2-Port-Isolation-group-processing) +- **[5 Serviceability and Debug](#5-Serviceability-and-Debug)** +- **[6 Warm Boot Support](#6-Warm-Boot-Support)** +- **[7 Scalability](#7-Scalability)** +- **[8 Upgrade/Downgrade considerations](#8-Upgrade_Downgrade-considerations)** +- **[9 Unit Test](#9-Unit-Test)** + - [9.1 DB table updates](#9_1-DB-table-updates) + - [9.2 FDB](#9_2-FDB) + - [9.3 Config handling](#9_3-Config-handling) + - [9.4 Port Isolation](#9_4-Port-Isolation) + - [9.5 MCLAG Interface Flap](#9_5-MCLAG-Interface-Flap) + - [9.6 MCLAG Unique IP enhancements](#9_6-MCLAG-Unique-IP-enhancements) + - [9.7 MCLAG system MAC](#9_7-MCLAG-system-MAC) + - [9.8 MCLAG Gateway MAC](#9_8-MCLAG-Gateway-MAC) +- **[10 Internal Design Information](#10-Internal-Design-Information)** + +# List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +| ---- | ---------- | ---------------------------------------- | ------------------ | +| 0.1 | 08/02/2019 | Lisa Nguyen, Praveen Elagala, Abhishek Dharwadkar, Sabarivel Sakthivel, Adam Yeung | Initial draft | +| 0.2 | 10/22/2019 | Tapash Das | L3 Protocol support enhancement | +| 0.3 | 04/15/2020 | Praveen Elagala | MCLAG System MAC support | +| 0.4 | 05/05/2020 | Tapash Das | MCLAG Gateway MAC support | + +# About this Manual +This document provides general information about SONiC MCLAG feature enhancements. The original MCLAG feature is described at https://github.com/Azure/SONiC/blob/f478fe7cbc03c144f3b147e9638f460f764ce4b7/doc/Sonic-mclag-hld.md + + +# Definition/Abbreviation +## Table 1: Abbreviations +| **Term** | **Definition** | +| -------- | ------------------------------------ | +| MCLAG | Multi-Chassis Link Aggregation Group | +| ICCP | Inter-Chassis Communication Protocol | +| FDB | Layer-2 (MAC) based forwarding table | +| MHD | Multi Home Device | +| MCLAG System MAC | Actor System MAC used in LACP PUD | + +# 1 Feature Overview + +This document captures the feature enhancements of SONiC ICCP MCLAG. This includes data structure changes, MAC event handling optimizations for scaling performance, support of static MAC address over MCLAG, port isolation ACL, and traffic recovery sequencing for traffic loop prevention. + +## 1.1 Functional Requirements + +### 1.1.1 Static MAC support over MCLAG +- Advertise locally configured Static MAC address to peer MCLAG node +- Withdraw MAC from peer MCLAG node when static MAC address is un-configured +- Dynamic MAC move is prohibited on Static MAC over MCLAG +- Support bridge-port isolation group for BUM control to MHD. + +### 1.1.2 Configuration and Management Requirements +- Add CLI to configure MCLAG domain and corresponding attributes domain-id, local-ip-address and peer-ip-address + +- Add CLI to configure optional attribute [peer-interface] + +- Add CLI to configure keep-alive timer value + +- Add CLI to configure session-timeout value + +- Add CLI to support addition/deletion of mclag_interfaces + +### 1.1.3 Performance and Scalability Requirements +- Optimize code flow and data structures to improve scaling performance +- Support 40K FDB entries +- Support 4K VLAN's + +### 1.1.4 Warm Boot Requirements +- MCLAG peer nodes should reconcile the local FDB table upon completion of warm boot, as the MAC learn and age updates from peer would be lost during the time ICCP control session is down due to warm boot. + +### 1.1.5 Unique IP for supporting L3 protocol over MCLAG VLAN interface Requirements +- For VLAN's associated with MCLAG interface, allow configuration of Unique IP address on VLAN interface of Peer nodes. +- Provide mechanism for running L3 protocols such as BGP, BFD between both Peer nodes VLAN interface. +- Provide mechanism for running L3 protocols such as BGP, BFD between Peer node VLAN interface and device connected to MCLAG Client. +- For MCLAG gateway functionality (Data path) Static Anycast Gateway (SAG) or VRRP configuration will be mandatory, SAG is preferred for active-active forwarding. + +### 1.1.6 Node reboot/down handling +- Add support to notify the peer device when an MCLAG node is about to reboot.. +- Add processing in Standby MCLAG node not to initiate the session down behavior while Active is rebooting. + +### 1.1.7 MCLAG system MAC support requirements +- Add support to allow configuration of a user-defined system MAC value for LACP PDUs transmitted by the switch on MCLAG port channels. +- Same MCLAG system MAC should be configured on both MCLAG peer nodes. +- If the ICCP session goes down and peer_link is up, then bring down the MCLAG port channel ports on one of the MCLAG peer units. This is to avoid loops that might be caused by 2 independent switches using the same MCLAG system MAC value in their LACP PDUs. + +### 1.1.8 MCLAG gateway MAC support requirements +- Add support to configure gateway MAC on MCLAG Nodes. +- Gateway MAC is updated for all VLAN routing Interfaces for which peer link is one of the VLAN member. +- Gateway MAC is independently configured on both MCLAG peers, and must be the same. +- Existing behavior of programing Active node's System MAC in the Standby node's MCLAG SVIs is the default configuration. + +# 2 Functionality + +## 2.1 Functional Description +### 2.1.1 Dynamic configuration of MCLAG +- Current MCLAG configuration is generated from config_db.json file with no CLI support. New CLIs will be added to configure MCLAG which includes domain-id, local ip address, peer ip address and peer_link. peer_link configuration is mandatory for Layer2 MCLAG while optional for Layer3 MCLAG. + +### 2.1.2 Dynamic configuration of mclag_interface +- In the current implementation the mclag_interfaces configuration is generated from config_db.json file no CLI support is present, a new configuration command is introduced to dynamically add/delete the mclag_interfaces. + +### 2.1.3 Keep-alive timer configuration +- In current implementation the keep-alive timer is hardcoded to 1 second. A new CLI command will be introduced to allow the configuration of range of values for keep-alive timer. + +### 2.1.4 Session timeout configuration +- In current implementation the session time out 15 seconds after the last keep-alive messages exchange, the timeout duration is hardcoded. A new CLI command will be added to allow a configurable duration for session timeout, session timeout has to be configured in multiples keep-alive timer value. + +### 2.1.5 Static MAC support +- Current implementation does not support static MAC address over MCLAG. Static MAC information is not exchanged in the FDB TLV between the MCLAG peer nodes. Changes will be added to support Static MAC with TLV. +- Changes will be added to support the static MAC syncing from local MCLAG node to peer MCLAG node. +- Changes will be added to handle the MAC move scenarios in presence of remote Static MAC. + +### 2.1.6 Aging disable for ICCP learned MAC addresses +- In the current implementation, MAC addresses learned in the control plane from the MCLAG peer are installed as dynamic entries. Therefore, if no local traffic from these MAC addresses is seen for the ageing period then they will age out, causing a notification back to ICCPd. However if the mac is learned from peer then it will immediately re-program MAC . The result is transient flooding for the MAC and some unnecessary control plane overhead. +- When the local MCLAG node learns the MAC addresses from peer MCLAG node the type of the MAC set as dynamic. For the remote MAC addresses learned via ICCP if no traffic received on local MCLAG interface MAC addresses get age out from HW. ICCPd process the age notifications as the MAC address is peer learned re-installs MAC address back to HW. Transient traffic flooding can occur during remote MAC re-installation. +- The process of remote MAC aging and re-installation is repetitive causing un-necessary messaging between modules and processing. +- To suppress the unwanted MAC age events, the new implementation programs remote MAC addresses with aging disabled. For the MAC address learned from ICCPd, FdbOrch to set new SAI attribute SAI_FDB_ENTRY_TYPE_STATIC_MACMOVE while programming, which causes SAI to not age out the MAC, but allows it to move. + +### 2.1.7 Performance improvements + +The following improvements are made in this enhancement: + +- Convert FDB and VLAN related linked list to binary trees to improve scaling performance +- When MCLAG node learns a new MAC address via ICCP from peer MCLAG node learned over remote orphan port, it sends a age notification back to peer to indicate MAC is locally aged, this updated is not necessary as MAC not learned locally. Avoid sending the local age notification back to peer MCLAG node each time a MAC address is learned from peer. + +### 2.1.8 Isolation group support + +To control the BUM traffic towards the MHD bridge-port isolation groups is used. The isolation group is applied on the ingress side of peer-link port to filter traffic towards the MHD to avoid duplicate traffic received at MHD. + +### 2.1.9 Unique IP enhancements + +- Current design mandates configuring same IP address on the VLAN interface for each VLAN of which an MCLAG is a member. +- L3 protocols such as BGP, BFD, VRRP etc cannot be configured on the VLAN interface associated with MCLAG interface because such protocols cannot address the individual routers. +- A new configuration command is introduced for allowing separate per-peer IP configuration on the VLAN interfaces associated with MCLAG interface. +- Both same IP configuration and unique IP configuration will co-exist based on the new configuration. + +### 2.1.10 Peer MCLAG node reboot/down handling. + +- Add support in ICCPd to notify the peer MCLAG node when the node is going for reload. +- Standby MCLAG node is then aware of Active going down, and when peer session is down Standby MCLAG node does not change the system MAC to keep the traffic flowing through the Standby node MCLAG portchannels. +- MAC and ARP information is retained while peer is rebooting for better convergence. +- If the session is not up after peer reboot timer expiry (fixed at 5 mins) , the ICCP session down behavior is initiated on Standby MCLAG node. +- Note in case of unplanned node down situation the MCLAG down notification is not sent to peer, and peer MCLAG node handles the session down according to the session-timeout timer expiry. + +### 2.1.11 MCLAG system MAC support + +- In current implementation both Active and Standby MCLAG nodes use the Active nodes system MAC address in LACP PDU's, when the user configures the MCLAG system MAC address, on both Active and Standby MCLAG nodes configured MCLAG system MAC is used in LACP. +- As part of the new enhancement user is allowed to configure MCLAG system MAC address . User configured MCLAG system MAC will take precedence over the local system MAC, MCLAG PortChannels will be updated with configured MAC address. +- In case of user configured MCLAG system MAC, It is recommended to configure the MCLAG system MAC before adding the MCLAG member PortChannels to domain, to avoid one additional PortChannel flap. +- By using MCLAG system MAC in LACP , there are no changes to MCLAG system MAC for MCLAG PortChannels during session down and MCLAG member delete on the Active node, which avoids flap of PortChannels. +- The MCLAG system MAC is only used for the MCLAG portChannels , and not used for orphan PortChannels or peer_link PortChannel interfaces. +- When the MCLAG ICCP session is detected as being down by the Standby node, it brings down all the links in its MCLAG port channels. This is to avoid loops and duplicates. + + +### 2.1.12 MCLAG gateway MAC support + +- Before this enhancement + + In current MCLAG community design, when ICCP session is Up, Active nodes system MAC is synced to Standby node and programmed as Vlan interface MAC in both kernel and hardware. + When ICCP session goes down, Vlan interface MAC in Standby node is changed to its own system MAC. + This behaviour causes traffic disruption during failover scenarios like Active node reboot or ICCP session flaps. + + In Logical VTEP deployment, Vlan interface MAC is advertised as gateway MAC in Type-5 EVPN routes. + During failover when Vlan interface MAC is changed in Standby Node, an updated BGP EVPN Type-5 routes should be advertised with updated gateway MAC. + On receving updated routes from BGP, remote VTEP nodes need to reprogram routes with updated gateway MAC. + This causes unnecessary churn in network. + +- Configurable gateway MAC enhancement + + As part of this enhancement, a new configurable gateway MAC is supported. + Same gateway MAC must be configured in both Active node and Standby Node. + Gateway MAC configuration overrides default behaviour of programing Active nodes MAC in Standby node's VLAN routing interface. + +# 3 Design + +## 3.1 Overview +New changes will be added on top of the existing MCLAG implementation. +Following diagram captures the high level module changes made for the MCLAG enhancements. + +![Design overview diagram](images/DesignOverviewDiagram.png "Overview of module changes") + +**Figure 1: Overview of module changes** + +Below is the summary of the changes for each flow number mentioned in the above diagram. + +1. New CLI commands are introduced to configure MCLAG and its related attributes. Two new tables are added in CONFIG_DB to store the MCLAG configuration and mclag_interface configuration generated from CLI. +2. MclagSyncd receives the new MCLAG configuration updates. +3. MclagSyncd receives new local (both static and dynamic) MAC updates from STATE_DB FDB_TABLE +4. MclagSyncd sends new MCLAG configuration to ICCPd. +5. ICCPd sends new remote static MAC updates, ICCP session and MCLAG remote interface state information updates to MclagSyncd. +6. ICCPd sets new static MAC flag in FDB TLV for local static MAC addresses advertisement. +7. MclagSyncd updates ICCP session and MCLAG remote interface state information to STATE_DB FDB_TABLE and MCLAG_REMOTE_INTF_TABLE +8. MclagSyncd updates MAC addresses learned from peer MCLAG node to new MCLAG FDB table . +9. FdbOrch registers for new MCLAG FDB table updates to process MAC updates from peer MCLAG node, ISOGRP Orch process new updates from MclagSyncd. +10. FdbOrch updates remote MAC addresses with new SAI attribute not to age them out. +11. Syncd gets the ICCP learned remote MAC addresses with new SAI attribute SAI_FDB_ENTRY_TYPE_STATIC_MACMOVE set . +12. Syncd programs the remote MAC to HW +13. FdbOrch registers with NeighborOrch for all the ICCP learned remote MAC addresses. +14. NeighborOrch update the MAC address to STATE_DB MCLAG ARP MAC Table. +15. FdbSycd gets the MAC updates from STATE_DB MCLAG ARP MAC Table. +16. FdbSyncd programs the MAC updates to kernel. + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB +#### 3.2.1.1 MCLAG Domain table + +Producer: Configuration +Consumer: MclagSyncd +Description: New table to store MCLAG domain configuration. Applications can use it to get MCLAG configuration + +``` +;New MCLAG Domain Table +key = MCLAG_DOMAIN|domain_id ; DIGIT 1-4095 + +;field = value +source_ip = IP ; local ip address +peer_ip = IP ; peer ip address +peer_link = port_name ; peer link name +mclag_system_mac = MAC ; MCLAG system MAC address. +keepalive_interval = time_in_secs ; keepalive time in seconds +session_timeout = time_in_secs ; session timeout in multiple of + ; keepalive_interval. + +;value annotations +port_name = 1*64VCHAR ; name of the port, must be unique +time_in_secs = dec-octet ; time in seconds +IP = IPV4address +IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet " + + dec-octet = DIGIT ; 0-9 + / %x31-39 DIGIT ; 10-99 + / "1" 2DIGIT ; 100-199 + / "2" %x30-34 DIGIT ; 200-249 + / "25" %x30-35 ; 250-255 +``` +#### 3.2.1.2 MCLAG Interface Table + +Producer: Configuration +Consumer: MclagSyncd +Description: New table to store MCLAG interfaces for a MCLAG domain. MCLAG applications can query this table to check whether a interface is configured as MCLAG interface + +``` +;New MCLAG Interface Table +key = MCLAG_INTERFACE|domain_id|ifname ; + +;value annotations +domain_id = DIGIT 1-4095 ; domain_id for the MCLAG +ifname = 1*64VCHAR ; name of the MCLAG Interface (PortChannel) + +``` + +#### 3.2.1.3 MCLAG UniqueIP Table + +Producer: Configuration +Consumer: MclagSyncd +Description: New table to store L3 VLAN interfaces that will have unique IP configuration. MCLAG applications can query this table to check whether a interface is configured for L3 protocol support. + +``` +;New MCLAG UniqueIP Table +key = MCLAG_UNIQUEIP_TABLE|ifname ; Only VLAN interface supported currently + +``` + +#### 3.2.1.4 MCLAG Gateway MAC Table + +Producer: Configuration +Consumer: MclagSyncd +Description: New table to store Gateway MAC configuration. MCLAG applications can query this table to check whether a Gateway MAC is configured. + +``` +;New MCLAG Gateway MAC Table +key = MCLAG_GW_MAC|gw_mac; MAC Address in xx:xx:xx:xx:xx:xx format. + +``` + +### 3.2.2 APP DB + +#### 3.2.2.1 ISOLATION_GROUP_TABLE table + +Producer: MclagSyncd +Consumer: Isolation Group Orch Agent +Description: Isolation group table contains information about the Isolation group type, members and ports. + +``` +key = ISOLATION_GROUP_TABLE:name + +;field = value +Description = 1*255VCHAR +TYPE = "port" / "bridge-port" +PORTS = [0-max_ports]*port_name ; Ports on which this port isolation group + ; is applied +MEMBERS = [0-max_ports]*port_name ; Port isolation group members + +;value annotations +port_name = 1*64VCHAR ; name of the port, must be unique +max_ports = 1*5DIGIT ; number of ports supported on the chip +``` + +#### 3.2.2.2 APP_LAG_TABLE +``` +;Add one new attribute to existing LAG_TABLE to control whether traffic collection and distribution should be disabled for all LAG member ports. +traffic_disable = "true"/"false" ;default is false +``` +#### 3.2.2.3 APP_MCLAG_FDB_TABLE + +Producer: MclagSyncd +Consumer: FdbOrch agent +Description: New table to stores the MAC addresses learned via ICCP. FdbOrch processes the FDB updates to program in HW. + +``` +; Stores MCLAG FDB entries which were learned via ICCP +; Notes: +; - only unicast FDB entries supported +key = FDB_TABLE:"Vlan"vlanid:mac_address ; +;fields +port = ifName ; interface where the entry is bound to +type = "static" / "dynamic" ; type of the entry +``` + +### 3.2.3 STATE DB + +#### 3.2.3.1 STATE_MCLAG_TABLE +Producer: MclagSyncd +Consumer: Applications over MCLAG +Description: New table to provide MCLAG domain states to applications running over MCLAG + +``` +;New STATE_MCLAG_TABLE +key = MCLAG_TABLE|domain_id +;value annotations +domain_id = DIGIT 1-4095 ; domain_id for the MCLAG +;fields +oper_status = "down"/"up" +role = "active"/"standby" +system_mac = 12HEXDIG +mclag_system_mac = 12HEXDIG +``` +#### 3.2.3.2 STATE_MCLAG_REMOTE_INTF_TABLE +Producer: MclagSyncd +Consumer: Applications over MCLAG +Description: New table to provide remote MCLAG interface state received by ICCP +Schema: + +``` +; New STATE_MCLAG_REMOTE_INTF_TABLE +key = MCLAG_REMOTE_INTF_TABLE|domain_id|ifname +oper_status = "down"/"up" + +;value annotations +domain_id = DIGIT 1-4095 ; domain_id for the MCLAG +ifname = 1*64VCHAR ; name of the MCLAG Interface (PortChannel) + +``` + +#### 3.2.3.3 STATE_MCLAG_LOCAL_INTF_TABLE +Producer: MclagSyncd +Consumer: TeamMgr +Description: New table to provide local MCLAG interface and indication to disable by ICCPD. +Schema: + +``` +; New STATE_MCLAG_LOCAL_INTF_TABLE +key = MCLAG_LOCAL_INTF_TABLE|domain_id|ifname +oper_status = "down"/"up" +is_disable = "true/false" +;value annotations +domain_id = DIGIT 1-4095 ; domain_id for the MCLAG +ifname = 1*64VCHAR ; name of the MCLAG Interface (PortChannel) + +``` + +#### 3.2.3.4 STATE_MCLAG_ARP_FDB_TABLE + +Producer: FdbOrch +Consumer: NeighborOrch +Description: New table to populate the ICCP learned remote MAC entries which have ARP entries are associated. +Schema: + +``` +; New STATE_MCLAG_ARP_FDB_TABLE +key = MCLAG_ARP_FDB_TABLE:"Vlan"vlanid:mac_address + +;value annotations +vlanid = DIGIT 1-4095 ; VLAN id. +mac_address = 12HEXDIG ; MAC address. + +;fields +port = 1*64VCHAR ; name of the L2 Port +type = "static/dynamic" ; tpye of the MAC address. + +``` + + + +## 3.3 ICCP Changes +### 3.3.1 MCLAG interface flap handling + +In the current implementation, in normal operation, a port isolation filter is applied on both peers to ensure that traffic received from the peer-link is never forwarded to local MCLAG member ports. This ensures loop-free and packet duplication-free operation. Then, if all ports in an MCLAG on one peer go down (MCLAG PortChannel is down), this filter is removed on the other peer, allowing traffic destined to the MCLAG to cross the peer link, maintaining connectivity to the MCLAG. Later, when at least one member port comes back up (MCLAG PortChannel is up) on the first peer, then the filter must be re-applied on the other peer. However, this can cause transient loops if the MCLAG member ports come up before the filter is re-applied on the remote peer. + +In this enhancement, acknowledgements are added to interface state notifications in ICCP, and these are used to ensure that the remote filter is applied before enabling traffic on the upcoming MCLAG member interface(s), thereby eliminating the possibility of loops and duplicates. + +### 3.3.2 MCLAG configuration change handling +ICCP is extended to dynamically handle changes in MCLAG configurations. In the current implementation, MCLAG config attributes local ip, peer ip, peer interface and MCLAG interface configuration has to be done prior to bringing up ICCP docker. To handle configuration changes, MclagSyncd subscribes to CONFIG_DB MCLAG changes, saves configuration data in internal cache and notifies ICCPd. In this enhancement, Pre-provisioning support for MCLAG interfaces and peer interfaces is added. + + +### 3.3.3 FDB handling changes +ICCPd will be extended to support static MAC addresses, data structure changes, data type changes and MAC syncing optimizations. More details are captured in the below sections. + +#### 3.3.3.1 Static MAC handling +Changes will be added to extend the current FDB TLV to notify with a static MAC flag. ICCP MAC processing will be extended for local and remote static MAC addresses handling. Local and remote static MAC move scenarios will be handled. + +#### 3.3.3.2 Data structure and data type modifications +In current ICCPd implementation MAC entries are stored in linked lists which can be costly in MAC scale scenarios for lookups. The existing linked list structures will be modified to binary trees for better add, delete and lookup operations. The MAC address entry is stored as 32 byte string in the messaging and local cache, the address will be converted to 6 bytes to optimize the space messages size while syncing the MAC addresses between MCLAG peer nodes. + +#### 3.3.3.3 MAC sync optimizations +When MCLAG node learns a new MAC address via ICCP from peer MCLAG node ( MAC learned on remote orphan port ), MAC address is added locally and an age notification is sent back to peer MCLAG node. These age notifications are not required to be sent, additional checks are added to stop sending such unwanted MAC updates. + +### 3.3.4 Port Isolation changes +In this enhancement, the port isolation logic is updated to use [SAI Port Isolation](https://github.com/opencomputeproject/SAI/blob/master/inc/saiisolationgroup.h) instead of Egress ACL. Using isolation group, unwanted traffic can be dropped at the ingress, i.e. before it gets queued etc. This will give better performance as unnecessary traffic will filtered at ingress. + +### 3.3.5 MCLAG Domain State +To support applications running over MCLAG, MCLAG domain states such as ICCP state, Active and Standby role, system MAC, and remote MCLAG interface state are populated in the STATE_DB for applications to consume. + +### 3.3.6 Unique IP Changes +To support Unique IP configuration on VLAN interface for the VLAN's associated with MCLAG interface following changes will be done. +- In Standby node, disable Active node's MAC programming into MY Station TCAM. Also disable updating VLAN interface MAC in kernel. +- Add code to Sync VLAN interface MAC between Peers and program L2 Table with Peers VLAN interface MAC pointing to Peer Link. This is required since MAC learning is disabled for Peer Link. +- Add code to do ARP/ND sync for Local VLAN interface IP. + +For MCLAG gateway functionality (Data path) Static Anycast Gateway or VRRP configuration will be mandatory. + +### 3.3.7 Peer MCLAG node reboot/down handling + +- MCLAG node to notify the peer MCLAG node when the node is going for a reboot. ICCPd code is extended to receive the ICCPd going down as part of the node reboot and notify the peer MCLAG node via a new TLV. +- When Standby MCLAG node receives the Active going down as part of peer node reboot, Standby MCLAG node to delay the session going down while the Active MCLAG node is down to keep the MCLAG PortChannel's up. Active MCLAG node does not process the Standby MCLAG node reboot notification event received. +- If the session is not up before the peer reboot timer expiry peer MCLAG node is assumed down and to trigger the MCLAG session down behavior. + +### 3.3.8 MCLAG system MAC changes + +In the current implementation, during the creation of MCLAG domain the local system MAC address is passed to ICCPd by mclagSyncd. In the Active MCLAG node the same MAC is used in LACP PDU's, Active MCLAG node sync the local system MAC address to peer Standby MCLAG node. Standby MCLAG node uses the Active MCLAG node system MAC in LACP PDU's. + +When user configures the MCLAG system MAC address, the MCLAG system MAC is used in LACP PDU's for MCLAG port channels, following changes are implemented in ICCPd. + +- ICCPd code enhanced to handle the new CLI for the MCLAG system MAC configuration. + +- Update the new MCLAG system MAC address for all the MCLAG portChannels both in Active and Standby MCLAG peer nodes. When PortChannel is deleted as member of MCLAG then update the PortChannel to use the local system MAC for LACP. + +- Update the STATE_DB with the MCLAG system MAC. + +- On Standby MCLAG node ICCPd is enhanced to update the PortChannel state during session down while peer_link is up to disable MCLAG PortChannels to avoid traffic loop and duplicates . When ICCP session is back up ICCPd enables the PortChannels back up. STATE_DB introduces new MCLAG local interface table with a flag to indicate to disable the PortChannel. TeamMgr processed the updates and enables and disables the PortChannel state accordingly. + +### 3.3.9 MCLAG Gateway MAC changes +To support MCLAG gateway MAC configuration on VLAN interface for which peer link is member port following changes will be done. +- On gateway MAC configuration update Vlan interface MAC in kernel. +- On gateway MAC configuration update MY Station TCAM in Hardware. +- Do not update the Vlan interface MAC for the interface those have Unique IP configured. +- On removing gateway MAC configuration restore the default behaviour. + +## 3.4 Switch State Service Design + +### 3.4.1 Orchestration Agent + +#### 3.4.1.1 Isolation group orch agent + +A new isolation group orch agent will be added. Isolation group orch agent is the consumer of ISOLATION_GROUP_TABLE in APP DB. MclagSyncd will produce the isolation group configuration in the APP DB and isolation group will program it in the ASIC DB. Only bridge port isolation group will be supported and will be used by MclagSyncd. + +#### 3.4.1.2 PortsOrch Changes + +To allow ICCP to control when traffic should be enabled on a LAG interface, PortsOrch is extended to support a new traffic disable attribute. When traffic disable attribute is set to true, LAG members are not added to hardware. They are kept in a temporary set until the traffic disable attribute is set to false. + + +#### 3.4.1.3 FdbOrch Changes + +As part of enhancements MclagSyncd populates the ICCP learned MAC addresses into the APP_DB MCLAG_FDB_TABLE instead of populating to APP_FDB_TABLE, FdbOrch registers to APP_DB MCLAG_FDB_TABLE to process the MAC updates from ICCP. + +Advertised FDB cache entries will now be marked about the learn source, MAC entries learned from APP_DB MCLAG_FDB_TABLE will be marked as learned from ICCP, these MAC addresses will be updated to ASIC_DB FDB_TABLE with new SAI attributes. + +Advertised FDB entries with type static will be programmed to HW similar to local static MAC addresses to avoid any dynamic MAC move to other interfaces. + +When a static MAC address is configured locally and same MAC address is learned as advertised, FdbOrch to discard moving the MAC address. When static MAC address is deleted FdbOrch to re-program the advertised MAC address to HW. + +If the ICCP learned MAC is static, then any dynamic MAC move will be discarded by FdbOrch. + +#### 3.4.1.4 MCLAG ARP MAC update to Kernel + +ARP entries present in the system will need the associated MAC entries to be programmed in kernel to allow the local system and applications to reach the associated IP address. When ICCP receives a MAC update, if an ARP entry is associated with the MAC address then the MAC address needs to be programmed in kernel. FdbOrch notifies NeighborOrch when MAC is learned via ICCP and NeighborOrch updates the MCLAG_ARP_FDB_TABLE in STATE_DB accordingly. FDBSyncd subscribes to this table and programs the kernel as required. A similar path is used to update kernel MAC entry If there is an interface change associated with the MAC or when the MAC address is deleted. + +Below Diagram show the remote ICCP MAC update to Kernel. + +![MCLAG ARP MAC kernel update Diagram](images/MclagArpFdbKernelUpdate.png "MCLAG ARP interested MAC kernel programming Flow") + +**Figure 2: MCLAG ARP interested MAC kernel programming Flow** + +## 3.5 SyncD +No change to SyncD + +## 3.6 MclagSyncD +### 3.6.1 FDB Changes +MclagSyncd is extended to optimize the FDB events handling. + +The changes include: - + +- sending local MAC updates to ICCPd from STATE_DB FDB_TABLE instead of ASIC_DB FDB_TABLE +- sending of the MAC entries is event driven instead of the periodic 60 seconds polling +- removal of the FDB cache in the MclagSyncd +- adding the ICCP learned MAC addresses to APP_DB MCLAG_FDB_TABLE + +#### 3.6.1.1 Local MAC upstream using STATE_DB FDB_TABLE +To distribute the local MAC addresses to ICCPd MclagSyncd will now listen for the STATE_DB FDB_TABLE updates. Current approach of periodic MAC table retrieval by polling and compare to find the MAC latest updates is costly in the scale scenarios. In the new approach STATE_DB FDB_TABLE MAC processing is event driven at MclagSyncd and no periodic polling is done. + +#### 3.6.1.2 Removal of FDB table in MclagSyncd +With the above change to event-driven updates, MclagSyncd no longer needs a local FDB table copy - it can update directly between ICCPd and FdbOrch. + +#### 3.6.1.3 Populating ICCP MAC updates to APP_DB MCLAG_FDB_TABLE + +In current implementation MclagSyncd updates the MAC addresses into the existing APP_DB FDB_TABLE. In this enhancement, a new APP_DB MCLAG_FBD_TABLE is used. This allows clients to distinguish between local and remote (ICCP received) MAC address. + +#### 3.6.1.4 Populating ICCP and MCLAG interface state information + +MclagSyncd is extended to receive the ICCP session related information from ICCPd and to populate this information in STATE_DB STATE_MCLAG_TABLE. MclagSyncd also receives MCLAG remote interface information which is populated in STATE_DB STATE_MCLAG_REMOTE_INTF_TABLE. + +### 3.6.2 MCLAG config handling +MclagSyncd will be enhanced to listen to CONFIG_DB changes in MCLAG domain table and MCLAG_INTF_TABLE and notify ICCPd. ICCPd is enhanced to process the configuration and handle config changes accordingly. + +MclagSyncd to handle the MCLAG system MAC configuration updates for MCLAG PortChannels use. MclagSyncd to update the Local interface table along with port channel disable flag when session is down. TeamMgr to listen for the state updates and disable/enable the PortChannel admin state. + + + +## 3.7 SAI +### 3.7.1 Port Isolation +The following SAI definitions will be used and no enhancements necessary + +- https://github.com/opencomputeproject/SAI/blob/master/inc/saiisolationgroup.h + +### 3.7.2 Gateway MAC +Existing SAI attribute SAI_ROUTER_INTERFACE_ATTR_SRC_MAC_ADDRESS is used to update the MAC. + + +## 3.8 CLI +### 3.8.1 Click based CLI + +#### 3.8.1.1 Configuration commands + +Two new sets of configuration commands are introduced to configure MCLAG domain attributes and domain's MCLAG interfaces + +##### 3.8.1.1.1 MCLAG Domain configuration +``` +config mclag {add | del} \ \ \ \ [\] + +- domain-id MCLAG node's unique domain-id + +- local_ip MCLAG node's local ipv4 address + +- peer_ip MCLAG node's peer ipv4 address + +- peer_ifname MCLAG peer interface name; optional for L3 MCLAG, mandatory for L2 MCLAG config + +``` + + +##### 3.8.1.1.2 MCLAG interface configuration +``` +config mclag member {add | del} \ + +- comma seperated MCLAG interfaces for given MCLAG domain; MCLAG interfaces can be only portchannels +``` + +**MCLAG Keepalive/session timeout configuration** +``` +config mclag keepalive-interval \ \ +config mclag session-timeout \ \ + +- keepalive-timer in the range of 1 to 60 seconds with default value of 1 seconds. +- session-timeout in the range of 1 to 3600 seconds with default value of 30 seconds +- session-timeout should be at least greater than 3 times keepalive-timer value +``` +##### 3.8.1.1.3 MCLAG unique IP configuration +``` +config mclag unique-ip {add | del} + +- Vlan-interface : VLAN interface for which unique IP can be configured. + +``` +##### 3.8.1.1.4 MCLAG system MAC configuration +``` +config mclag system_mac {add | del} \ + +- add : Configure MCLAG system MAC. +- del : Remove MCLAG system MAC. +- mac-address : MAC Address in xx:xx:xx:xx:xx:xx format + +``` + +##### 3.8.1.1.5 MCLAG gateway MAC configuration +``` +config mclag gw-mac {add | del} + +- add : Configure MCLAG Gateway MAC. +- del : Remove MCLAG Gateway MAC. +- mac-address : MAC Address in xx:xx:xx:xx:xx:xx format. + +``` + +### 3.8.2 SONiC CLI + +#### 3.8.2.1 Configuration commands + +Following KLISH based CLI configuration commands are introduced to configure MCLAG domain attributes and MCLAG interfaces. + +##### 3.8.2.1.1 MCLAG Domain configuration +``` +To configure and unconfigure MCLAG domain at global configuration level use below command. Config command creates MCLAG domain and enters domain configuration mode. + +sonic(config)#[no] mclag domain + +domain-id is a integer in the range of 1 to 4095 +``` + + +To configure MCLAG attributes enter MCLAG domain configuration mode and enter following command +``` +sonic(config-mclag-domain)#[no] source-address + +sonic(config-mclag-domain)#[no] peer-address + +sonic(config-mclag-domain)#[no] peer-link + +sonic(config-mclag-domain)#[no] system-mac + +``` + + +To configure MCLAG session keepalive interval and session timeout use following commands +``` +sonic(config-mclag-domain)#[no] keepalive-interval + +sonic(config-mclag-domain)#[no] session-timeout + +- Keepalive interval in the range of 1 to 60 seconds with default value of 1 seconds. + +<<<<<<< HEAD +- session-timeout in the range of 1 to 3600 seconds with default value of 30 seconds + +- session-timeout should be multipler of keepalive-timer value + +``` + +##### 3.8.2.1.2 MCLAG Interface configuration + +To configure/unconfigure MCLAG interfaces use following command. MCLAG interfaces can only be portchannels. Configure MCLAG interface under the portchannel interface configuration mode. + +**sonic(config-if-PortChannelx)# [no] mclag [domain-id] + +##### 3.8.2.1.3 MCLAG Gateway MAC configuration + +To configure/unconfigure MCLAG Gateway MAC use following command. + +**sonic(config)# mclag# [no] mclag gateway-mac xx:xx:xx:xx:xx:xx + +##### 3.8.2.1.4 MCLAG Separate IP configuration + +To configure/unconfigure separate IP on Vlan interface for L3 protocol support over MCLAG. +This is VLAN interface mode configuration. + +sonic(config)# interface Vlan 10 +sonic(conf-if-Vlan10)# [no] mclag-separate-ip + +#### 3.8.2.2 Show commands + +Following new KLISH based show commands are introduced + +Displays brief information about MCLAG domain + +``` +1. show mclag brief + +Sonic#show mclag brief + + Domain ID : 5 + Role : Active + Session Status : Up + Peer Link Status : Up + Source Address : 192.168.1.1 + Peer Address : 192.168.1.2 + Peer Link : PortChannel30 + Keepalive Interval : 1 secs + Session Timeout : 15 secs + System MAC : b8:6a:97:73:6c:96 + MCLAG System MAC Address : 00:80:c2:00:00:05 + Gateway MAC Address : 00:01:02:03:04:05 + Number of MCLAG Interfaces : 2 + MCLAG Interface Local/Remote Status + PortChannel50 Up/Up + PortChannel60 Up/Up + +``` + +2. **Show mclag interface ** +``` + Sonic#show mclag interface 5 PortChannel50 + Local/Remote Status : Up/Up + IsolateWithPeerLink : Yes + TrafficDisable : No +``` + +3. **Show mclag separate_ip_interfaces** +``` +Sonic#show mclag separate_ip_interfaces + + Interface Name + ============== + Vlan10 + ============== + Total count : 1 + ============== +``` + +# 4 Flow Diagrams +## 4.1 FDB Flow Diagrams +### 4.1.1 Local Static MAC upstream + Below diagram shows the flow for local static MAC addresses upstream to ICCPd. + +![Local Static MAC Update to ICCPd Flow Diagram](images/LocalStaticMacUpdateFlow.png "Local Static MAC upstream to ICCPd Flow Diagram") + +**Figure 3: Local Static MAC upstream to ICCPd Flow Diagram** + +### 4.1.2 Local Dynamic MAC update + Below diagram shows the flow for local dynamic MAC addresses update to ICCPd. + +![Local Dynamic MAC upstream to ICCPd Flow Diagram](images/LocalDynamicMacUpstreamFlow.png "Local Dynamic MAC upstream to ICCPd Flow Diagram") + +**Figure 4: Local Dynamic MAC upstream to ICCPd Flow Diagram** + +### 4.1.3 Remote MAC Processing +Below diagram shows the flow of MAC address programing learned from ICCP. + +![Peer MCLAG MAC programming flow Diagram](images/RemoteMCLAGMacDownload.png "Peer MCLAG MAC programming flow Diagram") + +**Figure 5: Peer MCLAG MAC programming flow Diagram** + +## 4.2 Port Isolation group processing +Below diagram show the flow of Port Isolation group from App DB to SyncD + +![Port Isolation group processing](images/PortIsolationAdd.png "Port Isolation group processing") + +**Figure 6: Port Isolation group processing** + +# 5 Serviceability and Debug +- Existing mclagdctl dump commands output will be extended with additional debug state information. + +Mclagdctl dump state command is added with new field to show the sync state. +``` +Example output: +root@sonic:/home/admin# mclagdctl -i 5 dump state +The MCLAG's keepalive is: OK +**MCLAG info sync is: completed** +Domain id: 5 +Local Ip: 192.168.1.1 +Peer Ip: 192.168.1.2 +Peer Link Interface: PortChannel30 +Keepalive time: 1 +sesssion Timeout : 15 +Peer Link Mac: b8:6a:97:73:6c:96 +MCLAG System MAC : 00:11:c2:00:00:05 +Gateway Mac : 00:01:02:03:04:05 +Role: Active +MCLAG Interface: PortChannel50 +root@sonic:/home/admin# +``` + Mclagdctl is extended with new option to dump the debug counters information. + +``` +Example Output: +root@sonic:/home/admin# mclagdctl -i 5 dump debug counters +ICCP session down: 1 +Peer link down: 0 +Warmboot: 0 + +ICCP to MclagSyncd TX_OK TX_ERROR +------------------ ----- -------- +PortIsolation 13 0 +MacLearnMode 1 0 +FlushFdb 1 0 +SetIfMac 0 0 +SetFdb 226003 0 +SetL2mc 0 0 +TrafficDistEnable 1 0 +TrafficDistDisable 1 0 +SetIccpState 2 0 +SetIccpRole 1 0 +SetSystemId 0 0 +DelIccpInfo 0 0 +SetRemoteIntfSts 9 0 +DelRemoteIntf 0 0 + +MclagSyncd to ICCP RX_OK RX_ERROR +------------------ ----------- -------- +FdbChange 161616 0 +CfgMclag 1 0 +CfgMclagIface 1 0 + +ICCP to Peer TX_OK RX_OK TX_ERROR RX_ERROR +------------ ----- ----- -------- -------- +SysConfig 2 2 0 0 +AggrConfig 2 1 0 0 +AggrState 7 9 0 0 +MacInfo 160896 142183 0 0 +ArpInfo 0 0 0 0 +L2mcInfo 0 0 0 0 +PoInfo 2 1 0 0 +PeerLinkInfo 2 2 0 0 +Heartbeat 2533 2533 0 0 +Nak 0 0 0 0 +SyncData 2 2 0 0 +SyncReq 1 0 0 0 +Warmboot 0 0 0 0 +IfUpAck 4 3 0 0 + +``` + +- MCLAG debug guide can be referred for trouble shooting issues. Below is the link to debug guide. + + https://docs.google.com/document/d/1exNQ1po7TYmVtctq59aSUeBsZYKctbHMkwStNVNgSrU/edit + +# 6 Warm Boot Support +- After a warm boot is completed, the FDB table is reconciled with the MCLAG peer. ICCP is extended to advertise the local FDB table. + +# 7 Scalability +- Support scale of 40K MAC addresses. + +# 8 Upgrade/Downgrade considerations + +As part of the current enhancements MCLAG configuration can be provisioned dynamically, MclagSyncd reads the MCLAG configuration from CONFIG_DB and updates to ICCPd. The older way of reading the configuration from configdb.json file during the ICCPd startup is deprecated. + +Upgrade/downgrade from/to the older version is not supported with new enhanced version. + +# 9 Unit Test + +## 9.1 DB table updates +1. Verify that MCLAG configuration of add/delete is updated in CONFIG_DB MCLAG Domain table. +2. Verify that mclag_interface configuration add/delete is updated properly in CONFIG_DB MCLAG interface table. +3. Verify that keep-alive and session-timeout values are updated properly in CONFIG_DB table. +4. Verify that MAC address updates received from peer MCLAG nodes is updated to APP MCLAG FDB table. +5. Verify that ICCP session state information is updated to STATE_MCLAG_TABLE. +6. Verify that MCLAG interface state information is updated to STATE_MCLAG_REMOTE_INTF_TABLE +7. Verify that Unique IP configuration if add/delete is updated in CONFIG_DB MCLAG UNIQUEIP TABLE + +## 9.2 FDB + +1. Verify that MAC address learned locally on orphan port are synced to the peer MCLAG node. +2. Verify that MAC address aged locally on orphan ports are withdrawn to peer MCLAG node. +3. Verify that MAC address learned locally on MCLAG portchannel are synced to the peer MCLAG node. +4. Verify that MAC address aged locally on MCLAG portchannel ports are withdrawn to peer MCLAG node. +5. Verify that when MCLAG portchannel is down MAC addresses pointing to the MCLAG portchannel are updated to point to peer-link interface. +6. Verify that when MClag portchannel is up MAC addresses pointing to peer-link interface are updated to point to MCLAG portchannel. +7. Verify that when peer-link is down MAC addresses pointing to peer-link are deleted on both the MCLAG peer nodes. +8. Verify that when clear FDB is executed local MAC addresses are cleared locally and withdrawn to the peer MCLAG node. Relearn the MAC addresses and advertise to peer MCLAG node. +9. Verify that static MAC configured locally is advertised to peer MCLAG node. +10. Verify that static MAC deleted locally is withdrawn to peer MCLAG node. +11. Verify that in presence of remote static MAC addresses any dynamic local MAC move is not allowed. +12. Verify MAC learn with scale of 40K MAC entries. + +## 9.3 Config handling + +1. Verify MCLAG domain configuration with local, peer ip, peer interface, MCLAG interface configures ICCP +2. Verify adding MCLAG interface to existing MCLAG domain +3. Verify deleting one MCLAG interface from existing MCLAG domain +4. Verify modifying local ip configuration for an existing MCLAG domain +5. Verify modifying peer interface for given MCLAG domain +6. Verify configuration change of MCLAG keep-alive timer value to min/max values are reflected +7. Verify configuration change of MCLAG session timeout value to min/max values are reflected +8. Verify deletion of existing MCLAG domain +9. Verify adding MCLAG domain without peer interface +10. Verify adding MCLAG peer interface later after adding MCLAG domain +11. Verify deleting MCLAG interfaces with synced up MAC addresses and make sure behavior is similar to having MCLAG interface down +12. Verify Unique IP configuration on VLAN interface + +## 9.4 Port Isolation + +1. Configure Port Isolation group with members apply it to ingress port. Send unicast and BUM traffic and verify traffic is not seen at any member ports. +2. Dynamically add and delete the Port Isolation group members and verify unicast and BUM traffic again. +3. Remove Port Isolation group from ingress interface and verify traffic unicast and BUM again. +4. Send traffic from interface which doesn't have any port isolation group attached and verify unicast and BUM traffic. + +## 9.5 MCLAG Interface Flap + +1. Verify traffic is disabled until interface up ack is received from remote MCLAG peer when MCLAG interface flaps +2. Verify traffic is re-enabled on MCLAG interface which is waiting for interface up ack from peer when ICCP goes down +3. Verify traffic is disabled on a port when it is added to a MCLAG interface which is waiting for interface up ack from peer +4. Verify traffic is re-enabled on a LAG interface when it is unconfigured as MCLAG interface +5. Verify mclagdctl dump local interface output shows traffic is disabled on MCLAG interface which is waiting for interface up ack from peer + +## 9.6 MCLAG Unique IP enhancements +1. Verify ARP and ND Sync. +2. Verify IPv4 and IPv6 routing with traffic +3. Veirfy Ping / Ping6 between Peer nodes VLAN interface +4. Veirfy Ping / Ping6 between Peer node VLAN interface and MCLAG Client device +5. Verify BGP session up on VLAN interface + +## 9.7 MCLAG system MAC +1. Verify the configuration of MCLAG system MAC address add/delete. +2. Verify that MCLAG PortChannels are UP on both the MCLAG nodes using MCLAG system MAC. +3. Verify that the user configured MCLAG system MAC takes precedence and used in LACP packets for MCLAG portChannels over local system MAC. +4. Verify that Standby MCLAG node disables the MCLAG PortChannels when session is down when MCLAG system MAC is in use. +5. Verify that MCLAG PortChannels do not flap during session down and Active node reboot. +6. Verify that when MCLAG member deletes from from MCLAG domain the system MAC is used instead of the MCLAG system MAC in LACP. + +## 9.8 MCLAG Gateway MAC + +1. Verify gateway MAC configuration in both Active and Standby node. +2. Verify IPv4 and IPv6 routing with configured MAC as Gateway MAC. +3. Verify default behavior is restored on removing gateway MAC configuration. +4. Verify no traffic drop during failover scenario with gateway MAC configuration. + +# 10 Internal Design Information + +NA + diff --git a/doc/mclag/images/DesignOverviewDiagram.png b/doc/mclag/images/DesignOverviewDiagram.png new file mode 100644 index 0000000000..a918f2022e Binary files /dev/null and b/doc/mclag/images/DesignOverviewDiagram.png differ diff --git a/doc/mclag/images/LocalDynamicMacUpstreamFlow.png b/doc/mclag/images/LocalDynamicMacUpstreamFlow.png new file mode 100644 index 0000000000..df07c8db47 Binary files /dev/null and b/doc/mclag/images/LocalDynamicMacUpstreamFlow.png differ diff --git a/doc/mclag/images/LocalStaticMacUpdateFlow.png b/doc/mclag/images/LocalStaticMacUpdateFlow.png new file mode 100644 index 0000000000..d6dd8d0293 Binary files /dev/null and b/doc/mclag/images/LocalStaticMacUpdateFlow.png differ diff --git a/doc/mclag/images/MclagArpFdbKernelUpdate.png b/doc/mclag/images/MclagArpFdbKernelUpdate.png new file mode 100644 index 0000000000..2d9c970720 Binary files /dev/null and b/doc/mclag/images/MclagArpFdbKernelUpdate.png differ diff --git a/doc/mclag/images/PortIsolationAdd.png b/doc/mclag/images/PortIsolationAdd.png new file mode 100644 index 0000000000..08f11d7a3b Binary files /dev/null and b/doc/mclag/images/PortIsolationAdd.png differ diff --git a/doc/mclag/images/RemoteMCLAGMacDownload.png b/doc/mclag/images/RemoteMCLAGMacDownload.png new file mode 100644 index 0000000000..e0df5bc374 Binary files /dev/null and b/doc/mclag/images/RemoteMCLAGMacDownload.png differ diff --git a/doc/mgmt/Developer Guide.md b/doc/mgmt/Developer Guide.md new file mode 100644 index 0000000000..95c10a87bf --- /dev/null +++ b/doc/mgmt/Developer Guide.md @@ -0,0 +1,3942 @@ +# SONiC Management Framework Developer Guide + +## Rev 1.3 + +## Table of Contents + +* [List of Tables](#list-of-tables) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Definition/Abbreviation](#definitionabbreviation) +* [References](#references) +* [Prerequisites](#prerequisite) +* [1 Architecture](#1-architecture) + * [1.1 Overview](#11-overview) + * [1.2 Repositories](#12-repositories) + * [1.3 Containers](#13-containers) +* [2 Developer Workflow](#2-developer-workflow) + * [2.1 ABNF Schema](#21-abnf-schema) + * [2.2 SONiC YANGs](#22-sonic-yang) + * [2.3 OpenConfig YANGs](#23-openconfig-yang) + * [2.3.1 Directory structure](#231-directory-structure) + * [2.3.2 Extension YANGs](#232-extension-yangs) + * [2.3.3 Extension YANG Style Guide](#233-extension-yang-style-guide) + * [2.3.4 Backward Compatibility](#234-backward-compatibility) + * [2.3.5 Versioning](#235-versioning) + * [2.3.6 Tools](#236-tools) + * [2.4 Config Translation App](#24-config-translation-app) + * [2.4.1 Translation Approaches](#241-translation-approaches) + * [2.4.2 YGOT](#242-ygot) + * [2.4.3 DB Access APIs](#243-db-access-apis) + * [2.4.4 Logging](#244-logging) + * [2.4.5 Error Response](#245-error-response) + * [2.4.6 D-Bus messaging](#246-d-bus-messaging) + * [2.4.7 Redaction API](#247-redaction-api) + * [2.5 Transformer](#25-transformer) + * [2.5.1 Annotation File](#251-annotation-file) + * [2.5.2 YANG Extensions](#252-YANG Extensions) + * [2.5.3 Guidelines for Choosing Annotation Type](#253-guidelines-for-choosing-annotation-type) + * [2.5.4 Manifest file](#254-manifest-file) + * [2.5.5 Transformer Callback functions](#255-Transformer Callback functions) + * [2.5.6 Transformer Best Practices](#256-Transformer Best Practices) + * [2.5.7 Case Examples](#257-Case Examples) + * [2.5.8 Query Parameter](#258-Query Parameter) + * [2.6 App Module](#26-app-module) + * [2.7 Config Validation Library](#27-config-validation-library) + * [2.7.1 CVL Schema](#271-cvl-schema) + * [2.7.2 Custom Validation](#272-custom-validation) + * [2.7.3 Platform Validation](#273-platform-validation) + * [2.7.4 CVL Debug Logs](#274-cvl-debug-logs) + * [2.8 Non-DB Data](#28-non-db-data) + * [2.8.1 Host Modules](#281-host-modules) + * [2.8.2 FRR Integration](#282-frr-integration) + * [2.9 KLISH CLI](#29-klish-cli) + * [2.9.1 CLI components](#291-cli-components) + * [2.9.2 CLI development steps](#292-cli-development-steps) + * [2.9.3 Enhancements to Klish](#293-enhancements-to-klish) + * [2.9.4 Preprocess XML files](#294-preprocess-xml-files) + * [2.9.5 CLI directory structure](#295-cli-directory-structure) + * [2.9.6 Generic REST Client](#296-generic-rest-client-for-actioner-scripts) + * [2.9.7 Tool for JSON navigation in jinja templates](#297-tool-for-json-navigation-in-jinja-templates) + * [2.10 REST Server](#210-rest-server) + * [2.11 gNMI](#211-gnmi) + * [2.12 Compilation](#212-compilation) + * [2.12.1 sonic-mgmt-common](#2121-sonic-mgmt-common) + * [2.12.2 sonic-mgmt-framework](#2122-sonic-mgmt-framework) + * [2.12.2 sonic-telemetry](#2122-sonic-telemetry) + * [2.13 Unit Testing](#213-unit-testing) + * [2.13.1 REST API testing](#2131-rest-api-testing) + * [2.13.2 Local REST Server and CLI](#2132-local-rest-server-and-cli) + * [2.13.3 gNMI Unit Testing](#2133-gnmi-unit-testing) + * [2.13.3 Spytest](#2133-spytest) +* [3 Appendix](#3-appendix) + * [3.1 Code Generators](#31-code-generators) + * [3.2 Debugging](#32-debugging) + * [3.2.1 Debugging performance issues](#321-debugging-performance-issues) + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +| :--: | :--------: | :---------------------: | ------------------------------------------------------------ | +| 0.1 | 09/12/2019 | Anand Kumar Subramanian | Initial version | +| 0.2 | 09/16/2019 | Prabhu Sreenivasan | Added references, prerequisites and updated section 2.1.2 SONiC YANG | +| 0.3 | 09/16/2019 | Partha Dutta | Updated SONiC YANG Guideline link | +| 0.4 | 09/18/2019 | Anand Kumar Subramanian | Updated transformer section | +| 0.5 | 09/20/2019 | Mohammed Faraaz C | Updated REST Unit testing | +| 0.6 | 09/20/2019 | Prabhu Sreenivasan | Updated reference links and yang path | +| 0.7 | 09/23/2019 | Partha Dutta | Updated SONiC YANG, CVL, gNMI section | +| 0.8 | 06/18/2019 | Kwangsuk Kim | Updated CLI section | +| 0.9 | 06/22/2020 | Sachin Hola | Incorporated Phase-II changes, with new repo structure. | +| 1.0 | 10/09/2020 | Anand Kumar Subramanian | Debugging performance issues | +| 1.1 | 08/10/2021 | Kwangsuk Kim | Updated Transformer section including query parameter etc. | +| 1.2 | 12/10/2021 | Arun Barboza | Added Pruning API description. | +| 1.3 | 12/28/2021 | Arun Barboza | Added Redaction API description. | +| 1.4 | 12/28/2021 | Balachandar Mani | Added details for YGOT section | + +## About this Manual + +This document provides developer guidelines for Management framework feature development in SONiC. + +## Scope + +This document describes the steps the feature developers need to follow to develop a CLI, REST and gNMI for a given feature using the Management framework. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| CVL | Config Validation Library | +| NBI | North Bound Interface | +| ABNF | Augmented Backus-Naur Form | +| YANG | Yet Another Next Generation | +| JSON | Java Script Object Notation | +| XML | eXtensible Markup Language | +| gNMI | gRPC Network Management Interface | +| YGOT | YANG Go Tools | +| SONiC YANG | YANG file which describes Redis DB schema for a given feature | +| RPC | Remote Procedure Call | + +## References + +| Document | Link | +|:--------:|:-------| +| SONiC YANG model guideline | https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md | +| SONiC Management Framework HLD | https://github.com/Azure/SONiC/pull/436 | +| RFC 7950 | August 2016 | https://tools.ietf.org/html/rfc7950 | + +## Prerequisite + +* Knowledge of YANG syntax +* GO Programming language +* SONiC Management Framework Architecture + +## 1 Architecture + +### 1.1 Overview + +![Management Framework Architecture diagram](https://github.com/Azure/SONiC/blob/34cac1aabdc865fc41cbe064a2ab2442645524b1/doc/mgmt/images/Mgmt_Frmk_Arch.jpg) + +In the above architecture diagram developer needs to write all the elements in green. Following are blocks that the developer has to write in order to get CLI, REST and gNMI support for the new feature being developed. + +1) YANG Models + +2) ABNF schema and SONiC YANG + +3) Transformer - includes YANG annotations and custom translation + +4) KLISH XML including Actioner and Renderer scripts + +### 1.2 Repositories + +Management framework and application code are distributed across 3 repositories. + +1) **sonic-mgmt-common** + + - Translib, CVL and transformer infrastructure code + - SONiC YANGs + - OpenConfig, IETF YANGs and their extension YANGs + - Trabsformer annotations and custom transformer functions + - CVL custom validation functions + - Host modules (application logic that runs on switch host) + +2) **sonic-mgmt-framework** + + - YANG to OpenAPI converter and OpenAPI codegen tools + - REST Server + - KLISH CLI core and SONiC specific patches + - CLI models (KLISH XML) + - CLI actioner scripts + - CLI renderer templates + +3) **sonic-telemetry** + + - Telemetry server + +All YANG data models and their translation logic goes into sonic-mgmt-common repository. +sonic-mgmt-framework adds REST and CLI interfaces for these YANG data models. +sonic-telemetry adds gNMI interface. + +### 1.3 Containers + +There will be 2 docker containers providing management services. + +1) **mgmt-framework** -- provides REST Server and CLI +2) **telemetry** -- Telemetry service + +## 2 Developer Workflow + +Following are the steps to be followed by the application developer. + +### 2.1 ABNF Schema + +First step is to define the redis DB schema for the application/feature. +Usual SONiC practice is to express the database schema in ABNF. +An existing DB table can be modified or new tables may be introduced. + +If you are defining new tables, try to keep it in line with the OpenConfig YANG model +for the feature (explained in next steps). Usually a redis table maps to YANG 'list' node. +Try to avoid mapping one redis table to multiple YANG lists. +This will make translation logic complex and requires custom translation code. + +### 2.2 SONiC YANG + +SONiC YANGs express the redis db schema in YANG format. +These YANGs should strictly adhere to the [SONiC YANG Model Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) published by SONiC Community. + +SONiC YANG models **MUST** be defined for every DB schema consumed by the management applications. +CVL, transformer and other framework components use these YANGs to understand the DB schema. + +SONiC YANGs are maintained in `sonic-mgmt-common/models/yang/sonic` directory. + +### 2.3 OpenConfig YANG + +Every feature should support corresponding OpenConfig YANG model. +An OpenConfig compliant YANG model should be defined if there was no standard OpenConfig YANG. +Extension YANGs should be written to add additional data nodes over standard YANG and +to remove unsupported nodes. + +Few existing features implemented IETF YANGs. However going forward only OpenConfig YANGs +and OpenConfig compliant extension YANGs should be used. + +#### 2.3.1 Directory structure + +All YANGs are maintained under `sonic-mgmt-common/models/yang` directory. + +``` +models +└── yang : Standard OpenConfig and IETF YANGs + ├── annotations : Transformer annotations (section 2.5.2) + ├── common : Dependencies for standard YANGs + ├── extensions : Extensions YANGs + ├── sonic : SONiC YANGs (section 2.2) + ├── testdata : Test YANGs - ignored + └── version.xml : YANG bundle version configuration file +``` + +Unmodified OpenConfig and IETF YANGs should be kept in `yang` directory. +Framework will automatically enable REST and gNMI interfaces for these YANGs. + +These YANGs refer few common YANG models, like ietf-interfaces.yang, for which we dont want +north bound interfaces. Such dependent YANGs must be kept in `yang/common` directory. +Framework will use these YANGs only to resolve compilation dependencies but not enable +REST and gNMI interaces for them. + +All custom YANGs should be kept in `yang/extensions` directory. + +#### 2.3.2 Extension YANGs + +**Standard OpenConfig and IETF YANGs must not be modified directly**. +All customizations must be applied through extension YANGs. +Even new OpenConfig compliant YANGs (when there is no standard YANG) will also +be treated as an extension to OpenConfig. + +Extension YANG should use YANG [augment](https://tools.ietf.org/html/rfc6020#section-4.2.8) +statements to add new data nodes to standard OpenConfig or IETF YANG data tree. + +YANG [deviation](https://tools.ietf.org/html/rfc6020#section-7.18.3) statement should +be used to mark an existing data node as "not-supported" or to modify it. + +Example: + +```yang +module openconfig-acl-ext { + namespace "http://github.com/Azure/oc-acl-ext"; + prefix oc-acl-ext; + + import openconfig-acl { prefix oc-acl; } + + /* Add an alt-name config to ACL */ + augment "/oc-acl:acl/acl-sets/acl-set/config" { + leaf alt-name { + type string; + } + } + + /* Mark ACL description not supported */ + deviation "/oc-acl:acl/acl-sets/acl-set/config/description" { + deviate not-supported; + } + + /* Restrict ACL name to max 127 characters */ + deviation "/oc-acl:acl/acl-sets/acl-set/config/name" { + deviate replace { + type string { + length "1..127"; + } + } + } +} +``` + +#### 2.3.3 Extension YANG Style Guide + +OpenConfig extension YANGs should adhere to [OpenConfig style guidelines](https://github.com/openconfig/public/blob/master/doc/openconfig_style_guide.md). + +Please review [current model hierarchy](http://www.openconfig.net/projects/models) +and IETF draft [draft-openconfig-netmod-opstate-01](https://tools.ietf.org/html/draft-openconfig-netmod-opstate-01) +on overall OpenConfig modeling approach. + +General guidelines for extension YANGs: + +- All extension YANGs should be named as **openconfig-{function}-ext.yang**. + +- **namespace** should be "http://openconfig.net/yang/{function}-ext" or "http://openconfig.net/yang/{function}/extension" + +- Namespace **prefix** should be "oc-{function}-ext". + +- Specify **organization**, **contact** and **oc-ext:origin** as "SONiC". + +- Add module **description** statement with new features covered in the extension yang; +like [this](https://github.com/openconfig/public/blob/master/release/models/policy/openconfig-routing-policy.yang#L24) + +- Include **revision** and **oc-ext:openconfig-version** statements. + - oc-ext:openconfig-version should be "0.1.0" for the initial version (when a new extension YANG is defined) + - Update the oc-ext:openconfig-version and add a revision statement for every enhancement to the mode. + An enhancement may be split into multiple commits within a release. + But all such related commits (within a release) should be grouped as one version change. + - Version updates should follow [sematic versioning](https://semver.org/#semantic-versioning-specification-semver) + guidelines. More details are at [section 2.3.5](#235-versioning) + - Revision statements should be reverse chronological order (latest should be at the top). + +- Do not use tab characters. Always expand tabs into 4 spaces. + +- Try to maintain 72 column width through-out the file (like RFC document) + +Naming conventions: + +- Use **lower case with dash separated words** for all YANG node names (container, list, leaf-list, leaf, grouping etc). + Example: "ip-address" + + Do not use upper case, underscores, camel case; like "IP_Address" or "ipaddress" or "ipAddress". + +- Use **upper case with underscore separated words** for enum and identity names. +Example: "IF_ETHERNET". + +- Try to avoid cryptic and abbreviated words, unless they are widely used feature names. + + Example: "arp", "mpls", "fbs" are okay. But avoid names like "addr", "ip-addr", "neigh". + +- Grouping names should make it easy to quickly understand the nature of the data within. +A suggested convention is ***xxx-yyy[-config|state|top]***, where xxx is the top-level module name +(without the openconfig prefix), yyy is a string which indicates the contents of the groupings. + + Example: "interface-common-state", "subinterfaces-config". + +- Use standard typedefs as much as possible. + Most of the common data types are already defined in one of these YANGs. + - openconfig-types.yang + - openconfig-inet-types.yang + - openconfig-yang-types.yang + - ietf-inet-types.yang + - ietf-yang-types.yang + + Also look for openconfig-{feature}-types.yang for the feature specific typedefs. + +Data tree guidelines: + +- TODO + +#### 2.3.4 Backward Compatibility + +All extensions to standard OpenConfig/IETF YANGs or incremental changes to the extension +YANGs can affect REST and gNMI clients. Strict adherence to +[YANG module update guidelines](https://tools.ietf.org/html/rfc6020#section-10) is a must. + +#### 2.3.5 Versioning + +Individual YANG models should be versioned through **revision** and **oc-ext:openconfig-version** statements. +In addition, a YANG bundle version (collective version number for all supported YANG modules) +is managed via `models/yang/version.xml` file. +YANG bundle version is used for API versioning feature (in future release). + +YANG bundle version uses sematic versioning format -- ie, **{major}.{minor}.{patch}** format. +It should be updated for every YANG update or once in a release. + +**Major version** should be incremented if YANG model is changed in a non backward compatible manner. +Such changes should be avoided. + +- Delete, rename or relocate data node +- Change list key attributes +- Change data type of a node to an incompatible type +- Change leafref target + +**Minor version** should be incremented if the YANG change modifies the API in a backward +compatible way. Patch version should be reset to 0. +Candidate YANG changes for this category are: + +- Add new YANG module +- Add new YANG data nodes +- Mark a YANG data node as deprecated +- Change data type of a node to a compatible type +- Add new enum or identity + +**Patch version** should incremented for cosmetic fixes that do not change YANG API. +Candidate YANG changes for this category are: + +- Change description, beautification. +- Expand pattern or range of a node to wider set. +- Change must expression to accept more cases. +- Error message or error tag changes. + +#### 2.3.6 Tools + +Framework provides multiple tools and code generators to work with the YANGs. +Details of each code generators is in [section 3.1](#31-code-generators). +All the tools are automatically run during sonic-mgmt-common build. +They can also be triggered manually to validate YANG changes. + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-common +make models +``` + +It validates all the YANGs under `yang` directory using [pyang tool](http://www.yang-central.org/twiki/pub/Main/YangTools/pyang.1.html) and generates human readable reports. + +##### 2.3.6.1 YANG Text Tree + +This is the standard pyang text tree output generated through "-f tree" option. +Two files are generated: + +1) `sonic-mgmt-common/models/yang/allyangs.tree` -- +Combined tree for the OpenConfig and IETF YANGs and their extension YANGs + +2) `sonic-mgmt-common/models/yang/sonic/allyangs.tree` -- +Combined tree for all SONiC YANGs + +Sample: + +```text +module: openconfig-lldp + +--rw lldp + +--rw config + | +--rw enabled? boolean + | +--rw hello-timer? uint64 + | +--rw suppress-tlv-advertisement* identityref + | +--rw system-name? string + | +--rw system-description? string + | +--rw oc-lldp-ext:multiplier? uint8 + | +--rw oc-lldp-ext:mode? lldp-ext-mode-type +``` + +##### 2.3.6.2 YANG HTML Tree + +This is also a standard pyang tree output generated through "-f jstree" option. +It is equivalent to the text tree but provides expand/collapse controls at each node. + +Two files are generated: +1) `sonic-mgmt-common/models/yang/allyangs_tree.html` +2) `sonic-mgmt-common/models/yang/sonic/allyangs_tree.html` + +##### 2.3.6.3 YANG Markdown Tree + +Markdown tree is similar to text tree but contains markdown annotations to highlight +the nodes added and removed though extension YANGs. +Newly added nodes (through augment statements) are highlighted in green colour. +Removed nodes (through "not-supported" deviation) are highlighted in red color. +This is equivalent of diff between standard YANG data tree and the final YANG tree +after applying all extension YANGs. + +Markdown tree file path: `sonic-mgmt-common/models/yang/yang_tree.md` + +Sample: + +```diff + module: openconfig-lldp + +--rw lldp + +--rw config + | +--rw enabled? boolean + | +--rw hello-timer? uint64 + | +--rw suppress-tlv-advertisement* identityref + | +--rw system-name? string + | +--rw system-description? string ++ | +--rw oc-lldp-ext:multiplier? uint8 ++ | +--rw oc-lldp-ext:mode? lldp-ext-mode-type +- | +--rw chassis-id? string +- | +--rw chassis-id-type? oc-lldp-types:chassis-id-type +``` + +##### 2.3.6.3 OpenConfig Style Checker + +TODO + + +### 2.4 Config Translation App + +Config Translation App translates data from northbound API schema to native +[ABNF schema](#21-abnf-schema) and vice versa. +Northbound API schema are defined by OpenConfig or IETF YANGs and ther extensions. + +#### 2.4.1 Translation Approaches + +There are two approaches to develop the translation app. + +1. [Transformer](#25-transformer) +2. [App Module](#26-app-module) + +Transformer is the preferred approach for develpoing the translation app. +It reduces the coding and maintenance effort by hiding the most of the YANG +hierarchy tranversal in the framework itself. +New features can be introduced into Translib without affecting existing translation apps. + +An App Module would receive the raw inputs passed by the clients and shloud code for every possible scenario. + +Type of translation app can be selected per YANG module. +New additions to an existing YANG module should use the current translation approach used by that YANG. +It is not possible to support part of the YANG tree through App Modules and part through Transformer. + +All new YANG modules should use transformer. + +#### 2.4.2 YGOT +To showcase the YGOT generated Go structure for the given yang model, and how to create the instance of Go structure and filling the values in it. + +1. openconfig-sample-interfaces.yang: + + This is the Sample YANG module (openconfig-sample-interfaces) for which the Go structure will be generated + to demonstrate how to use the Go strucuture to fill the values of the types defined in the yang model. + +``` + module openconfig-sample-interfaces { + + yang-version "1.1"; + + namespace "http://openconfig.net/yang/sample-interfaces"; + + prefix "oc-sample-if"; + + import ietf-interfaces { prefix ietf-if; } + + organization "Sonic sample interfaces yang"; + + contact + "SONiC"; + + description + "Openconfig sample interfaces yang"; + + grouping sample-interface-config { + description "sample interface config grouping"; + leaf name { + type string; + description + "The name of the interface."; + } + leaf type { + type identityref { + base ietf-if:interface-type; + } + description + "The type of the interface."; + } + leaf mtu { + type uint16; + description + "sample mtu"; + } + } + + grouping sample-interfaces-top { + description + "Top-level sample interface grouping"; + + container sample-interfaces { + description + "Top level container for sample interfaces."; + + list sample-interface { + key "name"; + description + "list of sample yang interfaces."; + + leaf name { + type leafref { + path "../config/name"; + } + description + "sample interface name."; + } + + container config { + description + "Configurable items"; + uses sample-interface-config; + } + + container state { + config false; + description + "Sample interface state"; + uses sample-interface-config; + } + } + } + } + uses sample-interfaces-top; +} +``` + +2. Generated Go structure for the YANG model "openconfig-sample-interfaces.yang": + + This is the generated Go structure for the given Sample YANG model and its structure's hierarchy is similar to the given YANG model "openconfig-sample-interfaces.yang". + + Note: OpenconfigSampleInterfaces_SampleInterfaces represents the /openconfig-sample-interfaces/sample-interfaces YANG schema element. + ``` + type OpenconfigSampleInterfaces_SampleInterfaces struct { + SampleInterface map[string]*OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface `path:"sample-interface" module:"openconfig-sample-interfaces"` + } + ``` + + + Note: NewSampleInterface creates a new entry in the SampleInterface list of the OpenconfigSampleInterfaces_SampleInterfaces struct. + The keys of the list are populated from the input arguments. + ``` + func (t *OpenconfigSampleInterfaces_SampleInterfaces) NewSampleInterface(Name string) (*OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface, error){ + + // Initialise the list within the receiver struct if it has not already been + // created. + if t.SampleInterface == nil { + t.SampleInterface = make(map[string]*OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface) + } + + key := Name + + // Ensure that this key has not already been used in the + // list. Keyed YANG lists do not allow duplicate keys to + // be created. + if _, ok := t.SampleInterface[key]; ok { + return nil, fmt.Errorf("duplicate key %v for list SampleInterface", key) + } + + t.SampleInterface[key] = &OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface{ + Name: &Name, + } + + return t.SampleInterface[key], nil + } + ``` + + + Note: OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface represents the /openconfig-sample-interfaces/sample-interfaces/sample-interface YANG schema element. + ``` + type OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface struct { + Config *OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_Config `path:"config" module:"openconfig-sample-interfaces"` + Name *string `path:"name" module:"openconfig-sample-interfaces"` + State *OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_State `path:"state" module:"openconfig-sample-interfaces"` + } + ``` + + + Note: OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_Config represents the /openconfig-sample-interfaces/sample-interfaces/sample-interface/config YANG schema element. + ``` + type OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_Config struct { + Mtu *uint16 `path:"mtu" module:"openconfig-sample-interfaces"` + Name *string `path:"name" module:"openconfig-sample-interfaces"` + Type E_IETFInterfaces_InterfaceType `path:"type" module:"openconfig-sample-interfaces"` + } + ``` + + + Note: OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_State represents the /openconfig-sample-interfaces/sample-interfaces/sample-interface/state YANG schema element. + ``` + type OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_State struct { + Mtu *uint16 `path:"mtu" module:"openconfig-sample-interfaces"` + Name *string `path:"name" module:"openconfig-sample-interfaces"` + Type E_IETFInterfaces_InterfaceType `path:"type" module:"openconfig-sample-interfaces"` + } + ``` + + +3. The mapping between the yang model hierarchy and Go structure hierarchy is depicted below: + + | **"openconfig-sample-interfaces.yang" model** | **Go Structure** | + |------------------------------------------------------|---------------------------------------------------------------------| + | container: sample-interfaces | OpenconfigSampleInterfaces_SampleInterfaces | + | list: sample-interface | OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface | + | --- leaf: name | Name | + | --- container: onfig | OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_Config | + | ------ leaf: name | Name | + | ------ leaf: type | Type | + | ------ leaf: mtu | Mtu | + | --- container: state | OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_State | + | ------ leaf: name | Name | + | ------ leaf: type | Type | + | ------ leaf: mtu | Mtu | + + +4. Fill the Go structure with the data values for the sample yang: + a. Initialize the top level container node's (sample-interfaces) Go struct (OpenconfigSampleInterfaces_SampleInterfaces) + ``` + sampleInterfaces := OpenconfigSampleInterfaces_SampleInterfaces{} + ``` + + b. If needed, using the top level Go struct, initialization of any non-list child node can be done using the Ygot function ygot.BuildEmptyTree(type). + This function will create and initialize the child level non-list node's Go struct with default value of the type + ``` + ygot.BuildEmptyTree(sampleInterfaces) + ``` + + c. To fill the list node (sample-interface) which present inside the container node (sample-interfaces), Go function "NewSampleInterface" can be used to create an instance of the list node which will return the instance of the instance list node's Go struct "OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface". + ``` + sampleIntf := sampleInterfaces.NewSampleInterface("TestInterface-1") + ``` + + d. To fill the container node ("config") inside the list node (sample-interface), the corresponding Go struct should be instantiated with filing the values of the fields and assigned to its field of the struct "OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_Config" + ``` + sampleIntf.Config := OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_Config{} + intfKey := "TestInterface-1" + sampleIntf.Config.Name = &intfKey + sampleIntf.Config.Type = IETFInterfaces_InterfaceType_IF_NVE + intfMtu := 200 + sampleIntf.Config.Mtu = &intfMtu + ``` + e. Container node "state" will be same as the way "config" node is being filled as mentioned in the previous step, that is to fill the + container node ("state") inside the list node (sample-interface), the corresponding Go struct should be instantiated with filing the values + of the fields and assigned to its field of the struct "OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_State" + ``` + sampleIntf.State := OpenconfigSampleInterfaces_SampleInterfaces_SampleInterface_State{} + intfKey := "TestInterface-1" + sampleIntf.State.Name = &intfKey + sampleIntf.State.Type = IETFInterfaces_InterfaceType_IF_NVE + intfMtu := 200 + sampleIntf.State.Mtu = &intfMtu + ``` + +#### 2.4.3 DB Access APIs + +The DB access layer implements a wrapper over the go-redis package enhancing the functionality as described in: +[Management Framework HLD, DB access layer](https://github.com/Azure/SONiC/blob/master/doc/mgmt/Management%20Framework.md#32264-db-access-layer) + +#### 2.4.4 Logging + +Translation apps should use Go "github.com/golang/glog" library for logging. + +Guidelines: +- Use `glog.Infof()` for logging info messges. Info is enabled by default. +- Use `glog.Errorf()` for logging error messages. +- Use `glog.V(1).Infof()` for debug messages. +Typically, payloads or DB data should be logged this way. +This style of logging is called verbose logging and they are not enabled by default. +Program should be started with a command line argument "-v" to enable verbosity levels. +- Use `glog.V(2).Infof()` for more verbose developer traces. +- Avoid using higer verbosity levels, fatal logs, `fmt.Printf()` and other logging libraries. +- Avoid logging sensitive information (like passwords, keys, ...). Please see [section 2.4.7](#247-redaction-api) for help with redacting logs. + +Examples: + +```go +import ( + "github.com/golang/glog" +) + +// INFO log +glog.Infof("Processing ACL %s, %v", aclName, aclType) + +// ERROR log +glog.Errorf("GetEntry returned error %v", err) + +// DEBUG log +glog.V(1).Infof("Payload received = %v", data) + +// DEBUG log which requires lot of formatting. +if glog.V(1) { + s := doNiceFormatting(bigdata) + glog.Infof("data = %s", s) +} +``` + +Log verbosity level for REST and gNMI servers can be changed via CONFIG_DB entries. +Details are in [Management Framework HLD](https://github.com/project-arlo/SONiC/blob/master/doc/mgmt/Management%20Framework.md#322414-db-schema). + +#### 2.4.5 Error Response + +Translation apps should return Go error objects defined in +`sonic-mgmt-common/translib/tlerr/app_errors.go` file to report errors. +Translib infrastructure stops further processing of the request and returns the error +back to the service (REST or gNMI) which invoked the Translib APIs. +Protocol specific error response will be prepared by individual service. + +Application error types: + +| Error object | Error condition | Constructor function | +|--------------------|--------------------------------------|---------------------------| +| InvalidArgsError | Bad request, constraint error | `tlerr.InvalidArgs()` | +| NotFoundError | Target resource not found | `tlerr.NotFound()` | +| AlreadyExistsError | Dupicate create | `tlerr.AlreadyExists()` | +| NotSupportedError | Unspported operation | `tlerr.NotSupported()` | +| InternalError | Generic error during app execution | `tlerr.New()` | +| AuthorizationError | User is not authorized for the operation | | + +Exampels: + +```go +import ( + "github.com/Azure/sonic-mgmt-common/translib/tlerr" +) + +// InvalidArgsError +return tlerr.InvalidArgs("Priority value %d is out of range[1-65535]", priority) + +// NotFoundError +return tlerr.NotFound("ACL %s/%v not found", aclName, aclType) + +// AlreadyExistsError +return tlerr.AlreadyExists("Duplicate ACL %s/%v", aclName, aclType) + +// NotSupportedError +return tlerr.NotSupported("Cannot delete Ethernet interface") + +// NotSupportedError +return tlerr.New("FRR returned %d", status) +``` + +Avoid returning other error objects created through `errors.New()` or `fmt.Errorf()`. +Translib will propagate such errors but REST/gNMI service cannot map them to correct protocol error codes. + +Error types defined in `sonic-mgmt-common/translib/tlerr/tlerr.go` are reserved for Translib's internal use. + +#### 2.4.6 D-Bus messaging + +TODO + +#### 2.4.7 Redaction API + +Ideally, sensitive information like passwords, protocol-keys, and confidential data should not be logged to debug log files. To this end, all developers must strive to avoid logging such material intentionally. On the other hand, debugging requires that information be logged to the debug log files. To this end, we'd like to propose that at default debugging log levels no sensitive data be logged. + +Sometimes, this data comes from the user and, at the time it is logged, hasn't been identified as sensitive information. The Redaction API provides an interface, for developers implementing translib golang functions, to log such data with redaction. + +Example Usage: + +``` + +... + +log.Info("Update request received with payload =", + string((utils.GRedactor.DoRedact(payload, nil, nil)).([]byte))) + +... + +``` + +Example Log: + +``` + +... + +IDec 28 22:11:26.904868+00:00 2021 20 translib.go:311] Update request received with payload =REDACTED: {"openconfig-system:server-groups": {"openconfig-system:server-group": [{"openconfig-system:name": "RADIUS", "openconfig-system:config": {"openconfig-system:name": "RADIUS", "openconfig-system:secret-key": "************", "openconfig-system:encrypted": false}}]}} + +... + +``` + +Limitations: + +- The Redaction API currently has support for JSON payloads only (but it can be extended, if needed). +- Developers need to provide the list (or patterns) of sensitive JSON keys. A starter set of keys has been provided (but it can be extended, trivially, if needed). + + +### 2.5 Transformer + +Transformer provides a generic infrastructure for Translib to programmatically translate YANG to ABNF/Redis schema and vice versa, using YANG extensions to define translation hints along the YANG paths. At run time, the translation hints are mapped to an in-memory Transformer Spec that provides two-way mapping between YANG and ABNF/Redis schema for Transformer to perform data translation. + +In case that SONiC YANG modules are used by NBI applications, the Transformer performs 1:1 mapping between a YANG object and a SONiC DB object without a need to write special translation codes or any translation hints. + +If you use the openconfig YANGs for NBI applications, you need special handling to translate the data between YANG and ABNF schema. In such case, you have to annotate YANG extensions and write callbacks to perform translations where required. + +In either case, the default application [common-app.go](https://github.com/project-arlo/sonic-mgmt-framework/blob/transformer-phase1/src/translib/common_app.go) generically handles both set and get requests with the returned data from Transformer. + +Basic worklflow for developing a Translation App using transformer: + +1) Create an annotation file template (for new YANGs) +2) Fill transformer annotations for YANG paths in the annotation file. +3) Register YANG and annotation file names in the manifest file (for new YANGs) +4) Implement the transformer callback functions if any. + +#### 2.5.1 Annotation File + +Annotation file contains translation hints in YANG format. +It uses YANG deviation statements to attach transformer annotation to specific YANG paths. +However these deviations do not affect the data model. +It only attaches additional metdata to YANG paths for Transformer's use. + +All annotation files should be kept in `sonic-mgmt-common/models/yang/annotations` directory. + +A template annotation file can be generated using `sonic-mgmt-common/tools/xfmr/annotate.sh` script. +It takes one or more YANG file names and writes the annotation template file contents to stdout. +Output can be redirected to specific annotation file under `yang/annotations` directory. +Template file contains empty "deviate add" statements for all YANG paths from the specified YANG files. + +```bash +$ cd /sonic/sonic-buildimage/src/sonic-mgmt-common +$ tools/xfmr/annotate.sh +usage: tools/xfmr/annotate.sh YANG_FILE_NAME... + +$ tools/xfmr/annotate.sh openconfig-lldp.yang + ..... +``` + +#### 2.5.2 YANG Extensions + +Transformer provides custom YANG statements to specify translation hints. +They are defined in `sonic-mgmt-common/models/yang/annotations/sonic-extensions.yang` file. +These custom YANG statements should be used to annotate specific paths in the annotation file. + +Example: + +```yang +import sonic-extensions { prefix sonic-ext; } + +deviation /oc-lldp:lldp/oc-lldp:config { + deviate add { + sonic-ext:table-name "LLDP"; + sonic-ext:key-transformer "lldp_global_key_xfmr"; + } +} + +deviation /oc-lldp:lldp/oc-lldp:config/oc-lldp:hello-timer { + deviate add { + sonic-ext:field-name "hello_time"; + } +} +``` + +Following annotation statements are supported. + +1. ##### sonic-ext:table-name {TABLE_NAME} + + Maps a YANG container or list to redis table name. + + The table-name is **inherited** to all descendant nodes unless another one is defined. + +2. **sonic-ext:field-name {FIELD_NAME}** + + Maps a YANG leaf or leaf-list node to redis hash field name. + +3. **sonic-ext:key-delimiter {DELIM}** + + Override the default delimiter "|" with a custom delimiter. + This is used to derive the redis table key by concatenating the table name and key components. + +4. **sonic-ext:key-name {VALUE}** + + Specify the fixed redis key value for singleton tables. + Useful when top level YANG container is mapped to a redis table. + + Example: below annotation results is redis key "UDLD|GLOBAL". + + ``` + deviate add { + sonic-ext:table-name "UDLD"; + sonic-ext:key-name "GLOBAL"; + } + ``` + +5. **sonic-ext:key-transformer {FUNCTION}** + + Overloads the default method to generate DB key(s). + Used when the key values in a YANG list are different from ones in DB table. + + A pair of Go callbacks should be implemented to support 2 way translation - YangToDB_xxx, DbToYang_xxx. + Function signatures: + + ```go + /** + * KeyXfmrYangToDb type is defined to use for conversion of Yang key to DB Key + * + * Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath + * Return: Database keys to access db entry, error + **/ + type KeyXfmrYangToDb func (inParams XfmrParams) (string, error) + + /** + * KeyXfmrDbToYang type is defined to use for conversion of DB key to Yang key + * + * Param: XfmrParams structure having Database info, operation, Database keys to access db entry + * Return: multi dimensional map to hold the yang key attributes of complete xpath, error + **/ + type KeyXfmrDbToYang func (inParams XfmrParams) (map[string]interface{}, error) + ``` + +6. **sonic-ext:field-transformer {FUNCTION}** + + Overloads default method for generating DB field value. + Used when the leaf/leaf-list values defined in a YANG list are different from the field values in DB. + + A pair of callbacks should be implemented to support 2 way translation - YangToDB_xxx, DbToYang_xxx. + Function signatures: + + ```go + /** + * FieldXfmrYangToDb type is defined to use for conversion of yang Field to DB field + * + * Param: Database info, YgotRoot, operation, Xpath + * Return: multi dimensional map to hold the DB data, error + **/ + type FieldXfmrYangToDb func (inParams XfmrParams) (map[string]string, error) + + /** + * FieldXfmrDbtoYang type is defined to use for conversion of DB field to Yang field + * + * Param: XfmrParams structure having Database info, operation, + * DB data in multidimensional map, output param YgotRoot + * Return: error + **/ + type FieldXfmrDbtoYang func (inParams XfmrParams) (map[string]interface{}, error) + ``` + +7. **sonic-ext:table-transformer {FUNCTION}** + + Map a YANG container/list to TABLE name(s). + Used to dynamically map a YANG list/container to table names based on URI and payload. + + The table-transformer is **inherited** to all descendant nodes until another one is defined + or table-name “NONE” is defined. + + This callback can also be used to populate the + + Table transformer function signature: + + ```go + /** + * TableXfmrFunc type is defined to use for table transformer function for + * dynamic derviation of redis table. + * + * Param: XfmrParams structure having database pointers, current db, operation, + * DB data in multidimensional map, YgotRoot, uri + * Return: List of table names, error + **/ + type TableXfmrFunc func (inParams XfmrParams) ([]string, error) + ``` + +8. **sonic-ext:subtree-transformer {FUNCTION}** + + Overloading default translation method for the current subtree. + Allows the callback function to take **full control of translation**. + Note that, if any other extensions are annotated to the nodes on the subtree, they are not effective. + + The subtree-transformer is **inherited** to all descendant nodes unless another one is defined. + i.e. the scope of subtree-transformer callback is limited to the current and descendant nodes + along the YANG path until a new subtree transformer is annotated. + + A pair of callbacks should be implemented to support 2 way translation - YangToDB_xxx, DbToYang_xxx. + Function signatures: + + ```go + /** + * SubTreeXfmrYangToDb type is defined to use for handling the yang subtree to DB + * + * Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath + * Return: multi dimensional map to hold the DB data, error + **/ + type SubTreeXfmrYangToDb func (inParams XfmrParams) (map[string]map[string]db.Value, error) + + /** + * SubTreeXfmrDbToYang type is defined to use for handling the DB to Yang subtree + * + * Param : XfmrParams structure having Database pointers, current db, operation, + * DB data in multidimensional map, output param YgotRoot, uri + * Return : error + **/ + type SubTreeXfmrDbToYang func (inParams XfmrParams) (error) + ``` + +9. **sonic-ext:db-name {NAME}** + + Specify DB name to access data from. + Valid values are: + + - CONFIG_DB (default value) + - APPL_DB + - ASIC_DB + - COUNTERS_DB + - FLEX_COUNTER_DB + - STATE_DB + + Used for GET operation to non CONFIG_DB, applicable only to SONiC YANG + + The db-name is **inherited** to all descendant nodes unless another one. + It must be defined with the **table-name** annotation. + +10. **sonic-ext:pre-transformer {FUNCTION}** + + A pre-transformer hook is invoked at the start of the translation within the transaction. This callback will be invoked after resource check for the requested URI. Pre-transformer hook can be used to support certain cases like accessing non-DB data source before proceeding to main translation process; It is registered only to the top-level container within a YANG module and can be invoked once per transaction. The following function template definition is used for pre-transformer callback functions. + + ``` + // PreXfmrFunc type is defined to use for handling any default handling operations required as part of the CREATE, UPDATE, REPLACE, DELETE & GET + // Transformer function definition. + // Param: XfmrParams structure having database pointers, current db, operation, DB data in multidimensional map, YgotRoot, uri + // Return: error + type PreXfmrFunc func (inParams XfmrParams) (error) + ``` + +11. **sonic-ext:post-transformer {FUNCTION}** + + A special hook to update the mapped DB data after all translation is done, but before writing to DB. + Analogous to the postponed YangToDB subtree callback that is invoked at the very end by the Transformer. + + This will be useful to add or update additional data to to be written to DB. + Example: adding the default ACL rule during ACL creation. + + Post-transformer can be annotated only to the top-level container(s) within each YANG module. + + Port-transformer callback function signature: + + ```go + /** + * PostXfmrFunc type is defined to use for handling any default handling operations + * required as part of the CREATE. + * + * Param: XfmrParams structure having database pointers, current db, operation, + * DB data in multidimensional map, YgotRoot, uri + * Return: Multi dimensional map to hold the DB data Map (tblName, key and Fields), error + **/ + type PostXfmrFunc func (inParams XfmrParams) (map[string]map[string]db.Value, error) + ``` + +12. **sonic-ext:get-validate {FUNCTION}** + + A special hook to validate YANG nodes, to populate data read from database. + + Allows developers to instruct the Transformer to choose a YANG node among multiple nodes + while constructing the response payload. + Typically used to check the "when" condition. + + Callback function signature: + + ```go + /** + * ValidateCallpoint is used to validate a YANG node during data translation + * back to YANG as a response to GET. + * + * Param : XfmrParams structure having Database pointers, current db, operation, + * DB data in multidimensional map, output param YgotRoot, uri + * Return : bool + **/ + type ValidateCallpoint func (inParams XfmrParams) (bool) + ``` + +13. **sonic-ext:rpc-callback {FUNCTION}** + + Handler function for YANG custom RPCs and actions. + Transformer does not process input or output payloads. + + Function signature: + + ```go + /** + * RpcCallpoint is used to invoke a callback for action + * + * Param : []byte input payload, dbi indices + * Return : []byte output payload, error + **/ + type RpcCallpoint func (body []byte, dbs [db.MaxDB]*db.DB) ([]byte, error) + ``` + +14. **sonic-ext:virtual-table {boolean}** + + In case there is no table in the DB schema mapped to Openconfig YANG list/container, the `virtual-table` extension can be used to instruct the Transformer to skip checking for resource presence during parent resource check (refer to 2.5.5). By default, the virtual-table is set to `False`. + + e.g. `intf_subintfs_table_xfmr` and `intf_subintfs_xfmr` map to `SUBINTF_TBL` table & `index` key respectively, but this table/key does not reside in CONFIG_DB. In this case, you can set `sonic-ext:virtual-table “True”` to skip the resource check. + + ``` + deviation /oc-intf:interfaces/oc-intf:interface/oc-intf:subinterfaces/oc-intf:subinterface { + deviate add { + sonic-ext:table-transformer "intf_subintfs_table_xfmr"; + sonic-ext:key-transformer "intf_subintfs_xfmr"; + sonic-ext:virtual-table “True” + } + } + ``` + + ​ e.g. there is no mapping table to /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol + +15. **sonic-ext:value-transformer {FUNCTION}** + + This extension is valid only for SONiC yang. Any yang-key-node or leaf-node can be annotated with the value-transformer extension. The value-transformer callback function supports data conversion before writing into the DB for CRUD request and after reading from DB for GET request. The value-transformer annotated for a leaf will also be invoked, for all other leaves having a leaf-ref to it. + + ``` + // ValueXfmrFunc type is defined to use for conversion of DB field value from one forma to another + // Transformer function definition. + // Param: XfmrDbParams structure having Database info, operation, db-number, table, key, field, value + // Return: value string, error + type ValueXfmrFunc func (inParams XfmrDbParams) (string, error) + ``` + +16. **sonic-ext:table-owner {boolean}** + By default, CRUD operation can create, update or delete an entry in the DB table mapped to a certain YANG node. This table-owner extension can be used to explicitly assign an ownership of the given DB Table in the case when multiple OpenConfig YANG nodes are mapped to a single DB Table. In this case, CRUD operation on the node annotated with this extension set to false can modify the fields, but not permitted to delete an entry mapped to the given DB table which is shared by other YANG nodes. + + e.g. Operation on the resource /oc-qos:qos/oc-qos:interfaces/oc-qos:interface is not permitted to delete an entry in the DB table, learned from the callback qos_intf_table_xfmr. + + ``` + deviation /oc-qos:qos/oc-qos:interfaces/oc-qos:interface { + deviate add { + sonic-ext:key-transformer "qos_intf_tbl_key_xfmr"; + sonic-ext:table-transformer "qos_intf_table_xfmr"; + sonic-ext:table-owner "false"; + } + } + ``` + + + +17. **sonic-ext:cascade-delete {enable|disable}** + This extension can be used to de-configure all related nodes from the target resource on DELETE operation. The extension "cascade-delete" enables dependent configuration clean up for a DELETE request, based on DB Table dependency defined in SONiC YANG. + +18. **sonic-ext:subscribe-preference {sample|onchange}** + This extension can be used to hint the transformer infra on the preferred subscription type for a yang-node. By default, this will be set to "sample" for all yang nodes. User can override it to "onchange". + +19. **sonic-ext:subscribe-on-change {enable|disable}** + This extension can be used to hint the transformer infra onchange subscription support in a yang-node. By default, this will be set to "enable" for all yang nodes. + +20. **sonic-ext:subscribe-min-interval {interval-in-secs}** + Default min-interval is 60secs, set in translib. User can over-ride this value, using "subscribe-min-interval" annotation, if required. + + + + + +#### 2.5.3 Guidelines for Choosing Annotation Type + + + +1) If the translation is simple mapping between YANG container/list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter + +2) If the translation requires a complex translation with your codes, consider the following transformer extensions - key-transformer, field-transformer, subtree-transformer to take a control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope + +3) If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-transformer to dynamically choose tables based on URI/payload + +4) In Get operation access to non CONFIG_DB, you can use the db-name extension + +5) In Get operation, you can annotate the subtree-transformer on the node to implement your own data access and translation with DbToYang_xxx function + +6) In case of mapping a container to TABLE/KEY you can use the key-name/key-transfomer along with the table-name/table-transformer extension + + + +#### 2.5.4 Manifest file + +Manifest file `sonic-mgmt-common/config/transformer/models_list` contains all the YANG +file names that require transformer based translation. + +Each line is treated as one file name; lines stating with `#` are ignored. +Add all the standard OpenConfig, IETF YANG names along with their extension YANGs and annotation file here. +SONiC YANG file names should not be added here. + +Example: + +``` +openconfig-lldp.yang +openconfig-lldp-ext.yang +openconfig-lldp-annot.yang +. . . +``` + + + +#### 2.5.5 Transformer Callback functions + +The callback function signatures and xfmrParams strcutures are defined in +`sonic-mgmt-common/translib/transformer/xfmr_interface.go` file. + +###### 2.5.5.1 registering callbacks + +Transformer callback functions can be implemented in Go source files in `sonic-mgmt-common/translib/transformer` directory. +It is suggested to have feature and sub-feature specific source files for easy browsing. + +Go function for each of the callback names (defined of annotation file) should be registered through `XlateFuncBind` API call, so that those callbacks can be plugged into the transformer infra during transformer module initialization. +This can be done in the `init()` function of every Go source file where the callback functions are implemented. + +Example: + +```go +func init () { + XlateFuncBind("intf_table_xfmr", intf_table_xfmr) + XlateFuncBind("YangToDb_intf_name_xfmr", YangToDb_intf_name_xfmr) + XlateFuncBind("DbToYang_intf_name_xfmr", DbToYang_intf_name_xfmr) +. . . +} +``` + + + +Note that [key-transformer](#2525-sonic-extkey-transformer-function), [field-transformer](#2526-sonic-extfield-transformer-function) and [subtree-transformer](#2528-sonic-extsubtree-transformer-function) requires a pair of callbacks - with `YangToDb_` and `DbToYang_` prefix added to the function name specified in annotation file. They are required to support bi-directional translation from YANG to DB and vice versa, for SET (CRUD) and GET operations. + +For the rest of transformer callbacks, you need only one callback : [table-transformer](#2527-sonic-exttable-transformer-function), [post-transformer](#25210-sonic-extpost-transformer-function), [get-validate](#25211-sonic-extget-validate-function) and [rpc-callback](#25212-sonic-extrpc-callback-function) etc. + + + +###### 2.5.5.2 xfmrParams structure + +All the callback functions, except rpc-callback and subscibe-subtree callback, receive context information through a `XfmrParams` object. +The rpc-callback callback function receives raw bytes sent by the client. + +```go +// XfmrParams contains context, data passed to the callback functions. +ype XfmrParams struct { + d *db.DB + dbs [db.MaxDB]*db.DB + curDb db.DBNum + ygRoot *ygot.GoStruct + uri string + requestUri string //original uri using which a curl/NBI request is made + oper int + table string + key string + dbDataMap *map[db.DBNum]map[string]map[string]db.Value + subOpDataMap map[int]*RedisDbMap // used to add an in-flight data with a sub-op + param interface{} + txCache *sync.Map + skipOrdTblChk *bool + isVirtualTbl *bool + pCascadeDelTbl *[] string //used to populate list of tables needed cascade delete by subtree overloaded methods + yangDefValMap map[string]map[string]db.Value +} +``` + +, where each field is used in below. RO - read-only by application. RW - read-write by application + +d: (RO) pointer to the CONFIG_DB + +dbs: (RO) pointer to all the DBs, **used for GET only** + +curDB:(RO) DB number which is annotated to the YANG node by sonic-ext:db-name. By default, it is set to 4, CONFIG_DB. + +ygRoot: (RW for GET operation, RO for SET operation) ygot struct initialized by translib + +- For GET operation, during traversal of subtrees which has subtree xfmr annotated, the transformer infra builds the ygRoot tree till the uri which invokes the subtree-xfmr fucntion. Then, the subtree-xfmr fills the ygRoot tree for the subtree pointed by the uri, with result data fetched from data source. At the end of transaction, the ygRoot tree is referenced by the infra to construct the return payload. +- For SET, the translib builds the ygRoot tree as taken from the request uri and body. + +uri: (RO) current URI pointing to the node where a callback gets invoked + +requestUri: (RO) target resource URI specified in client requests + +oper: (RO) 1(GET), 2(CREATE), 3(REPLACE), 4(UPDATE), 5 (DELETE) + +table: (RO) DB table name of current node + +key: (RO) DB key of current node + +dbDataMap: (RW) For GET operation, the dbDataMap has data fetched from Redis-DB by the transformer infra, optionally filled by transformer callbacks to provide the hint to the infra to traverse subtrees. **Used for GET only** + +subOpDataMap: (RW) data map filled by callbacks with the sub-op different from the operation of the request, **used for SET only** +Note that the final result data map generated by infra gets merged with subOpDataMap before data written to CONFIG_DB + +Param: (RO for SET) placeholder for either YANG node or default value where YANG defval is defined. Applicable to OpenConfig YANG only. **Used for SET only** + +txCache: (RW) placeholder pointing to the cache block, to store data between callbacks for further lookup along data tree traversal, Mainly used for optimization + +isVirtualTbl: (RW) a flag to tell the infra to avoid resource check + +pCascadeDelTbl: (RO) used to populate list of tables needed cascade delete by subtree callbacks, **Used for Delete only** + +yangDefValMap: (RW) Map that contains the YANG default values by the infra, and can be referenced by Post-xfmr to alter default values as needed, **Used for SET only** + + + +###### 2.5.5.3 Subscribe-subtree callback + +The subscribe-sbutree callback can be used in two different contexts - Subscribe and Resoure check. The subscribe conext serves the gNMI subscripton request with either on-change or sample, while the resource check is used for CRUD and Get operation. + +Below are the fucntion signature and parameter structures for subscribe-sbutree callback. + +``` +// SubTreeXfmrSubscribe type is defined to use for handling subscribe(translateSubscribe & processSubscribe) subtree +// Transformer function definition. +// Param : XfmrSubscInParams structure having uri, database pointers, subcribe process(translate/processSusbscribe), DB data in multidimensional map +// Return : XfmrSubscOutParams structure (db data in multiD map, needCache, pType, onChange, minInterval), error +type SubTreeXfmrSubscribe func (inParams XfmrSubscInParams) (XfmrSubscOutParams, error) + +// XfmrSubscInParams represents input to subscribe subtree callbacks - request uri, DBs info access-pointers, DB info for request uri and subscription process type from translib. +type XfmrSubscInParams struct { + uri string + dbs [db.MaxDB]*db.DB + dbDataMap RedisDbMap + subscProc SubscProcType +} + +type XfmrSubscOutParams struct { + dbDataMap RedisDbMap + needCache bool + onChange bool + nOpts *notificationOpts //these can be set regardless of error + isVirtualTbl bool //used for RFC parent table check, set to true when no Redis Mapping +} +``` + + + +For a request-Uri pointing to a yang-node having a subtree-transformer annotation, the transformer infra supports subscribe-subtree callback. Existing "subtree-transformer" annotation will be used for this. + +Subscription support: + +- Infra will invoke subscribe-subtree, similar to YangToDB(for CRUD) and DbToYang(for GET) subtree functions. +- Subscription type i.e. translate-subscribe & process-subscribe, will be passed as an input parameter to the subscribe-subtree, along with incoming URI. +- Depending upon the incoming Uri, the subscribe-subtree callback should return appropriate DB number, table, key, need- cache, subscribe preference type and sampling min interval info. +- The transformer infra will validate a translate-subscribe request URI and return error if the request-URI-node has child(sub) container/list, since subscribe is supported only for URIs pointing to containers that has terminal nodes, i.e. leaf & leaf-list. + +Resource check support: + +- for CRUD/Get request, if a subtree xfmr is annotated on the path, you MUST have a subscribe-subtree callback implemented to return the tables mapped at the given URI to perform an existence check. The subscribe-subtree callback is intended for use by the transformer to perform the existence check for the given resource. + + + + + + ###### 2.5.5.4 callback chains + +​ A series of transformer callback functions can be invoked by the transformer infra while the YANG model trees are traversed to execute the the request. Some of callbacks are invoked in precedence to others. + + + + CRUD: + + NBI request uri with payload(input) + 1) *Parent resource check* + 2) pre-transformer + 3) + { (recursive call along tree based on annotation) + table-transformer (either at list level or container level) + key-transformer (at list instance level with key, or at container level with key if present) + field-transformer + subtree-transformer + } + 4) post-transformer + 5) value-transformer + + ==> dbDataMap(output) + + + + + + GET: + + 1) table-transformer and key-transformer invoked to read DB data into dbdDataMap + 2) value-transformer if present + 3) *Parent resource check* + 4) pre-transformer + 5) + { (recursive call along tree based on annotation) + table-transformer (either at list level or container level) + key-transformer (at list instance level with key, or at container level with key if present) + field-transformer + subtree-transformer + } + 6) post-transformer + + ==> response to NBI request(output) + + +​ + + Note here is the callback sequence during Parent resource check - + CRUD and GET: table-transformer/key-transformer (at list level with key) ----> value-transformer for transformed DB key at list level (if present) ---> (subscribe-subtree transformer) at list level (if subtree is annotated) + DELETE at request URI - table-transformer/key-transformer (at container level with key) + + +#### 2.5.6 Transformer Best Practices + +This section illustrates the best practices using transformer callbacks and extensions. + +###### 2.5.6.1 List/container + +In general, there is no need to annotate YANG extensions or implement transformer callbacks if you plan to use SONiC yang for northbound API. However, if you use OpenConfig yang, it means that you need to do some tranformation between OC yand and underlying SONic yang to access database and below is a common practice. + +Typical OpenConfig yang has a wrapper container on a list which has sub-containers - config and state conatiner. + +If a list or container can be statically mapped to a Redis Table, you can simply use the table-name extension annotated to the node. If the mapping is not static, i.e. it requires some checks to deremine a table, you can annotae the table-tarnsformer extension to implement the callback to find the table name corresponding the given node. + +Note that the key-transformer shoud also be provided to translate the key value because the DB keys are constrcuted with a delimeter. + +e.g. + +``` +/* BGP Neighbors*/ + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:neighbors/oc-netinst:neighbor { + deviate add { + sonic-ext:table-transformer "bgp_nbr_tbl_xfmr"; + sonic-ext:key-transformer "bgp_nbr_tbl_key_xfmr"; + } + } +``` + + + +###### 2.5.6.2 Leaf/Leaf-list + +As mentioned in 2.5.6.1, there is no need to traslate data between YANG leaf or leaf-list if you plan to use SONiC yang. + +If you use OpenConfig yang, you also need to consider the field transformer if the field value from data source, i.e. DB or other applications, is different from corresponding node, leaf or leaf-list, defined in Openconfig yang. + +e.g. + +``` + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:config/oc-netinst:name { + deviate add { + sonic-ext:field-transformer "network_instance_name_field_xfmr"; + } + } +``` + + + +###### 2.5.6.3 Field-transformer at yang list-key + +According to OpenConfig modeling guidelines, the key leaf(s) defined in OC yang list is always defined with leafref type to point at the leaf defined in the subcontainer config. + +It is not recommended, for performance reasons, to have a field-transformer at YANG list key nodes (leaf-nodes directly under list) when there is already a field-name/field-transformer annotation at the **same leaf-nodes**(leaf-referenced by list keys-nodes) in list/config containers of OC-yang. + +``` +E.g. This field transformer is not needed because the interface/config/name already has one + deviation /oc-intf:interfaces/oc-intf:interface/oc-intf:name { + deviate add { + sonic-ext:field-transformer "intf_name_empty_xfmr"; + } + } + + deviation /oc-intf:interfaces/oc-intf:interface/oc-intf:config/oc- intf:name { +``` + +​ sonic-ext:field-transformer "intf_name_empty_xfmr"; + +} + + + +###### 2.5.6.4 More on Table Transformer extension + +1) Static virtual table annotation + +Before calling any transform callbacks, the transformer Infra performs the target URI resource check of a gien request to validate all parent resources, i.e. at list instance, as per open-config yang hierarchy. + +The table name mapped at the list level is used for DB resource check. In case the table mapped at the list level is not determined, the virtual-table extension can be annotated to the list on URI to skip the resource check. This would be useful when a list is mapped to no Redis table during resource check. + +Eg: there is no redis table that can be mapped at protocol list level in openconfig-network-instances yang. Hence it can be annotated with virtual-table “true” + +``` +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol { + deviate add { + sonic-ext:table-transformer "network_instance_protocols_ptotocol_table_name_xfmr"; + sonic-ext:key-transformer "network_instance_protocol_key_xfmr"; + sonic-ext:virtual-table "true"; + } +} +``` + + + +2) Dynamic virtual table annotation + +When you need to dynamically determine table name, e.g. PORT or VLAN table etc. based on interface type , you need to implement the table transformer. In some cases, you also need to skip the resource check if a table is not known but you want to continue to traverse the tree without returning "resource not found error". In such case, you can use the special flag, isVirtualTbl, set to true in the table transformer to tell the infra to skip resource check. + +Eg: bgp_nbr_tbl_xfmr annotated at the below yang path sets the isVirtualTbl flag when there are dynamic entries available. + + /* BGP Neighbors*/ + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:neighbors/oc-netinst:neighbor { + deviate add { + sonic-ext:table-transformer "bgp_nbr_tbl_xfmr"; + sonic-ext:key-transformer "bgp_nbr_tbl_key_xfmr"; + } + } + +``` +/* For dynamic BGP nbrs, if isVirtualTbl not set, infra will try to get from config DB and fails with Resource not found. For now for any specific nbr requests, check and set isVirtualTble. From xfmr infra when parent table key check happens the dbDataMap is nil. So for this condition, and if cache is not updated set the virtual table */ + + if (inParams.dbDataMap == nil && !present) { + reqUriPath, _ := getYangPathFromUri(inParams.requestUri) + if strings.HasPrefix(reqUriPath, "/openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor") { + *inParams.isVirtualTbl = true + } + return tblList, nil +} + +``` + + + +###### 2.5.6.5 More on Table Name extension + +Table name of current node is inherited to descendant nodes by default, unless specified with a new table name/transformer or NONE. The table-name extension can be used for static mapping from YANG container or list to DB table name, while the table-transformer can be used for dynamic mapping when a callback needs to check a certain condition to determine table name(s). + +Table name "NONE" has a special meaning and can be annotated where you want to stop inheritance of the Parent table to its children. When DELETE operation is performed on a YANG container having table-name "NONE" extension, all child tables having table-name annotation and its entries will be considered for delete. + +Eg: + +If the container dynamic-neighbor-prefixes is not annotated with NONE, it derives the table name from its parent in this case BGP_GLOBALS table mapped to /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:global. + +If a DELETE operation is performed on the target uri ‘dynamic-neighbor-prefixes’ container, the table name associated with this container is the derived table from its parent. Hence BGP_GLOBALS is considered for the delete processing by the Infra, which is not expected by the DELETE operation on the target uri. + +So, you can avoid this problem using table-name "NONE". If you annotate the table-name to NONE at the container, then the table name inheritance stops from there, and let the infra continue to find the tables mapped to the child nodes. + +``` +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:global/oc-netinst:dynamic-neighbor-prefixes { + deviate add { + sonic-ext:table-name "NONE"; + } + } + +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:global/oc-netinst:dynamic-neighbor-prefixes/oc-netinst:dynamic-neighbor-prefix { + deviate add { + sonic-ext:table-name "BGP_GLOBALS_LISTEN_PREFIX"; + sonic-ext:key-transformer "bgp_dyn_neigh_listen_key_xfmr"; + } + } +``` + + + +###### 2.5.6.6 Pre-Transformer + +Consider using the pre-transformer if you need to prepare some data before starting data translation for the request. The pre-transformer would be useful when you need to perform pre-requisite action, e.g. cacheing data by direct access to data sources, DB or host applications etc. Note that, if the pre-transformer is defined, it is called only once at the start of the data translation, prior to any other callbacks except subcribe callback which is called during URI (resource) check. + +Examples that can be handled in pre transformer are - + +- storing the routes data in the txCache by invoking vtysh call to FRR, so this txCache block can be accessed subsequently by the key/subtree/table transformers when invoked during traversal. This would be useful to optimize callback invocations when a callbacks are annotated to the list, where callbacks are invoked per list instance. With this approach, you can avoid the expensive access like vtysh every time the transformer callback is invoked. +- setup in dbDataMap with anonymous table and dbKey entries that are required for list traversal. + + + +###### 2.5.6.7 **Deletion of child tables along OC YANG tree** + +When the DELETE operation is requested on the container, all the tables mapped to decendant nodes are also deleted. Below are the cases when it delete tables. + +1) If the child table instances must be deleted based on the parent table key, you can use the post transformer to identify the child table instances that needs to be deleted. + +Eg: Refer to hdl_del_post_xfmr, pim_hdl_post_xfmr + +2) If DELETE on a container will result in complete table deletion or will result in empty table name then, transformer infra will delete the annotated static tables (table-name annotated tables) for its children. Note that the transformer will rely on the CVL api provided parent child order to delete the child tables. + +Eg: In the below annotations if a delete is issued at /openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/peer-groups/peer-group/afi-safis, then the child tables annotated with table-name annotation are also deleted, i.e, all instances of BGP_PEER_GROUP_AF. + +``` +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:peer-groups/oc-netinst:peer-group/oc-netinst:afi-safis { + deviate add { + sonic-ext:table-name "NONE"; + } + } + +/* peer-group->afi-safis->afi-safi */ +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:peer-groups/oc-netinst:peer-group/oc-netinst:afi-safis/ocnetinst:afi-safi { + deviate add { + sonic-ext:db-name "CONFIG_DB"; + sonic-ext:table-name "BGP_PEER_GROUP_AF"; + sonic-ext:key-transformer "bgp_af_pgrp_tbl_key_xfmr"; + } + } +``` + + + + + +###### 2.5.6.8 Default Value altered by post-transformer + +For CRU operations, the transformer infra fills the default values for all SONIC tables that the payload maps to. If the applications need to alter the default values filled by the transformer infra, the post transformer can be used. + +Eg: Refer to hdl_del_post_xfmr + + + +###### 2.5.6.9 Skip traversal during GET for yang containers/list: + +1) get-validate : +Based on yang hierarchy if some containers/lists are not relevant for traversal on specific parent list instances then the traversal can be controlled by having a get-validate annotation. + +Eg-1: For static routes container is relevant only if the protocol list instance is STATIC, for all other protocols, static routes is not relevant. In such cases to avoid traversal of static routes container and its children during GET, get-validate annotation can be used to indicate traversal is required for this subtree or not. + +``` + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:static-routes { + deviate add { + sonic-ext:get-validate "static_routes_validate_proto"; + } + } +``` + + + +``` +module: openconfig-network-instance + +--rw network-instances + +--rw network-instance* [name] + +--rw protocols + | +--rw protocol* [identifier name] + | +--rw identifier -> ../config/identifier + | +--rw name -> ../config/name + | +--rw config + | +--ro state + | +--rw static-routes + | +--rw bgp + | | +--rw global + | | | +--rw config +``` + + + +Eg-2: Based on the afi-safi instance the get validate function is used to decide if traversal if the container is required or not for GET. + +``` + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:neighbors/oc-netinst:neighbor/oc-netinst:afi-safis/oc-netinst:afi-safi/oc-netinst:l2vpn-evpn { + deviate add { + sonic-ext:get-validate "bgp_validate_nbr_af"; + } + } + + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:neighbors/oc-netinst:neighbor/oc-netinst:afi-safis/oc-netinst:afi-safi/oc-netinst:ipv4-unicast { + deviate add { + sonic-ext:get-validate "bgp_validate_nbr_af"; + } + } + + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:neighbors/oc-netinst:neighbor/oc-netinst:afi-safis/oc-netinst:afi-safi/oc-netinst:ipv6-unicast { + deviate add { + sonic-ext:get-validate "bgp_validate_nbr_af"; + } + } +``` + + + +###### 2.5.6.10 Subtree Transformer + +Subtree transformer can be used used when there is no one-one mapping between SONiC and OC yang and the mapping to Redis tables is complex or if the DB entries mapped to OC yang are not in stored in RedisDB + +eg: FRR stored data + +When annotating with a subtree annotation, if the subtree is annotated at list instance level, care should be taken to make sure that the annotation at list level is done only when essential as this may lead to subtree being invoked for every list instance resulting in increased processing by the infra and hence a hit on performance in a scaled setup. + +Applications can consider annotating the subtree at the container enclosing the list and handling all list instances in a single subtree callback invocation. + +Eg: For annotation like below + +``` +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:static-routes { + deviate add { + sonic-ext:table-name "STATIC_ROUTE"; + sonic-ext:key-transformer "static_routes_key_xfmr"; + sonic-ext:get-validate "static_routes_validate_proto"; + } + } + +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:static-routes/oc-netinst:static/oc-netinst:next-hops/oc-netinst:next-hop { + deviate add { + sonic-ext:key-transformer "static_routes_key_xfmr"; + sonic-ext:subtree-transformer "static_routes_nexthop_xfmr"; + } + } +``` + + + +The subtree can been annotated at "/oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:static-routes" because for the above annotation only key in "static" list will be added by infra and then for each list instance ygot is created and then subtree is called for each instance (ygot creation at each list instance in scaled config will result in a performance hit) Instead the subtree annotation when done like below will result in better performance + +``` + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:static-routes { + deviate add { + sonic-ext:subtree-transformer "static_routes_subtree_xfmr"; + } +``` + + + + + +###### 2.5.6.11 Reevaluate the subtree transformer annotated at child- list + +Annotate the subtree-transformer at a wrapper container of parent-list and not at child list-level, if the parent list has only container having just the key leaf, for performance reasons. + +e.g. + +``` +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:static-routes/oc-netinst:static/oc-netinst:next-hops/oc-netinst:next-hop + { + deviate add { + sonic-ext:key-transformer "static_routes_key_xfmr"; + sonic-ext:subtree-transformer "static_routes_nexthop_xfmr"; + } + } +``` + + + +In this example, the subtree transformer should have been annotated at "/oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:static-routes" because the infra was used only for filling the key in "static" list. And then for each list instance, the ygot was created and then subtree callback called for each instance (creating ygot at each list instance in scaled config was a performance hit). + ++--rw static-routes + +​ | | +--rw static* [prefix] + +​ | | +--rw prefix -> ../config/prefix + +​ | | +--rw config + +​ | | | +--rw prefix? inet:ip-prefix + +​ | | +--ro state + +​ | | | +--ro prefix? inet:ip-prefix + +​ | | +--rw next-hops + +​ | | +--rw next-hop* [index] + + + +###### 2.5.6.12 Get-validate and subtree transformer at same level + +Do not combine the “get-validate” and “subtree-transformer” annotation at same yang node. Instead add the check to validate data at the entry point of subtree transformer. + + + + + + + +###### 2.5.6.1.13 Target resource check + +For target resource check (CRUD/GET context), a rule of thumb is , if a subtree xfmr is annotated on the path, you MUST have a subscribe-subtree callback implemented to return the tables mapped at the given URI to perform an existence check. The subscribe-subtree callback is intended for use by the transformer to perform the existence check for the given resource. Otherwise, either table-transformer/key-transformer or table-name/key-name along with DB number must be there to return table names mapped to the given URI. + +Below is an example code of subscribe-subtree cllback for the given path. The subscribe-subtree callback will be invoked before any other callback by the Transformer to determine the existence for the target resource. The returned **dbDataMap** of XfmrSubscOutParams should have DB number, table and keys specific to the given Uri that can be mapped to Redis table and key. + +``` +var Subscribe_ntp_server_subtree_xfmr = func(inParams XfmrSubscInParams) (XfmrSubscOutParams, error) { + var err error + var result XfmrSubscOutParams + result.dbDataMap = make(RedisDbMap) + + pathInfo := NewPathInfo(inParams.uri) + targetUriPath, _ := getYangPathFromUri(pathInfo.Path) + keyName := pathInfo.Var("address") + + log.Infof("Subscribe_ntp_server_subtree_xfmr path %v key %v ", targetUriPath, keyName) + if (keyName != "") { + result.dbDataMap = RedisDbMap{db.ConfigDB:{NTP_SERVER_TABLE_NAME:{keyName:{}}}} + log.Infof("Subscribe_ntp_server_subtree_xfmr keyName %v dbDataMap %v ", keyName, result.dbDataMap) + } else { + result.dbDataMap = RedisDbMap{db.ConfigDB:{NTP_SERVER_TABLE_NAME:{"*":{}}}} + log.Infof("Subscribe_ntp_server_subtree_xfmr keyName %v dbDataMap %v ", keyName, result.dbDataMap) + } + result.isVirtualTbl = false + + /* subscription context only */ + result.needCache = true + result.nOpts = new(notificationOpts) + result.nOpts.mInterval = 15 + result.nOpts.pType = OnChange + + log.Info("Returning Subscribe_ntp_server_subtree_xfmr") + return result, err +} +``` + + + +Note that if the URI points to a list without keys, then the key of returned dbDataMap needs to be set to "*". +Note that if there is no mapping table at all, please refer to the section 3 and 4 below. +Nota that the request URI contains multiple parent lists, the subscribe-subtree callback will be invoked per parent list. +Note that the fields - needCache, onChange, nOpts.mInterval, nOpts.pType - are specific to on-change subscription. As of now, on-change subscription is allowed for only container or list that has direct terminal nodes - leaf & leaf-list. Hence, these fields are not applicable to the subscribe-subtree callback associated to non-immediate parent lists. + + + +If multiples keys (instances) are returned from a subscribe-subtree callback, you can return with **"\*"** as Db key. + +e.g. /openconfig-qos:qos/scheduler-policies/scheduler-policy has subtree-transformer. + +``` +Request path("/openconfig-qos:qos/scheduler-policies/scheduler-policy[**name=sp1**]/schedulers/scheduler[**sequence=2**]/two-rate-three-color/config/cir") +``` + + Note that, in the above example, the key "**sp1**" and "**2**" gets concatenated to make a DB key, **"SCHEDULER|sp1@2"**. + The subscribe-subtree callback will be invoked at both scheduler-policy[name=sp1] and scheduler[sequence=2] in this order. At scheduler-policy[name=sp1], the subscribe-subtree callback has to return table="SCHEDULER" and key="sp1*" because the sequence 2 is not known at this level. The infra will perform pattern-based query using "sp1*" to ensure resource existence. Subsequently, at scheduler[sequence=2], the subscribe-xfmr has to return table="SCHEDULER", key="sp1@2". + + + +· **XfmrSubscOutParams.isVirtualTbl** flag – This flag can be used for following **2 cases** : + +1.) In order to skip target resource check, when there is no table mapping available for a certain target resource, subscribe subtree callback, can set the isVirtualTbl flag in the return type XfmrSubscOutParams + +``` +type XfmrSubscOutParams struct { + dbDataMap RedisDbMap + needCache bool + onChange bool + nOpts *notificationOpts //these can be set regardless of error + isVirtualTbl bool //used for RFC parent table check, set to true when no Redis Mapping +} +``` + +**e.g.** below path can be mapped to both Redis table and dataset (e.g. sensor) fetched from Host via DBus. Hence your subscribe callback can set the isVirtualTbl to false if the URI points to the sensor. + +``` + deviation /oc-platform:components/oc-platform:component { + deviate add { + sonic-ext:subtree-transformer "pfm_components_xfmr"; + } + } +``` + +Here is an example code to set the isVirtualTbl flag. + +``` +var Subscribe_pfm_components_xfmr SubTreeXfmrSubscribe = func (inParams XfmrSubscInParams) (XfmrSubscOutParams, error) { + var err error + var result XfmrSubscOutParams + key := NewPathInfo(inParams.uri).Var("name") + + mstr := strings.ToLower(key) if key == "" || mstr == "sensor" { + /* no need to verify dB data if we are requesting ALL + components or if request is for sensor */ + result.isVirtualTbl = true + return result, err + } + + result.dbDataMap = make(RedisDbMap) + if mstr == "system eeprom" { + result.dbDataMap = RedisDbMap{db.StateDB: {EEPROM_TBL:{"*":{}}}} + } else if mstr == "software" { + /* software component reads from XML file but also + \* gets EEPROM information from dB */ + result.dbDataMap = RedisDbMap{db.StateDB: {EEPROM_TBL:{"*":{}}}} + } else if validPsuName(&key) { + result.dbDataMap = RedisDbMap{db.StateDB: {PSU_TBL:{key:{}}}} + } else if validFanName(&key) { + result.dbDataMap = RedisDbMap{db.StateDB: {FAN_TBL:{key:{}}}} + } else if validTempName(&key) { + result.dbDataMap = RedisDbMap{db.StateDB: {TEMP_TBL:{key:{}}}} + } else if validXcvrName(&key) { + result.dbDataMap = RedisDbMap{db.StateDB: {TRANSCEIVER_TBL:{key:{}}}} + } else { + return result, errors.New("Invalid component name") + } return result, err +} +``` + + + +2.) For GET case defer target resource check to DbToyang subtree. Since the target resource check happens at list instance level in uri, sometimes an app needs to take decision for list instance based on the child container in Uri. Or in some cases a list instance spans over 2 tables (at child container levels). In such cases app can set isVirtualTbl flag in subscribe subtree and then do the target resource check in their DbToyang subtree and return error from there if resource does not exist. + +**Eg**.Targt resource check happens at interface=vlan100 in subscribe subtree. However, this interface list spans over 2 tables depending on child container. igmp-snooping/interfaces/interface=Vlan100/config/mrouter-interface maps to CFG_L2MC_MROUTER_TABLE_TS whereas . igmp-nooping/interfaces/interface=Vlan100/config/ staticgrps maps to CFG_L2MC_STATIC_MEMBER_TABLE_TS/CFG_L2MC_STATIC_GROUP_TABLE.In such case subscribe-subtree can set the isVirtualTbl flag and defer resource check to the DbToYang subtree. + + + + + +· **Debug subscription** - To check if annotation on a yang node is loaded by transformer, look for fields :- 1.) subscribeOnChg(1- enable/0-disable), 2.) subscribeMinIntvl : int value and 3.) hasNonTerminalNode(true/false) – whether yang node has child container/list, in file /tmp/fullSpec.txt, in the mgmt-framework container. For a given URI for subscription , to check for what got registered with translib, look for info logs from common_app. + + + +###### 2.5.6.14 Subscribe-subtree vs virtual-table annotation + +When there is a subtree annotated at a yang node, it is recommended to have a subscribe -subtree implementation to notify on virtual-table, instead of static virtual-table annotation, for target resource check. This is because a subtree callback is inherited to child hierarchy but static annotation of virtual-table is not. If developer still chooses to statically annotate virtual-table then it has to be done at every list level in the yang hierarchy annotated with subtree. + +#### 2.5.7 Case Examples + +###### 2.5.7.1 **Static mapping: table-name, field-name** + +**Usage + ** table-name: map a YANG container/list to TABLE name + field-name: map a YANG leaf/leaf-list to FIELD name + +**YANG annotation + **1) maps a YANG list “policy-definition” to Redis table “ROUTE_MAP_SET” + +``` + deviation /oc-rpol:routing-policy/oc-rpol:policy-definitions/oc-rpol:policy-definition { + deviate add { + sonic-ext:**table-name** "ROUTE_MAP_SET"; + } + } +``` + +2) maps a YANG leaf “call-policy” to Redis field “call_route_map” + +``` +deviation /oc-rpol:routing-policy/oc-rpol:policy-definitions/oc-rpol:policy-definition/oc-rpol:statements/oc-rpol:statement/oc-rpol:conditions/oc-rpol:state/oc-rpol:call-policy { + deviate add { + sonic-ext:**field-name** "call_route_map"; + } + } +``` + + + +###### 2.5.7.2 **Field** **Xfmr** **(Interface enabled)** + +**Usage + ** Convert field-name and/or value to DB field name and value and vice-versa + +In the below example, the field transformer is used to convert yang field name “enabled” to Db field name “admin_status” as well as convert the yang value(true/false) to Db format value (up/down) and vice-versa. + +**YANG annotation** + +``` +deviation /oc-intf:interfaces/oc-intf:interface/oc-intf:config/oc-intf:enabled { + deviate add { + sonic-ext:field-transformer "intf_enabled_xfmr"; + } + } +``` + +**SET:** var **YangToDb**_intf_enabled_xfmr FieldXfmrYangToDb = func(inParams XfmrParams) (map[string]string, error) + +``` +curl -X PATCH https://10.11.56.28/restconf/data/openconfig-interfaces:interfaces/interface=Ethernet32" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" -d "{ \"openconfig-interfaces:interface\": [{\"name\":\"Ethernet32\",\"config\": {\"name\":\"Ethernet32\",\"enabled\":false,\"mtu\": 2500}}]}" +``` + +**GET:** var **DbToYang**_intf_enabled_xfmr FieldXfmrDbtoYang = func(inParams XfmrParams) (map[string]interface{}, error) {} + +``` +curl -X GET "https://10.11.56.28/restconf/data/openconfig-interfaces:interfaces/interface=Ethernet32/config/enabled" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" -k -u "admin:admin123" +``` + + + +###### 2.5.7.3 **Key** **Xfmr** **(Annotated at List)** + +**Usage** + +Convert yang key/field value to DB key value and vice-versa. Can be annotated at list or container level. + +**network_instance_table_key_xfmr** – This key transformer is used to convert yang list key “name” to Db table key format and vice-versa. The Db table corresponds to key. If key value is “mgmt” then the DB key is “vrf_global” (in MGMT_VRF_CONFIG); if its default/Vrf* the key value is taken as is for table VRF. If its Vlan* then also key value is taken as is for VLAN table. + +**YANG annotation** + +``` + deviation /oc-netinst:network-instances/oc-netinst:network-instance { + deviate add { + sonic-ext:table-transformer "network_instance_table_name_xfmr"; + sonic-ext:key-transformer "**network_instance_table_key_xfmr**"; + } + } +``` + +**SET:** var **YangToDb_network_instance_table_key_xfmr** KeyXfmrYangToDb = func(inParams XfmrParams) (string, error) + +``` +curl -X POST "https://10.11.56.28/restconf/data/openconfig-network-instance:network-instances/network-instance=mgmt""accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" -d "{ \"openconfig-network-instance:config\": {\"name\": \"mgmt\",\"enabled\": true}}" +``` + +**GET:** var **DbToYang_network_instance_table_key_xfmr** KeyXfmrDbToYang = func(inParams XfmrParams) (map[string]interface{}, error) + +``` +curl -X GET "https://10.11.56.28/restconf/data/openconfig-network-instance:network-instances/network-instance=mgmt/config/enabled" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" +``` + + + +###### 2.5.7.4 **Key** **Xfmr** **(Annotated at List)** + +**Usage** + + **nat_global_key_xfmr** – This key transformer, annotated at container level, is used to assign NAT_GLOBAL table key value “Values” on Set/Configuration (yangToDb ) and nothing on the Get (DbToYang) side(since there is no corresponding yang field/key to be filled) + +**YANG annotation** + +``` + deviation /oc-nat:nat/oc-nat:instances/oc-nat:instance/oc-nat:config { + deviate add { + sonic-ext:table-name "NAT_GLOBAL"; + sonic-ext:key-transformer "**nat_global_key_xfmr**"; + } + } +``` + +**SET:** var YangToDb_**nat_global_key_xfmr** KeyXfmrYangToDb = func(inParams XfmrParams) (string, error) + +curl -X POST "[https://10.11.56.28/](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[restconf](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[/data/](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[openconfig-nat:nat](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[/instances/instance=1/config](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)" -H "accept: application/yang-data+json” -H "Content-Type: application/yang-data+json" -d "{ \"openconfig-nat:enable\": true, \"openconfig-nat:timeout\": 456, \"openconfig-nat:tcp-timeout\": 567, \"openconfig-nat:udp-timeout\": 333}" + +**GET:** var DbToYang_**nat_global_key_xfmr** KeyXfmrDbToYang = func(inParams XfmrParams) (map[string]interface{}, error) + +curl -X GET "[https://10.11.56.28/](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[restconf](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[/data/](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[openconfig-nat:nat](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)[/instances/instance=1/config](https://10.11.56.28/restconf/data/openconfig-nat:nat/instances/instance=1/config)" -H "accept: application/yang-data+json" + + + +Note that the key-transformer can also be annotated in SONiC yang to support a GET case, where the yang list key-leaves need to be extracted from DBKey string, and Dbkey separator is also present in the key-component value. + +e.g. list MCLAG_FDB_TABLE_LIST { key "vlan mac-address mac-type-str"; } +In case the Db-key string is "Vlan3195:**00:a0:00:00:01:26**:remote", where the key-separator ":" is also present in the key-value (mac-address). +From the above single string value, the YANG based list key-leaves need to be extracted. The key-transformer can be implemented to tokenize the string into populate the YANG data. + + + +###### 2.5.7.5 **Table** **Xfmr** **(Annotated at List)** + +**Usage** + +Table xfmr can be used for dynamically deriving the TABLE name based on the URI and key data provided via OC yang request, table-name annotation can be used for static case like OC list maps to particular table in Redis db. + + Example - + openconfig-interfaces.yang is used to support Ethernet, PortChannel, Management and Vlan interface types, but in Redis db. Each interface can be maped to separate TABLE (PORT, PortChannel,MGMT_PORT and VLAN respectively) , Table name changes based on the interface name we are acting on, so table-name annotation can’t be used in this case, it’s a good use case to use table transformer. + +**YANG annotation** + +``` +deviation /oc-intf:interfaces/oc-intf:interface { + deviate add { + sonic-ext:key-transformer "intf_tbl_key_xfmr"; + sonic-ext:table-transformer "**intf_table_xfmr**"; + } + } +``` + +**xfmr** **callback:** + var **intf_table_xfmr** TableXfmrFunc = func (inParams XfmrParams) ([]string, error) + + + +###### 2.5.7.6 **Table** **Xfmr** **(Annotated at Container)** + +**Usage** + +The table transformer shall be used when the yang container/list maps to more than one Redis Tables. e.g: for config node under a server-group, depending on the key value of the server-group, it can be mapped to different tables, i.e. “TACPLUS” table if the key is “TACACS”, or “RADIUS” if the key is RADIUS”. The table transformer can return a list of tables that the yang path maps to in Redis DB, e.g. get all the interfaces. + Note-1: when the yang entry maps to a single table always, the **table-name** annotation can be used. + Note-2: a single table-xfmr is used for both set and get. + +**YANG annotation** + +``` + deviation /oc-sys:system/oc-sys:aaa/oc-sys:server-groups/oc-sys:server-group/oc-sys:config { + deviate add { + sonic-ext:key-transformer "global_sg_key_xfmr"; + sonic-ext:table-transformer "**global_sg_tbl_xfmr**"; + } + } +``` + +**xfmr** **callback:** + var **global_sg_tbl_xfmr** TableXfmrFunc = func(inParams XfmrParams) ([]string, error) + +**SET:** + +``` + curl -X POST "https://localhost/restconf/data/openconfig-system:system/aaa/server-groups/server-group=TACACS/config" -H "accept: application/yang-data+json" -H "authorization: Basic YWRtaW46c29uaWNAZGVsbA==" -H "Content-Type: application/yang-data+json" -d "{ \"openconfig-system-ext:source-address\": \"1.1.1.1\", \"openconfig-system-ext:auth-type\": \"mschap\", \"openconfig-system-ext:secret-key\": \"secret1\", \"openconfig-system-ext:timeout\": 10, \"openconfig-system-ext:retransmit-attempts\": 10}" +``` + +**GET: ** + +``` +curl -X GET "https://localhost/restconf/data/openconfig-system:system/aaa/server-groups/server-group=TACACS/config" -H "accept: application/yang-data+json" -H "authorization: Basic YWRtaW46c29uaWNAZGVsbA==" +``` + + + +###### 2.5.7.7 **Reset Table** **Xfmr** **inheritance** + +**Usage +** By default, the table transformer gets inherited to children nodes along the hierarchical subtree. It can be reset once the table-name “NONE” is annotated or new table-transformer or table-name is annotated to the descendant nodes. + e.g. + +deviation /oc-netinst:network-instances/oc-netinst:network-instance { + deviate add { + **sonic-ext:table-transformer** **"****network_instance_table_name_xfmr****";** + sonic-ext:key-transformer "network_instance_table_key_xfmr"; + } + } + +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols { + deviate add { + **sonic-ext:table-name** **"NONE";** **à** **stop inherit** + } + } + +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol { + deviate add { + **sonic-ext:table-transformer** **"****network_instance_protocols_ptotocol_table_name_xfmr****"; -> new table-transformer** + sonic-ext:key-transformer "network_instance_protocol_key_xfmr";** +** } + } + +deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst:bgp/oc-netinst:global { + deviate add { + **sonic-ext:table-name** **"BGP_GLOBALS"; -> new table-name** + sonic-ext:key-transformer "bgp_gbl_tbl_key_xfmr"; + } + + + +###### 2.5.7.8 **Subtree** **Xfmr** **(Ip Address)** + +**Usage +** Allows the sub-tree transformer to take full control of translation. Note that, if any other extensions are annotated to the nodes on the subtree, they are not effective. + +Main reason to use subtree transformer in this case was in OC yang only “IP address” is only key attribute but in Redis-db “ipaddress/prefix-length” is the key. So one of the non key and non mandatory attribute in OC yang is used as key attribute in Redis db, so as part of subtree xfmr method added logic to make prefix-length attribute. + +**YANG annotation** + +``` +deviation /oc-intf:interfaces/oc-intf:interface/oc-intf:subinterfaces/oc-intf:subinterface/oc-ip:ipv4/oc-ip:addresses { + deviate add { + sonic-ext:table-name "NONE"; + sonic-ext:subtree-transformer "**intf_ip_addr_xfmr**"; + } + } +``` + +**SET:** var YangToDb**_intf_ip_addr_xfmr** SubTreeXfmrYangToDb = func(inParams XfmrParams) (map[string]map[string]db.Value, error) + +``` + curl -X POST "https://localhost/restconf/data/openconfig-interfaces:interfaces/interface=Ethernet0/subinterfaces/subinterface=0/openconfig-if-ip:ipv4/addresses" -H "accept: application/yang-data+json" –H "Content-Type: application/yang-data+json" -d "{ \"openconfig-if-ip:address\": [ { \"ip\": \"10.1.1.1\", \"config\": {\"ip\": \"10.1.1.1\", \"prefix-length\": 24}}]}" +``` + + + + + +**GET:** var DbToYang_**intf_ip_addr_xfmr** SubTreeXfmrDbToYang = func (inParams XfmrParams) (error) + +``` + curl -X GET "https://localhost/restconf/data/openconfig-interfaces:interfaces/interface=Ethernet0/subinterfaces/subinterface=0/openconfig-if-ip:ipv4/addresses" -H "accept: application/yang-data+json" +``` + + + +###### 2.5.7.9 **Subtree** **Xfmr** **(IGMP-Snooping)** + +**Usage +** The subtree transformer maps to/from different Redis DB tables depending on the config available in the igmp-snooping subtree. + For both Set and Get Request the data translation is done from the different tables to the corresponding yang constructs. + +**YANG annotation +** + +``` + deviation /oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:protocols/oc-netinst:protocol/oc-netinst-deviation:igmp-snooping { + deviate add { + sonic-ext:subtree-transformer "**igmp_snooping_subtree_xfmr**"; + } + } +``` + +**SET:** var **YangToDb**_*igmp_snooping_subtree_xfmr* SubTreeXfmrYangToDb = func(inParams XfmrParams) (map[string]map[string]db.Value, error) + +``` +curl -X POST "https://localhost/restconf/data/openconfig-network-instance:network-instances/network-instance=default/protocols/protocol=IGMP_SNOOPING,IGMP-SNOOPING/openconfig-network-instance-deviation:igmp-snooping" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" -d "{\"openconfig-network-instance-deviation:interfaces\":{\"interface\":[{\"config\":{\"enabled\": true,\"last-member-query-interval\":1000,\"name\":\"Vlan1\",\"version\":3},\"name\":\"Vlan1\"},{\"config\":{\"enabled\":true,\"last-member-query-interval\":1000,\"name\":\"Vlan2\",\"version\":3},\"name\":\"Vlan2\"}]}}“ +``` + + + Below is the result map returned from the above subtree callback: + \----------------------------------------------------------------- + + Below is the result map returned from the above subtree callback: + \----------------------------------------------------------------- + +``` +CFG_L2MC_TABLE + + key : Vlan1 + + version :3 + + last-member-query-interval :1000 + + enabled :true + + key : Vlan2 + + version :3 + + last-member-query-interval :1000 + + enabled :true +``` + +e.g. + \- for data in “igmp-snooping/interfaces/interface/config” the corresponding Redis table is CFG_L2MC_TABLE + +\- for data in “igmp-snooping/interfaces/interface/config/mrouter-interface” the corresponding Redis table is CFG_L2MC_MROUTER_TABLE, + +\- for data in “igmp-snooping/interfaces/interface/config/static-multicast-group” the corresponding Redis table is CFG_L2MC_STATIC_MEMBER_TABLE + +-for data in “igmp-snooping/interfaces/interface/config/static-multicast-group/outgoing-interface” the corresponding Redis table is CFG_L2MC_STATIC_GROUP_TABLE. + + + +###### 2.5.7.10 **Subtree** **Xfmr** **(Interface counters)** + +**Usage** + +**intf_get_counters_xfmr** – This subtree transformer first fetches OID corresponding to queried Interface name. For this it refers to COUNTERS_PORT_NAME_MAP Db table from COUNTERS DB and finds field whose name matches to incoming interface. Once matched the field’s value is the desired OID. This OID is then used as key to find the counters data from COUNTERS table. Once the counters data for an OID is retrieved from Db table its is filled into appropriate yang fields under interfaces/interface[]/state/counters tree. If no interface key is given during query counters data for all interfaces is fetched. + +**YANG annotation** + +``` +deviation /oc-intf:interfaces/oc-intf:interface/oc-intf:state/oc-intf:counters { + deviate add { + sonic-ext:subtree-transformer "**intf_get_counters_xfmr**"; + sonic-ext:db-name "COUNTERS_DB"; + } + } +``` + +**SET:** N/A + +**GET:** var **DbToYang**_intf_get_counters_xfmr SubTreeXfmrDbToYang = func(inParams XfmrParams) error + +``` +curl -X GET "https://10.11.56.28/restconf/data/openconfig-interfaces:interfaces/interface=Ethernet32/state/counters" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" +``` + + + +###### 2.5.7.11 **Post** **Xfmr** **(NAT)** + +**Usage** + + The post transformer can be used to handle any transformations required at the end of the transaction after the infra has formed the DBMap, either for SET or GET use cases. + Note that the annotation for post transformer is always done at the **module** level and will be invoked **once** for the given transaction for the module. + + + +**YANG annotation** + +``` +deviation /oc-nat:nat { + deviate add { + sonic-ext:post-transformer "nat_post_xfmr"; + } + } +``` + + + +###### 2.5.7.12 **SubOpDataMap** **(Ip Address)** + +**Usage + ** SubOpDataMap map is one of the member in “inParams” parameter to all the xfmr method types, its used as out parameter. + +Take case where for an UPDATE operation from user via OC yang, like configure a same ip address with different prefix length on the Ethernet0, but in Redis -db. prefix-length is part of a key (ipaddress/prefix-length) and it can’t be update as key is different, to solve this as part of subtree YangtoDb implementation UPDATE operation on OC yang maps to DELETE and CREATE/UPDATE of entries in INTERFACE table. + +To perform multiple and different operation types on DB as part of any CRUD operations, user can make use of subOpDataMap. + + + +**Example** + xfmr_intf.go: + +var **YangToDb_intf_ip_addr_xfmr** SubTreeXfmrYangToDb = func(inParams XfmrParams) (map[string]map[string]db.Value, error) + { + . . . + +​ subIntfmap_del[tblName] = (map[string]db.Value) +​ key := ifName + "|" + overlapIP +​ subIntfmap_del[tblName][key] = value +​ subOpMap[db.ConfigDB] = subIntfmap_del +​ log.Info("subOpMap: ", subOpMap) +​ inParams.subOpDataMap[DELETE] = &subOpMap + +. . . + +} + + + +###### 2.5.7.13 **txCache** **(User management)** + +**Usage +** The txCache syncMap is one of the member in “inParams” parameter to all the xfmr method types. It can be used for the case that a callback needs to cache data to the txCache of inParams, and pass to another callback invocation for subsequent use of the same data retrieved from the txCache of the inParams. The txCache is bound to the given transaction scope. + + + +**Example** + Below example illustrates a username cached in the txCache on the callback invoked on the list user, subsequent callback on the config container reads the username to create a user with the password and role on the host via hamd. + +curl -X POST "https://localhost/restconf/data/openconfig-system:system/aaa/authentication/users" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" -d "{ \"openconfig-system:user\": [{\"username\": \“sonicadmin\", \"config\": {\"username\": \“sonicadmin\", \"password\": \“AbraCaDaBra\", \"role\": \"admin\"}}]}“ + + + + deviation /oc-sys:system/oc-sys:aaa/oc-sys:authentication/oc-sys:users/oc-sys:user { + +​ deviate add { + +​ sonic-ext:subtree-transformer "sys_aaa_auth_xfmr"; + +​ } + + } + +xfmr_system.go: + +var YangToDb_sys_aaa_auth_xfmr SubTreeXfmrYangToDb = func(inParams XfmrParams) (map[string]map[string]db.Value,error) { + { + . . . + + if _ , _ok := inParams.txCache.Load(userName);!_ok { + inParams.txCache.Store(userName, userName) } + + else {. . . + + } + +. . . + +} + + + +###### 2.5.7.14 **Cascade-delete** + +**Usage +**"cascade-delete" annotation can be used to support cascaded dependent config delete handling for a yang-node. By default, this will be set to false. + +Annotating a yang-node with "cascade-delete: true" will enable dependent config-delete. + +Like all other yang annotations, “cascade-delete" will be inherited by the child-yang-nodes. To stop inheritance, “cascade-delete" should be set to "NONE", in the child-yang-node. + + + +**App:** + +New redis-db table-based callback api [ e.g. _cascade_cfg_hdl () ] should be written by app-owner for any additional dbDataMap processing, if needed. + +The redis-db table-based callback api will be registered with xfmr-infra, like YangToDb and DbToYang overloaded fns. + + + +**Transformer-infra:** + +For the DELETE operation, an additional table-list ("cascade-del-tbl") will be introduced in the infra, to store cascade-delete enabled tables. + +Infra will update this list with the cascade-delete enabled tables referring the xspec-map (created with the help of oc/sonic yang and yang-annot files), during the yang to db data(dbDataMap) conversion. + +This table-list will be passed to the overloaded functions, so that the app owners can update(dynamically) this list, with cascade-delete enabled table-names. + + + +**Example Code:** + +``` +func init() { + XlateFuncBind("ABC_TABLE_cascade_cfg_hdl", ABC_TABLE_cascade_cfg_hdl) +. . . +} + +var ABC_TABLE _cascade_cfg_hdl XfmrDbTblCbkMethod = func (inParams XfmrDbTblCbkParams) error { + log.Infof("ABC_TABLE _cascade_cfg_hdl Received inParams %v ", inParams) + if inParams.delDepDataMap != nil { + /* + Populate what all operations need to be carried out in + inParams.delDepDataMap + */ + dbMap := make(map[db.DBNum]map[string]map[string]db.Value) + dbMap[db.ConfigDB] = make(map[string]map[string]db.Value) + dbMap[db.ConfigDB][" ABC_TABLE "] = make(map[string]db.Value) + + m := make(map[string]string) + value := db.Value{Field:m} + dbMap[db.ConfigDB][" ABC_TABLE "][inParams.dbKey] = value + inParams.delDepDataMap[DELETE] = &dbMap + } + return nil +} +``` + + + + + +###### 2.5.7.15 **Debugging Tips**** + +1.Delete operation: + /tmp/yangToDbDataDel.txt + +2.CRU operation: + /tmp/yangToDbDataCreate.txt + +3.Turn on debug log + +you can set the log_level in config_db. + +``` +hmset REST_SERVER|default log_level +hmset TELEMETRY|gnmi log_level +``` + + + +#### 2.5.8 Query Parameter + +Query parameters is supported only for GET request from NBI. The QPs supported by mgmt-framework are depth, content and fields. +Please refer to the HLD for more details on QP support in mgmt-framework. +https://github.com/project-arlo/SONiC/blob/master/doc/mgmt/SONiC_QueryParameterForGet_HLD.md + +###### 2.5.8.1 Query parameters in Transformer + +"QueryParams" struct will be used by the transformer infra to process the Query parameters and will be shared with the app subtree. This will part of "XfmrParams" struct. + +The data structures are as below: + +``` +type XfmrParams struct { + .... + queryParams QueryParams +} +const ( + QUERY_CONTENT_ALL ContentType = iota + QUERY_CONTENT_CONFIG + QUERY_CONTENT_NONCONFIG + QUERY_CONTENT_OPERATIONAL +) +type ContentType uint8 +type QueryParams struct { + depthEnabled bool //flag to indicate if "depth" is enabled + curDepth uint //current depth, will be decremented at each yang node level + content ContentType //content type all/config/non-config/operational + fields []string //list of fields(container, leaf/leaflist).List is not allowed. + //following attributes are for transformer infra use only + fieldsFillAll bool //flag to fill all fields in container/list + allowFieldsXpath map[string]bool //proceed further only if xpath is present in this set + tgtFieldsXpathMap map[string][]string //process & fill data only if xpath is present in this map. +} +``` + + + +- Depth: + For processing the depth QP, the inParams.queryParams.curDepth will be referenced by the subtree. + The value in the curdepth indicates how much depth is yet to be traversed by the subtree. + + Eg: + If the curDepth is 3, the subtree needs to fill the ygot upto 3 levels of depth in the yang subtree. + The curDepth value is decremented by 1 for every level processed and once the curDepth value becomes 0, + the subtree can stop the GET processing for subtree below it. + + + +- Content: + For processing the content QP, the inParams.queryParams.content will be used. + The subtree shall fill the ygot based on the content field in the yang subtree + Refer the below table for supported content types and what needs to filled by the subtree + + | Value | Description | + | ----------- | ------------------------------------------------------------ | + | config | Populate only configuration descendant data nodes (RW nodes) | + | non config | Populate only non-configuration descendant data nodes (RO nodes) | + | operational | Populate data in “non-config” & not in “config”.(applicable only for OC yang) (RO nodes) | + | all | Populate all descendant data nodes (RW and RO nodes) | + + + +- Fields: + For processing the fields QP, the inParams.queryParams.fields can be referenced by the subtree. + The fields will have the list of relative xpath of the fields that needs to be filled by the subtree. + The relative xpath is in reference to the requestUri. + + + +###### 2.5.8.2 Example to support query parameters + +Example implementation of subtree callbacks using query parameters can be seen below. Note that the data trees not annotated with subtree callbacks are implicitly handled by transformer core, and they do not require additional code to support query parameters. + +``` + type _xfmr_bgp_rib_key struct { + niName string + afiSafiName string + prefix string + origin string + pathId uint32 + pathIdKey string + nbrAddr string + queryParamDepth uint + queryParamFields []string +} + +var DbToYang_bgp_routes_get_xfmr SubTreeXfmrDbToYang = func(inParams XfmrParams) error { + + var err error + oper_err := errors.New("Operational error") + cmn_log := "GET: xfmr for BGP-RIB" + + var bgp_obj *ocbinds.OpenconfigNetworkInstance_NetworkInstances_NetworkInstance_Protocols_Protocol_Bgp + var rib_key _xfmr_bgp_rib_key + + bgp_obj, rib_key.niName, err = getBgpRoot(inParams) + if err != nil { + log.Errorf("%s failed !! Error:%s", cmn_log, err) + return oper_err + } + + bgpRib_obj := bgp_obj.Rib + if bgpRib_obj == nil { + log.Errorf("%s failed !! Error: BGP RIB container missing", cmn_log) + return oper_err + } + + pathInfo := NewPathInfo(inParams.uri) + targetUriPath, _ := getYangPathFromUri(pathInfo.Path) + + rib_key.afiSafiName = pathInfo.Var("afi-safi-name") + rib_key.prefix = pathInfo.Var("prefix") + rib_key.origin = pathInfo.Var("origin") + rib_key.pathIdKey = pathInfo.Var("path-id") + _pathId, _ := strconv.Atoi(pathInfo.Var("path-id")) + rib_key.pathId = uint32(_pathId) + rib_key.nbrAddr = pathInfo.Var("neighbor-address") + + if inParams.queryParams.depthEnabled { + rib_key.queryParamDepth = inParams.queryParams.curDepth + } + // Rib container level, for content (inParams.queryParams.content) - all, no action is required + // and config & operational informations are not available and + // what we have is only non-config (state) and all & non-config are same, + // so, no action is required. + rib_key.queryParamFields = inParams.queryParams.fields + + var fall_through bool = false + + switch targetUriPath { + case "/openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/rib": + fallthrough + case "/openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/openconfig-rib-bgp:rib": + fall_through = true + fallthrough + case "/openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/rib/afi-safis": + if fall_through { + if rib_key.queryParamDepth != 0 { + // Reduce the depth for afi-safis and afi-safi + if rib_key.queryParamDepth < 4 { + return nil + } + rib_key.queryParamDepth -= 2 + } else if len(rib_key.queryParamFields) > 0 { + for i, field := range rib_key.queryParamFields { + log.Infof("queryParamFields %s", field) + rib_key.queryParamFields[i] = strings.TrimPrefix(field, "afi-safis/afi-safi") + log.Infof("queryParamFields %s", rib_key.queryParamFields[i]) + } + } + } else if rib_key.queryParamDepth != 0 { + if rib_key.queryParamDepth < 3 { + return nil + } + // Reduce the depth for afi-safi + rib_key.queryParamDepth -= 1 + } else if len(rib_key.queryParamFields) > 0 { + for i, field := range rib_key.queryParamFields { + log.Infof("queryParamFields %s", field) + rib_key.queryParamFields[i] = strings.TrimPrefix(rib_key.queryParamFields[i], "afi-safi") + log.Infof("queryParamFields %s", rib_key.queryParamFields[i]) + + } + } + err = get_all_bgp_rib_objs(bgpRib_obj, &rib_key, &dbg_log) + if err != nil { + return oper_err + } + case "/openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/rib/afi-safis/afi-safi": + if rib_key.afiSafiName == "" { + err = get_all_bgp_rib_objs(bgpRib_obj, &rib_key, &dbg_log) + if err != nil { + return oper_err + } + } else { + if rib_key.queryParamDepth != 0 { + // Reduce the depth for afi-safis and afi-safi + if rib_key.queryParamDepth < 6 { + return nil + } + } + switch rib_key.afiSafiName { + case "IPV4_UNICAST": + err = get_all_ipv4_bgp_rib_objs(bgpRib_obj, &rib_key, &dbg_log) + if err != nil { + return oper_err + } + case "IPV6_UNICAST": + err = get_all_ipv6_bgp_rib_objs(bgpRib_obj, &rib_key, &dbg_log) + if err != nil { + return oper_err + } + } + } + case "/openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/rib/afi-safis/afi-safi/ipv4-unicast": + if rib_key.afiSafiName == "IPV4_UNICAST" { + if rib_key.queryParamDepth != 0 { + // Reduce the depth for afi-safis and afi-safi + if rib_key.queryParamDepth < 5 { + return nil + } + } + err = get_all_ipv4_bgp_rib_objs(bgpRib_obj, &rib_key, &dbg_log) + if err != nil { + return oper_err + } + } + case "/openconfig-network-instance:network-instances/network-instance/protocols/protocol/bgp/rib/afi-safis/afi-safi/ipv6-unicast": + if rib_key.afiSafiName == "IPV6_UNICAST" { + if rib_key.queryParamDepth != 0 { + // Reduce the depth for afi-safis and afi-safi + if rib_key.queryParamDepth < 5 { + return nil + } + } + err = get_all_ipv6_bgp_rib_objs(bgpRib_obj, &rib_key, &dbg_log) + if err != nil { + return oper_err + } + } + } + + return err + +} + +func get_all_bgp_rib_objs(bgpRib_obj *ocbinds.OpenconfigNetworkInstance_NetworkInstances_NetworkInstance_Protocols_Protocol_Bgp_Rib, + rib_key *_xfmr_bgp_rib_key, dbg_log *string) error { + var err error + oper_err := errors.New("Opertational error") + + if rib_key.queryParamDepth != 0 { + // valid data present only at the depth level 5, so, return from here if the depth is < 5 + if rib_key.queryParamDepth < 5 { + var ribAfiSafis_obj *ocbinds.OpenconfigNetworkInstance_NetworkInstances_NetworkInstance_Protocols_Protocol_Bgp_Rib_AfiSafis + if ribAfiSafis_obj = bgpRib_obj.AfiSafis; ribAfiSafis_obj == nil { + var _ribAfiSafis ocbinds.OpenconfigNetworkInstance_NetworkInstances_NetworkInstance_Protocols_Protocol_Bgp_Rib_AfiSafis + bgpRib_obj.AfiSafis = &_ribAfiSafis + ribAfiSafis_obj = bgpRib_obj.AfiSafis + } + + var ok bool + afiSafiType := ocbinds.OpenconfigBgpTypes_AFI_SAFI_TYPE_IPV4_UNICAST + if _, ok = ribAfiSafis_obj.AfiSafi[afiSafiType]; !ok { + ribAfiSafis_obj.NewAfiSafi(afiSafiType) + } + afiSafiType = ocbinds.OpenconfigBgpTypes_AFI_SAFI_TYPE_IPV6_UNICAST + if _, ok = ribAfiSafis_obj.AfiSafi[afiSafiType]; !ok { + ribAfiSafis_obj.NewAfiSafi(afiSafiType) + } + return nil + } + } + + err = get_all_ipv4_bgp_rib_objs(bgpRib_obj, rib_key, dbg_log) + if err != nil { + return oper_err + } + err = get_all_ipv6_bgp_rib_objs(bgpRib_obj, rib_key, dbg_log) + if err != nil { + return oper_err + } + return err + +} +``` + + +###### 2.5.8.3 Pruning API for Subtree callbacks + +In many cases, subtree transformers may want the query parameter support to be done by the infrastructure by pruning the (ygot) data tree. This support for pruning in the infrastructure should only be used if: + +- The time to compute the data tree is not significant. (i.e. where it is ok to have the overhead of building the tree, before pruning it for query parameter support by the infrastructure) +- The time to prune the data tree is not significant. +- The pruning API supports all the data types in the data tree being pruned. + +By default, the infrastructure will prune data trees created by subtree transformer callbacks. A subtree callback function may handle query parameters for it's section of the data tree. If a subtree callback desires to prune it's section of the data trees itself, it should tell the infra to skip pruning by setting the *pruneDone flag to true in the inParams. + +``` +type XfmrParams struct { +... + queryParams QueryParams + pruneDone *bool +... +} +``` + +Follow is a partial list of unsupported data types, features. + +- nonconfig (config=false) lists which have no keys defined. +- ... + +Feature developers may use the output of the REST (or Telemetry) server log files, to assist with making the determination for a subtree callback to prune their section of the tree themselves. Lines containing the following markers can be used: + +- xfmrPruneQP: func +- xfmrPruneQP: Totals: + +For example: + +``` +... +IDec 17 22:46:33.135333+00:00 2021 157611 xlate_xfmr_handler.go:51] xfmrPruneQP: func ntp_server_subtree_xfmr uri /openconfig-system:system/ntp/servers/server, requestUri /openconfig-system:system/ntp/servers +... +IDec 17 22:46:33.136191+00:00 2021 157611 xlate_prune.go:76] xfmrPruneQP: Totals: LastTime: 724.669µs LastUri: /openconfig-system:system/ntp/servers/server Hits: 1 TotalTime: 724.669µs PeakTime: 724.669µs PeakUri: /openconfig-system:system/ntp/servers/server +... + +``` + +In the above example, the *func* gives the subtree transformer function name, and the *LastTime* gives the time taken in the infra to prune the section of the data tree handled by that subtree transformer. + +### 2.6 App Module + +Instead of using the transformer, developers can write the complete App module that aids in the conversion of the incoming request to the SONiC ABNF format and vice versa. Following are the steps to be performed. + +1. Define a structure to hold all the incoming information as well as the translation information + +Example: + + type AclApp struct { + pathInfo *PathInfo + ygotRoot *ygot.GoStruct + ygotTarget *interface{} + + aclTs *db.TableSpec + ruleTs *db.TableSpec + + aclTableMap map[string]db.Value + ruleTableMap map[string]map[string]db.Value + } + +2. App modules will implement an init function which registers itself with the translib. In addition it should also add the YANG models it supports here to serve the capabilities API of gNMI. + Example: + func init () { + log.Info("Init called for ACL module") + err := register("/openconfig-acl:acl", + &appInfo{appType: reflect.TypeOf(AclApp{}), + ygotRootType: reflect.TypeOf(ocbinds.OpenconfigAcl_Acl{}), + isNative: false, + tablesToWatch: []*db.TableSpec{&db.TableSpec{Name: ACL_TABLE}, &db.TableSpec{Name: RULE_TABLE}}}) + + if err != nil { + log.Fatal("Register ACL App module with App Interface failed with error=", err) + } + + err = appinterface.AddModel(&gnmi.ModelData{Name:"openconfig-acl", + Organization:"OpenConfig working group", + Version:"1.0.2"}) + if err != nil { + log.Fatal("Adding model data to appinterface failed with error=", err) + } + } + +App Modules will implement the following interface functions that will enable conversion from YGOT struct to ABNF format. + + initialize(data appData) + translateCreate(d *db.DB) ([]db.WatchKeys, error) + translateUpdate(d *db.DB) ([]db.WatchKeys, error) + translateReplace(d *db.DB) ([]db.WatchKeys, error) + translateDelete(d *db.DB) ([]db.WatchKeys, error) + translateGet(dbs [db.MaxDB]*db.DB) error + translateSubscribe(dbs [db.MaxDB]*db.DB, path string) (*notificationOpts, *notificationInfo, error) + processCreate(d *db.DB) (SetResponse, error) + processUpdate(d *db.DB) (SetResponse, error) + processReplace(d *db.DB) (SetResponse, error) + processDelete(d *db.DB) (SetResponse, error) + processGet(dbs [db.MaxDB]*db.DB) (GetResponse, error) + +initialize – Populate the app structure that we created in step 1 with the incoming appData which contains path, payload, YGOT root structure, YGOT target structure. + +translate(CRUD) – Convert the information in the YGOT root and target structure to the corresponding ABNF key value pair and store it in the structure created in step 1 + +translateGet – Convert the incoming path to corresponding ABNF keys to be got from the redis DB using the DB access layer APIs similar to the python SWSSSDK. + +translateSubscribe – Convert the incoming path in the argument to corresponding ABNF keys as part of the notificationInfo structure. In addition to this subscribe parameters like type of subscription supported, SAMPLE interval can also be specified as part of the notificationOpts structure. + +process(CRUD) – Write the converted data from translate(CRUD) function into the Redis DB using the DB access layer APIs + + +### 2.7 Config Validation Library + +Config Validation Library (CVL) performs the automatic syntactic and semantic validation +of DB writes based on the YANG constraints defined in SONiC YANG. +These constraints include data type, ranges, patterns, "leafref", "must" and "when" statements. +Constraints written for read-only state nodes (non CONFIG_DB tables), RPC and notification nodes are ignored. +Complex validations that cannot be expressed via YANG constraints can be implemented via +custom validation functions written in Go. + +All DB writes performed via Translib DB Acceess APIs would automatically trigger CVL validations. + +#### 2.7.1 CVL Schema + +CVL uses YIN schema derived from SONiC YANGs for validation. +CVL schema is automatically generated during sonic-mgmt-common build. +Any changes to SONiC YANG requires sonic-mgmt-common compilation for it to reflect in CVL. +Details of the schema generator are in [Appendix 3.1.3](#313-cvl-schema-generator). + +#### 2.7.2 Custom Validation + +When to use custom validation? +* Complext validations, which can't be defined using YANG must/when expression. +* YANG based must/when consraints are taking lot of time + +Custom validation functions can be registered to a DB table or field definition in SONiC YANGs. +CVL provides a YANG extension statement **sonic-ext:custom-validation** for this. + +```yang +import sonic-extension { prefix sonic-ext; } + +.... +list ACL_RULE_LIST { + leaf IP_TYPE { + sonic-ext:custom-validation ValidateAclRuleIPAddress; + ... + } +.... +``` + +A Go function with same name should be defined in a Go source file under `sonic-mgmt-common/cvl/custom_validation` directory. +Use feature specific source file for better code organization - like "custom_validation_{feature}.go". +Function name should be unique across all SONiC YANGs. +So it is suggested to include table name and field name in the function name. + +Signature for custom validation handler function: + +```go +func (t *CustomValidation) (vc *CustValidationCtxt) CVLErrorInfo + +// CustValidationCtxt is the validation context passed to custom validation function +type CustValidationCtxt struct { + ReqData []CVLEditConfigData //All request data + CurCfg *CVLEditConfigData //Current request data for which validation should be done + YNodeName string //YANG node name + YNodeVal string //YANG node value, leaf-list will have "," separated value + YCur *xmlquery.Node //YANG data tree + SessCache *CustValidationCache //Session cache, can be used for storing data, persistent in session + RClient *redis.Client //Redis client +} + +//CVLErrorInfo CVL Error Structure +type CVLErrorInfo struct { + TableName string /* Table having error */ + ErrCode CVLRetCode /* CVL Error return Code. */ + CVLErrDetails string /* CVL Error Message details. */ + Keys []string /* Keys of the Table having error. */ + Value string /* Field Value throwing error */ + Field string /* Field Name throwing error . */ + Msg string /* Detailed error message. */ + ConstraintErrMsg string /* Constraint error message. */ + ErrAppTag string +} + + +``` + +#### 2.7.3 Platform Validation + +CVL allows platform specific validation in following ways: + +* Add YANG 'deviation' files (e.g. sonic-acl-deviation.yang for ACL YANG) per platform and place it in 'models/yang/sonic/platform' folder for static platform validation. These files are automatically picked up during build time, processed and applied at run time. + +``` +models/yang/sonic/platform/ +├── accton_as5712 +│ └── sonic-acl-deviation.yang +└── quanta_ix8 + └── sonic-acl-deviation.yang +``` + +Note : All platform specific deviation files are packaged in a single image file. At runtime, based on detected platform from provisioned “DEVICE_METADATA:platform” field, deviation files are applied. + +* Implement platform specific custom validation through CVL custom validation support for dynamic platform validation. + +#### 2.7.4 CVL Debug Logs + +For enabling CVL debug log please refer to "Debugging Info" section in `sonic-mgmt-common/cvl/README.md`. + + +### 2.8 Non-DB Data + +#### 2.8.1 Host Modules + +TODO + +#### 2.8.2 FRR Integration + +TODO + + +### 2.9 KLISH CLI + +Open source Klish is integrated to sonic-mgmt-framework to provide the command line interface tool to perform network operations more efficiently in SONiC. Klish will provide the core functionality of command parsing, syntax validation, command help and command auto-completion. + +Open Source [klish](http://libcode.org/projects/klish/.) is used here. + +#### 2.9.1 CLI components + +1. CLI Parser engine: Open source Klish + +2. XML files: +XML files, defined by developer, that describe the CLI command structure. +Klish uses XML based command tree inputs to build the parser command tree. +All CLI commands to be supported are specified in XML format in module specific XML file. +XML files can be defined with macros and entity references, preprocessed by scripts to generate the expanded XML files. + +3. Actioner: Script that will transform CLI commands to form the corresponding REST requests and invoke the REST client API. + +4. Renderer: Script that will receive the JSON response from REST client API and use the jinja2 template file to render(display) the CLI output in the desired format. + +#### 2.9.2 CLI development steps + +Following are the steps to add a new CLI command. Please refer to https://github.com/sipwise/klish/blob/master/doc/klish.md for detail. + +1. Create a CLI XML .xml file that defines the CLI command structure. This file defines the following + * CLI command format + * Parameters that the command requires + * Help string to be displayed for the command and parameters + * Datatype of the parameters. + * View name for which the command needs to be available. Eg: configure-view(config mode) or enable-view(exec mode) + +Example: + +```xml + + + + + + + if test "${access-list-name}" = ""; then + python $SONIC_CLI_ROOT/sonic-cli.py get_acl_acl_sets show_access_list.j2 + else + python $SONIC_CLI_ROOT/sonic-cli.py get_acl_set_acl_entries ${access-list-name} ACL_IPV4 show_access_list.j2 + fi + + +``` + +2. Write/Update an actioner script: The actioner script prepares the message body having the JSON request and invokes the REST client API. The actioner script is invoked by the klish and the input parameters are passed to it from the XML file. + Actioner can be defined with the tag in the XML file. + + There are three different methods available to implement the Actioner: sub-shell, clish_restcl and clish_pyobj. Sub-shell is spawned by Klish to run the script defined in tag. Both clish_pyobj and clish_restcl methods are part of Klish built-in fucntions and invoked by Klish. The built-in fucntions can be used for commands that would reduce time taken to execute a command by eliminating the sub-shell interpreter overhead. + + * Spawn a sub-shell to run the scripts defined in a command's tag. The shebang can be specified for the script execution. By default the "/bin/sh" is used. To customize shebang the 'shebang' field of the ACTION tag is used. + + The sub-shell runs the Python script sonic_cli_.py + + sonic_cli_.py [parameters . . .] + The sonic_cli_.py has a dispatch function to call a REST client method with parameters passed from user input. + + **Example**: + Refer the tag in the above example command. + The actioner scripts are placed in the following location: + sonic-mgmt-framework/CLI/actioner/ + One actioner script will be written per module. + Eg: sonic_cli_if.py can be used to handle the interface cases. + + * Invoke the built-in function, clish_restcl, to use libcurl to make REST client call + This builtin uses libcurl to connect to the REST server Format of ACTION tag argument: + + oper= url= body={..} + where oper can be PUT/POST/PATCH/DELETE and body is optional. Note that oper GET is not supported as we currently don't handle rendering using jinja templates. + + **Example**: + + oper=PATCH url=/restconf/data/openconfig-interfaces:interfaces/interface=Vlan${vlan-id}/config body={"openconfig-interfaces:config": {"name": "Vlan${vlan-id}"}} + + * Invoke the built-in function, clish_pyobj, to use the embedding Python to make REST client call + This builtin uses embedded python library. + Format of ACTION tag argument is similar to the one of sub-shell. + For 'show' commands, the jinja2 template is passed to the Python script to apply template to redender CLI output. + The ${__full_line} variable is also required to support Pipe, e.g sonic# show vlan | no-more. + Note that the "if-else" statement defined in the sub-shell can be moved to Python script. + + **Example**: + sonic_cli_vlan get_sonic_vlan_sonic_vlan Vlan${id} show_vlan.j2 ${__full_line} + + **Example**: + + Below example shows that the clish_pyobj can be used to set a dynamic variable "supported_breakout_modes" to check the breakout capability for a given port. + Once the result is returned from the Python fucntion, the variable keeps the result and pass to like below. +``` + + sonic_cli_breakout.py capability + + + +``` +3. Write/Update Renderer scripts and templates. + The JSON response from the swagger client API is received by the actioner and passes the response to the renderer script. + The renderer script will invoke the jinja2 template with the JSON response. The template will parse the JSON response and generate the CLI output. + Please use [JSON tool](#297-tool-for-json-navigation-in-jinja-templates) to efficiently access the JSON data. + Refer files in the below path for an example of usage + + **Renderer path**: + sonic-mgmt-framework/CLI/renderer + **Renderer script**: + scripts/render_cli.py + **Renderer template**: + templates/show_access_list.j2 + + Every show command response can have a separate template file. + +#### 2.9.3 Enhancements to Klish + +Additional enhancements can be done to open source klish as below. Enhancements may include defining a new data types for CLI command parameters, define new macros that can be referenced for CLI commands structure that have repetitive definitions and defining platform specific entity values. + +1. PTYPES + New parameter types (PTYPES) can be defined and used in the CLI XML files. + * PTYPE represents a parameter type, a syntactical template which parameters reference. + * sonic-clish.xsd defines the tag and the attributes associated with the PTYPE tag. + * Whenever a new attribute or tag introduced the xsd rules should also be updated. + * sonic-clish.xsd is avilable at sonic-mgmt-framework/CLI/clitree/scripts/ + * The klish source supports certain primitive PTYPEs. + * New user defined PTYPEs can be added to the SONIC project and can be defined in sonic_types.xml file. + * sonic_types.xml is available in sonic-mgmt-framework/CLI/clitree/cli-xml/sonic_types.xml + + **Example**: STRING_32 + + ```xml + + ``` + +2. MACROS + * Macros can be introduced by defining them in _macro.xml + * Macros are used where repetitive command snippets (or command parameter snippets) with minor variations are needed in XML files. + * It is possible to create a macro definition in a _macro.xml file and use the macro in the .xml file. + * For cases where variations in the values of these macro options are needed, the XML attributes can be passed as arguments to macro and substituted as values inside the macro definition. + * As part of parser tree preprocessing during Make, the referenced macro is expanded with the macro definitions and then the parser file is exported to the target XML files in CLI/target/command-tree to be consumed by Klish parser. + * One macro file can be written per module. + * The macro files are placed at sonic-mgmt-framework/CLI/clitree/macro/ + * Macros can also be nested with reference to another macro definition. + + **Example**: + + ```xml + + + + + + + ``` + + * The previous macro “IPV4-SRC-OPTIONS“ is used by the macro below "IPV4-ACL". + + ```xml + + + + + + + + + + + + + + + + + ``` + +3. Entities + + * Entities can be defined and referenced in the XML files. + * One common use case for entities is to specify platform specific parameters and parameter value limits in XML files. + * To define an entity based parameter value range limit, three tasks must be done: + + 1. Define the feature value as entity in mgmt_clish_feature_master.xsd + **Example**: + + ```xsl + + + This contains the allowed feature names for any platform specific customizable feature names. + If you are adding a new platform customizable feature, add a record in the enumerated list here. + + + + + + + + + + This contains the allowed feature-value names for any platform specific customizable feature-value + strings. If you are adding a new platform customizable entity, add a record in the enumerated list here. + + + + + + + ``` + + 2. Set the attribute 'value' to the right limit on respective platform configuration file that can be defined statically (used at build time). + Refer to a dummy platform (platform_dummy.xml) configuration file. + The platform_dummy.xml file can be located at sonic-mgmt-framework/CLI/clitree/scripts/. + + **Example**: + + MAX_MTU + + 3. Use the ENTITY just like any other XML ENTITY in the CLI parser XML file. + +4. Extended support with ext_help and regexp_select + The ext_help is similar to the pattern used in select method and used with regexp_select method. The value of the ext_pattern attribute is used for auto completion in regexp_select method. + + **Example**: + Interfaces can be defined with regexp_select method and ext_pattern to support abbreviated interface naming, i.e eth20, e20, Ether20 etc. Here, we are left the help part as empty, and define the help string in the place where we use this PTYPE. +``` + + + +``` + +#### 2.9.4 Preprocess XML files + +* The preprocessing scripts are invoked at compile time and no action is required to add a new CLI command. + +* This section gives information about what is done during preprocessing stage. + + * The preprocessing scripts preprocess the raw CLI XML files and generate a target XML file that can be consumed by the klish open source parser. + * The inputs to the preprocessing scripts are the raw CLI XML files, macro files and other utility files like platform specifics. + * The cli-xml files are validated as part of compilation. + * The 'xmllint' binary is used to validate all the processed XML files (i.e. after macro substitution and pipe processing) against the detailed schema kept at sonic-clish.xsd + * The following preprocessing scripts are introduced: + + * **klish_ins_def_cmd.py**: This script is used to append the "exit" and "end" commands to the views of the Klish XML files + * **klish_insert_pipe.py**: This script extends every show and get COMMAND with pipe option + * **klish_platform_features_process.sh**: Validate all platform XML files. Generate the entity.xml files. + * **klish_replace_macro.py**: This script does macro replacement on the XML files which are used by klish to define CLI structure. + +#### 2.9.5 CLI directory structure + +All CLI infrastructure, CLI definitions and their handlers are maintained +under `sonic-mgmt-framework/CLI` directory. + +``` +CLI +├── actioner /* CLI actioner scripts */ +├── clicfg /* Entity substitution */ +├── clitree +│   ├── cli-xml /* CLI model XMLs */ +│   ├── macro /* Macro definitions */ +│   └── scripts /* Preprocessor scripts */ +├── klish /* KLISH patches and launcher scripts */ +│   └── patches +└── renderer + ├── scripts /* Renderer scripts */ + └── templates /* Renderer jinja templates */ +``` + +Below directories are used to collect XML files and utility scripts to generate the target XML files for CLI command tree build-out. + +**clitree/cli-xml** - contains unprocessed/raw CLI XML files defined by developer + +**clitree/macro** - contains macro definitions used/referenced in CLI XML files defined by developer + +**clitree/scripts** - contains utility scripts to process raw CLI XML files defined by developer into processed CLI XMLs to be consumed by Klish Parser. + +**clitree/Makefile** - rules to preprocess the raw CLI XMLs and validate the processed output against the DTD in sonic-clish.xsd + +**clicfg** - files for platform specific entity substitution. + +**renderer** - templates and scripts to be used in rendering the show commands output. + +After compilation the processed CLI XMLs can be found in sonic-mgmt-framework/build/cli/command-tree + +#### 2.9.6 Generic REST Client for Actioner Scripts + +Actioner scripts should use the generic REST client for makeing to communicate to REST Server. +Generic client code is avalable at `sonic-mgmt-framework/CLI/actioner/cli_client.py`. +This is tailor made to connect REST Server on local switch and handle the CLI actioner usecases. + +Typical usage: + +Create a CLI client object. +Client object can be created as a global variable in every actioner script. +Server sonnection and session are managed internally. + +```python +import cli_client as cc + +api = cc.ApiClient() +``` + +Create a path object. It accepts parameterized path template and parameter values. +Path template is similar to the template used by Swagger UI. +Parameter values will be URL-encoded and substituted in the path template. + +```python +path = cc.Path('/restconf/data/openconfig-acl:acl/acl-sets/acl-set={name},{type}/acl-entries', + name='MyNewAccessList', type='ACL_IPV4') +``` + +Invoke REST API.. `ApiClient` object supports get, post, put, patch and delete operations. +All these operations send a REST request and return a response object wrapping the REST response data. + +```python +response = api.get(path, response_type=None) +``` + +An optional argument known as response_type is applicable only for GET and POST operations. +It instructs the REST Client infrastructure to return the response from REST server in the form of +JSON string instead of JSON (python dictionary). +If the API does not specify value for response_type, the default is JSON. +The supported values for response_type are string and json. + +**The use case for using response_type** + +If the response from the rest server contains a non-hierarchical JSON string and if it's quite big enough, +then the recommendation is to operate on the JSON string itself instead of converting it to python dictionary. +This will help the actioner consume less memory. It was observed that loading into dictionary takes considerably +large memory. + +Check API status through `response.ok()` function, which returns true if API was success (HTTP 2xx status). +`ok()` function returns false if server returned an error. +Use `response.error_message()` function to get the displayable error message. + +```python +if response.ok() { + respJson = response.content + # invoke renderer to display respJson +} else { + print response.error_message() + return -1 +} +``` + +Examples of other REST API calls. + +```python +# construct your request data json +jsonDict = {} +jsonDict["acl-set"]=[{ "name":the_acl_name, .... }] + +# POST request with default response_type i.e. JSON +response = api.post(path, data=jsonDict) + +# POST request with string as response_type +response = api.post(path, data=jsonDict, response_type='string') + +# PUT reuest +reponse = api.put(path, data=jsonDict) + +# PATCH request +response = api.patch(path, data=jsonDict) + +# GET request with default response_type i.e. JSON +response = api.get(path) + +# GET request with string as response_type +response = api.get(path, response_type='string') + +# DELETE request +response = api.delete(path) +``` + +Above example used `response.error_message()` function to get ser displayable error message from the REST response. +REST server always returns error information in standard RESTCONF error format. +The `error_message()` function looks for the **error-message** attribute to get user displayable message. +If there was no **error-message** attribute, a generic message will be returned based on **error-tag** value. +Actioners can override this logic through customiz error message formatter function. +The custom formtter function receives HTTP state code and RESTCONF error json; and should return a string. +However default error formatter is sufficient for most of the cases. + +```python +if not response.ok(): + print response.error_message(formatter_func=my_new_error_message_formatter) + +# ..... + +def my_new_error_message_formatter(status_code, error_entry): + if err_entry['error-type'] == 'protocol': + return "System error.. Please reboot!!" + return "Application error.. Please retry." +``` + +#### 2.9.7 Tool for JSON navigation in jinja templates + +Developers can use this tool to retrieve the JSON data efficiently and check for the existence of nodes. +The tool uses the path in the GNMI style for querying and checking the nodes. +Following are the APIs exposed by the tool + +#### get(json_data, path) + params + json_data: JSON data in the form of dictionary + path: The GNMI style path, examples below + return value + On success returns the queried nodes (format/type will be as present in JSON data) + on Failure returns None + +##### Example: + +```code +(Pdb) get(data, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/") +{'name': 'MyACL1', 'type': 'ACL_IPV4', 'description': 'Description for MyACL1'} + +``` + +#### get_str(json_data, path) + params + json_data: JSON data in the form of dictionary + path: The GNMI style path, examples below + return value + On success returns the queried nodes (format/type will be strigified) + on Failure returns None + +##### Example: + +```code +(Pdb) get_str(data, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/") +"{'name': 'MyACL1', 'type': 'ACL_IPV4', 'description': 'Description for MyACL1'}" + +``` + +#### contains(json_data, path) + params + json_data: JSON data in the form of dictionary + path: The GNMI style path, examples below + return value + On success returns True + on Failure returns False + +##### Example: + +```code +(Pdb) contains(data, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/") +True +(Pdb) contains(data, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/not_present") +False + +``` + +#### Sample usage in the renderer + +```text +{{json_tools.get(json_output, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]")}} +{{json_tools.get(json_output, "/acl-sets/acl-set[name=MyACL1]")}} +{{json_tools.get(json_output, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/type")}} +{{json_tools.get_str(json_output, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/type")}} +{{json_tools.contains(json_output, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/type_not")}} +{{json_tools.contains(json_output, "/acl-sets/acl-set[name=MyACL1][type=ACL_IPV4]/config/type")}} + +``` + + +### 2.10 REST Server + +sonic-mgmt-framework repository contains REST Server infrastructure code. +It provides [RESTCONF](https://tools.ietf.org/html/rfc8040) APIs for the YANGs defined in sonic-mgmt-common. +These APIs are serviced by Translib; see [HLD](https://github.com/Azure/SONiC/blob/master/doc/mgmt/Management%20Framework.md#3224-REST-server) for details. + +YANGs are automatically plugged into REST Server infrastructure during sonic-mgmt-framework compilation. +Details are in [section 3.1.5](#315-swagger-server-generator). + +### 2.11 gNMI + +gNMI interfaces are automatically supported for all YANGs definied under `sonic-mgmt-common/models/yang` directiry. +This includes OpenConfig & IETF YANGs with their extensions and SONiC YANGs. + +gNMI and telemetry infrastructure code is maintained in **sonic-telemetry** repository. + +### 2.12 Compilation + +#### 2.12.1 sonic-mgmt-common + +sonic-mgmt-common repository compiles into **sonic-mgmt-common_1.0.0_{arch}.deb** and +**sonic-mgmt-common-codegen_1.0.0_{arch}.deb** packages. + +The sonic-mgmt-common_1.0.0_{arch}.deb contains YANGs, YANG derivatives and configuration files +required by Translib, transformer and CVL. +These are installed in **mgmt-framework** and **telemetry** containers. + +sonic-mgmt-common-codegen_1.0.0_{arch}.deb contains YGOT bindings and other generated source files +which are required for sonic-mgmt-framework and sonic-telemetry compilation. +When package cache mode is enabled, the build infrastructure will not compile sonic-mgmt-common +files if there are no changes in it since last compilation. +Generated source files are restored by installing sonic-mgmt-common-codegen_1.0.0_{arch}.deb in +the build container. + +#### 2.12.2 sonic-mgmt-framework + +Code from sonic-mgmt-framework repository compiles into **sonic-mgmt-framework_1.0-01_{arch}.deb** +and a docker image **docker-sonic-mgmt-framework.gz**. +REST Server and CLI code are packaged in sonic-mgmt-framework_1.0-01_xyz.deb. +This and sonic-mgmt-common_1.0.0_{arch}.deb are included in the docker image. + +To rebuld the docker image both these debs should be cleaned. + +```bash +# Only if you have changed something in sonic-mgmt-common +find target -name "sonic-mgmt-common*" -delete -print + +# Always do it - for both mgmt-common and mgmt-framework changes +find target -name "*sonic-mgmt-framework*" -delete -print + +make target/docker-sonic-mgmt-framework.gz +``` + +To load new docker image on the switch: + +```bash +admin@sonic:~$ sudo sonic_installer upgrade_docker mgmt-framework docker-sonic-mgmt-framework.gz +``` + +#### 2.12.2 sonic-telemetry + +TODO + +### 2.13 Unit Testing + +#### 2.13.1 REST API testing + +REST server provides a test UI (Swagger UI) will display the operations defined in OpenAPI spec along with options to test it inline. To launch the Swagger UI, open “https://SWITCH_IP/ui” from the browser. Browser may complain that the certificate is not trusted. This is expected since the REST Server uses a temporary self signed certificate by default. Ignore the warning and proceed, this loads a home page as shown below, which contains links for OpenAPIs both derived from yang models and also manually written OpenAPI specs. + +![Swagger UI Home Page](images/homepage.jpeg) +Upon clicking of any one of the link, The Swagger UI correspoding to that openAPI spec will be opened as shown below. + +![OpenAPI Page](images/openapis.jpeg) + +Expand any section to view description, parameters and schema/sample data.. + +Use “Try it out” button test the API from the Swagger UI itself. It opens a form to enter URI parameters and body data. Enter values and hit “Execute” button. REST API will be invoked and results are shown. + +#### 2.13.2 Local REST Server and CLI + +TODO + +#### 2.13.3 gNMI Unit Testing + +gNMI unit testing can be performed using open-source gNMI tools. 'telemetry' docker already contains such tools e.g. gnmi_get, gnmi_set, gnmi_cli etc. You can directly invoke these tools from linux shell of a switch as below : + +``` +docker exec -it telemetry gnmi_get -xpath /openconfig-interfaces:interfaces/interface[name=Ethernet0] -target_addr 127.0.0.1:8080 -insecure + +``` + +* Use 'gnmi_get' tool to perform a get request against the gNMI target in switch. For sample get request and response, refer to https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_get.test and https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_get.result respectively. +* Use 'gnmi_set' tool to perform a set request against the gNMI target in switch. For sample set request and response, refer to https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_set.test and https://github.com/project-arlo/sonic-telemetry/blob/master/test/acl_set.result respectively. +* Use 'gnmi_cli' to get the capabilities of gNMI target. For example : + +``` +docker exec -it telemetry gnmi_cli -capabilities -insecure -logtostderr -address 127.0.0.1:8080 +``` +* Use 'gnmi_cli' to perform subscription request. For sample subscribe request and response, refer to https://github.com/project-arlo/sonic-telemetry/blob/master/test/ethernet_oper_status_subscribe.test and https://github.com/project-arlo/sonic-telemetry/blob/master/test/ethernet_oper_status_subscribe.result respectively. + +#### 2.13.3 Spytest + +TODO + + +## 3 Appendix + +### 3.1 Code Generators + +#### 3.1.1 YANG Tree Generator + +YANG tree generators are run during compilation of `sonic-mgmt-common/models/yang` directory. + +Following trees are generated: +1) Standard [Text tree](#2361-yang-text-tree) using `pyang -f tree` option. +2) Standard [HTML tree](#2362-yang-html-tree) using `pyang -f jstree` option. +3) Custom [Markdown tree](#2363-yang-markdown-tree) which highlights nodes added/removed by extension YANGs. +Uses a custom pyang plugin `sonic-mgmt-common/tools/pyang/pyang_plugins/doctree.py`. + +All tree generators are automatically run during sonic-mgmt-common compilation. +It can be manually trigged through following commands: + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-common/models/yang +make # generates all trees +make allyangs.tree # generates only text tree +make allyangs_tree.html # generates only html tree +make yang_tree.md # generates only md tree +``` + +Requires: +- Pyang 2.1.1 +- Python 2.7.14 or later (or Python 3) + +#### 3.1.2 YGOT Binding Generator + +This tool generates YGOT binding code for YANG files under `sonic-mgmt-common/models/yang` directory. +YGOT bindings are written to `sonic-mgmt-common/translib/ocbinds/ocbinds.go` file. + +It is run automatically during compilation of sonic-mgmt-common repository. +It can be manually triggered to test YANG changes through following steps. + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-common +make go-deps +cd -C translib ocbinds/ocbinds.go +``` + +Requires: +- Go 1.13.8 or later +- customized ygot (patched during sonic-mgmt-common compilation) + +#### 3.1.3 CVL Schema Generator + +This tool generates CVL YIN schema from the SONiC YANGs. +It removes all read-only state nodes, RPC and notification nodes using a +pyang plugin `sonic-mgmt-common/tools/pyang/pyang_plugins/yin_cvl.py`. + +Schema files will be generated under `sonic-mgmt-common/build/cvl/schema` directory. + +CVL Schema generator is automatically run during sonic-mgmt-common compilation. +It can be manually trigged through following commands: + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-common/cvl +make schema +``` + +Requires: +- Pyang 2.1.1 +- Python 2.7.14 or later (or Python 3) + +#### 3.1.4 OpenAPI Generator + +This tool generates OpenAPI specs (aka swagger spec) from YANGs. +It is implemented as a pyang plugin `sonic-mgmt-framework/tools/pyang/pyang_plugins/openapi.py`. + +YANG file to YAML file mapping is not one to one; i.e, there will not be a `.yaml` for every `.yang` file. +OpenAPI generator first constructs a normalized YANG tree using all the YANG files defined under +`sonic-mgmt-common/models/yang`, `sonic-mgmt-common/models/yang/extensions` +and `sonic-mgmt-common/models/yang/sonic` directories. +YANGs from `sonic-mgmt-common/models/yang/common` are used to resolve compilation dependencies only. +YAML files will be generated for each top level YANG module in normalized tree. + +Eg, below 3 YANG files will result in only one OpenAPI spec - main.yaml. + +```text ++---------------------------+-------------------------------+-------------------------------+ +| yang/main.yang | yang/common/sub1.yang | yang/extensions/sub2.yang | ++---------------------------+-------------------------------+-------------------------------+ +| module main { | module sub1 { | module sub2 { | +| container feature1 { | grouping newstuffs {...} | grouping morestuffs {...} | +| } | augment /main:feature1 { | augment /main:feature1 { | +| } | uses newstuffs; | uses morestuffs; | +| | } | } | +| | } | } | ++---------------------------+-------------------------------+-------------------------------+ +``` + +Generated OpenAPI specs will be written to `sonic-mgmt-framework/build/yaml` directory. + +Note that transformer annotations YANGs are not considered - they are not data models. + +OpenAPI Generator is automatically run during sonic-mgmt-framework compilation. +It can be manually trigged through following commands: + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-framework/models +make -f yang_to_openapi.mk +``` + +Requires: +- Pyang 2.1.1 +- mmh3 2.5.1 (python library) +- Python 2.7.14 or later (or Python 3) + +#### 3.1.5 Swagger Server Generator + +This tool generates Swagger Go server handler code from the OpenAPIs specs. +It consumes YANG generated OpenAPI specs from `sonic-mgmt-framework/build/yaml` directory +and manually written OpenAPI specs from `sonic-mgmt-framework/models/openapi` directory, if any. + +Generated Go source files will be available in `sonic-mgmt-framework/build/rest_server/dist/openapi` directory. +No manual edits are required in the generated server handler functions. +Customized mustache templates are used to generate the server handlers with REST Server specific logic. +Mustache templates are in `sonic-mgmt-framework/tools/swagger_codegen/go-server` directory. + +Swagger server generator is automatically run during sonic-mgmt-framework compilation. +It can be manually trigged through following commands (assumes OpenAPI generator is already run): + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-framework/models +make -f openapi_codegen.mk go-server +``` + +Requires: +- Java 1.8 +- Python 3 +- jinja2 2.10.3 + +#### 3.1.6 Swagger Client SDK Generator + +This tool generates swagger client SDK code in python from the OpenAPI specs. +Python client SDK was originally intended to be used by KLISH CLI actioners, +but now CLI actioners are using a generic REST client. +Hence this code generator is disabled by default. +It can be enabled for specific OpenAPIs specs by listing them in `sonic-mgmt-framework/models/codegen.config` file. + +```makefile +# Generated OpenAPI specs from sonic-mgmt-framework/build/yaml directory +PY_YANGAPI_CLIENTS += openconfig-interfaces +PY_YANGAPI_CLIENTS += openconfig-platform +PY_YANGAPI_CLIENTS += openconfig-system + +# OpenAPI specs from sonic-mgmt-framework/models/openapi directory +PY_OPENAPI_CLIENTS += my-new-apis +``` + +Generated client SDK code will be available in `sonic-mgmt-framework/build/openapi_client_py` directory. + +Note: Swagger generates lot of client SDK code but only fraction of it will be exercised by KISH CLI. +Framework provides a generic REST client for CLI use which is more lighter and faster than swagger client SDK. +**Recommendation for CLIs is to use this generic REST client and not Swagger Client SDK**. + +Swagger client SDK generator is automatically run during sonic-mgmt-framework compilation +if some OpenAPI specs are whitelisted in `sonic-mgmt-framework/models/codegen.config`. +It can be manually run through following commands (assumes OpenAPI generator is already run): + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-framework/models +make -f openapi_codegen.mk py-client # generates for all whitelited specs + +# To force generation for a specific spec +PY_YANGAPI_CLIENTS=openconfig-interfaces make -f openapi_codegen.mk py-client +``` + +Requires: +- Java 1.8 + +#### 3.1.7 Swagger UI Generator + +This tool generates Sweagger UI for the REST Server as expaned in [section 2.13.1](#2131-rest-api-testing). +It is automatically run along with [Swagger Server Generator](#315-swagger-server-generator). + +#### 3.1.8 RESTCONF Documentation Generator + +RESTCONF documentation is generated automatically as part of build during the phase when OpenAPIs(YAML) for YANG modules are generated. +No extra step is needed from developers. RESTCONF document generator is a submodule in OpenAPI Generator and it relies on YANG's description +statements for documentation text. + +Developers are requested to have a description statements in YANG models for all Data nodes such as container/list/leaf/leaf-list/rpc. +For any data node which does not have description statement, the generated document will have blank text i.e. empty sections + +The generated document will be in a github complaint markdown format (.md) +The document will be generated for each top level YANG module in normalized tree. For more information on how generator works please refer [section 3.1.4](#314-OpenAPI-Generator) + +The generated documents will be written to `sonic-mgmt-framework/build/restconf_md` directory. + +#### 3.1.9 CLI Documentation Generator + +CLI Document generator tool will generate the documentation using KLISH CLI model XMLs. +Command name, mode, syntax and parameter description are already available in model XML. +Additional tags will be introduced for developers to specify command description, usage info and examples for each COMMAND. + +- DOCGEN tag to group all document generation related tags - it can include one DESCRIPTION, one USAGE and one or more EXAMPLE, ALTCMD and FEATURE tags. +- DESCRIPTION tag to provide a detailed description of the command. +- USAGE tag to specify usage guidelines - when to use, preconditions, suggested next actions, etc. +- EXAMPLE tags to specify examples and sample output. +- ALTCMD tags to specify alternative commands in other CLI framework such as VTYSH, CLICK and in fact other alternate commands in KLISH itself (if available). + +All these documentation tags will be optional. They are used only for document generation and not to render the commands. + +Document generator tool will consume all model XMLs from src/CLI/clitree/cli-xml directory to generate CLI document in markdown syntax. +Below table summarizes how each section of document template will be derived from above defined XML tags. + +```text ++----------------+-----------------+-------------------+---------------------------------------------------------------------------+ +| Section. | XML TAG | Attribute | Comments | ++----------------+-----------------+-------------------+---------------------------------------------------------------------------+ +| Command name | COMMAND | | | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------+ +| Description | DESCRIPTION | | Use “help” value of COMMAND tag if DESCRIPTION tag is not specified. | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------| +| Syntax | PARAM | name | Lists all possible combinations of parameters | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------| +| Parameters | PARAM | name, help, ptype | | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------| +| Modes | VIEW | name | Parent command name will be shown here. The VIEW “name” attribute provides| +| | | | internal name of the mode. Tool will lookup the parent command by matching| +| | | | the VIEW “name” value with other COMMAND tag’s “view” attribute value | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------| +| Usage | | | | +| guidelines | USAGE | | Will be skipped if developer did not specify USAGE tag | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------| +| Examples | EXAMPLE | | Will be skipped if developer did not specify EXAMPLE tags | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------| +| Features | FEATURE | | Will be skipped if developer did not specify FEATURE tags | +|----------------|-----------------|-------------------|---------------------------------------------------------------------------| +| ALTCMDS | ALTCMD | type | Will be skipped if developer did not specify ALTCMD tags | ++----------------+-----------------|-------------------|---------------------------------------------------------------------------| + +``` +CLI document generator is automatically run during sonic-mgmt-framework compilation. +It can be manually trigged through following commands: + +```bash +cd /sonic/sonic-buildimage/src/sonic-mgmt-framework +make clidocgen (to generate the document) +make clidocgen-clean (to clean the document) +``` +The document will be written to `sonic-mgmt-framework/build/cli/command-tree/industry_standard_cli_reference_guide.md` directory. + +#### 3.1.9.1 Sample CLI Model + +```text + + + + + + + ... + + + Detailed command description + + + Usage information + + + Example1 body + + + Example2 body + + + ZTP + ZTP-cli + + config ztp -y enable + + + .... + + + + + + + +``` + +#### 3.1.9.2 Sample Generated Document + +```text +### command-tokens + +Detailed command description + +#### Syntax + +command-tokens param1 +command-tokens param2 + +#### Parameters + +Name | Description | Data type +-------|------------------------|-------- +Param1 | Help string for param1 | UINT +Param2 | Help string for param2 | STRING + +#### Mode + +Parent command - derived from view-name + +#### Usage Guidelines + +Usage information + +#### Examples + +Example1 description + +Example1 body + +Example2 description + +Example2 body + +### Alternate Commands + +#### click +config ztp -y enable + +#### vtysh +... + +### Features this CLI belongs to +* ZTP +* ZTP-cli + +``` +### 3.2 Debugging + +#### 3.2.1 Debugging performance issues + +For debugging performance issues we need to collect the profiling numbers which the REST query or the KLISH CLI is executed. Following is the procedure to collect the same. + +1. Restart the rest-server (command: "systemctl restart mgmt-framework") as root user (This is to ensure the old data is not collected as part of profiling) +2. From the client, execute only the REST API query or KLISH CLI for which you need to collect the profiling data +3. Go to the mgmt-framework docker (command: docker exec -it mgmt-framework bash) +4. Send the "SIGUSR1" signal to the rest process (command: Kill -10 `pidof rest_server`) +5. Copy the /var/log/rest_server/cpu.pprof.1 and /usr/sbin/rest_server file to your VDI +6. To generate the report in a text file (command: go tool pprof --txt ./rest_server ./cpu.pprof.1 > report.txt) +7. You would need to install the "graphviz" package to generate the pdf report (command: sudo apt-get install -y graphviz) +8. To generate the report in pdf format (command: go tool pprof --pdf ./rest_server ./ cpu.pprof.1 > report.pdf) + +The generated pdf will give the profiling information of all the calls in the REST server during the REST query or KLISH CLI. From this data we will come to know which function in the REST server is consuming more time during the execution. + diff --git a/doc/mgmt/Docker to Host communication.md b/doc/mgmt/Docker to Host communication.md new file mode 100644 index 0000000000..85b19d0231 --- /dev/null +++ b/doc/mgmt/Docker to Host communication.md @@ -0,0 +1,204 @@ +# Feature Name +Docker to Host communication + +# High Level Design Document +#### Rev 0.2 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/28/2019 | Nirenjan Krishnan | Initial version | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.2 | 12/08/2019 | Mike Lazar | Add details about architecture | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.3 | 12/16/2019 | Mike Lazar | Add security and logging info | + + +# About this Manual +This document provides general information about the Docker to Host +communication feature in SONiC. + +# Scope +This document describes the high level design of Docker to Host communication. +This describes the infrastructure provided by the feature, and example usage, +however, it does not describe the individual host-specific features. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|---------------------------------------------------| +| D-Bus | Desktop Bus: https://en.wikipedia.org/wiki/D-Bus | + +# 1 Feature Overview + +The management framework runs within a Docker container, and performs actions +translating the user CLIs or REST requests to actions. Most of these actions +perform some operation on the Redis database, but some of them require +operations to be done on the host, i.e., outside the container. This document +describes the host server, and translib API that are used to communicate between +the Docker container and the host. + + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +* The SONiC Management Framework and Telemetry containers must be able to issue + requests to the host, and return the responses from the host. +* The individual applications that need access to the host must be able to + create a host module and easily issue requests and get responses back from the + host. +* The host communication API shall be available in Translib, and shall provide + both synchronous and asynchronous communication methods. +* It shall be possible to configure the identity of the Linux user accounts who have access to a D-Bus socket. +* It shall be possible to configure containers in such a way that only certain containers (e.g. SONiC Mgmt.) + have access to the D-Bus socket. + +### 1.1.2 Configuration and Management Requirements + +N/A + +### 1.1.3 Scalability Requirements + +N/A + +### 1.1.4 Warm Boot Requirements + +N/A + +## 1.2 Design Overview +### 1.2.1 Basic Approach + +The code will extend the existing Translib modules to provide a D-Bus based +query API to issue requests to the host. The host service will be a Python based +application which listens on known D-Bus endpoints.https://en.wikipedia.org/wiki/D-Bus + +The individual app modules can extend the host service by providing a small +Python snippet that will register against their application endpoint. + +### 1.2.2 Container + +SONiC Management Framework, gNMI Telemetry containers + +### 1.2.3 SAI Overview + +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases + +All deployments + +## 2.2 Functional Description + +This feature enables management applications to issue +requests to the host to perform actions such as: +* image install / upgrade +* ZTP enable/disable +* initiate reboot and warm reboot using existing scripts +* create show-tech tar file using existing show-tech script +* config save/reload using existing scripts + +# 3 Design +## 3.1 Overview + +The feature extends the SONiC management framework to add a D-Bus service on the +host. This service will register against a known endpoint, and will service +requests to the endpoint. Application modules will add snippets to the host +service, which will automatically register their endpoints, and the app module +in the container can use the APIs provided in Translib to send the request to +the host, and either wait for the response (if the request was synchronous), or +receive a channel and wait for the request to return the response on the +channel (asynchronous request). + +The architecture of a D-Bus host service in a SONiC environment is illustrated in the diagram below: +![](images/docker-to-host-services-architecture.jpg) + +Note. The Linux D-Bus implementation uses Unix domain sockets for client to D-Bus service communications. +All containers that use D-Bus services will bind mount +(-v /var/run/dbus:/var/run/dbus:rw) the host directory where D-Bus service sockets are created. +This ensures that only the desired containers access the D-Bus host services. + +D-Bus provides a reliable communication channel between client (SONiC management container) and service (native host OS) – all actions are acknowledged and can provide return values. It should be noted that acknowledgements are important for operations such as “image upgrade” or “config-save”. In addition, D-Bus methods can return values of many types – not just ACKs. For instance, they can return strings, useful to return the output of a command. + +### 3.1.1 Security of D-Bus Communications +In addition to standard Linux security mechanisms for file/Unix socket access rights (read/write), D-Bus provides a separate security layer, using the D-Bus service configuration files. +This allows finer grain access control to D-Bus objects and methods - D-Bus can restrict access only to certain Linux users. + +### 3.1.2 Command Logging + +It is possible to track and log the user name and the command that the user has requested. +The log record is created in the system log. + +## 3.2 DB Changes +### 3.2.1 CONFIG DB +N/A +### 3.2.2 APP DB +N/A +### 3.2.3 STATE DB +N/A +### 3.2.4 ASIC DB +N/A +### 3.2.5 COUNTER DB +N/A + +## 3.3 Switch State Service Design +### 3.3.1 Orchestration Agent +N/A +### 3.3.2 Other Process +N/A +## 3.4 SyncD +N/A +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models +N/A +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands +N/A +#### 3.6.2.2 Show Commands +N/A +#### 3.6.2.3 Debug Commands +N/A +#### 3.6.2.4 IS-CLI Compliance +N/A +### 3.6.3 REST API Support +N/A + +# 4 Flow Diagrams + +![](images/docker-to-host-service.svg) + +# 5 Error Handling + +The `hostQuery` and `hostQueryAsync` APIs return a standard Go `error` object, +which can be used to handle any errors that are returned by the D-Bus +infrastructure. + +# 6 Serviceability and Debug +N/A + +# 7 Warm Boot Support +N/A + +# 8 Scalability +N/A + +# 9 Unit Test +List unit test cases added for this feature including warm boot. + +# 10 Internal Design Information +N/A diff --git a/doc/mgmt/Management Framework.md b/doc/mgmt/Management Framework.md new file mode 100644 index 0000000000..c61ba7d589 --- /dev/null +++ b/doc/mgmt/Management Framework.md @@ -0,0 +1,2568 @@ +# SONiC Management Framework + +## High level design document + +### Rev 0.17 + +## Table of Contents + +* [List of Tables](#list-of-tables) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Definition/Abbreviation](#definitionabbreviation) +* [Table 1: Abbreviations](#table-1-abbreviations) +* [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + * [1.2 Design Overview](#12-design-overview) + * [1.2.1 Basic Approach](#121-basic-approach) + * [1.2.2 Container](#122-container) +* [2 Functionality](#2-functionality) + * [2.1 Target Deployment Use Cases](#21-target-deployment-use-cases) +* [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.1.1 Build time flow](#311-build-time-flow) + * [3.1.2 Run time flow](#312-run-time-flow) + * [3.1.2.1 CLI](#3121-cli) + * [3.1.2.2 REST](#3122-REST) + * [3.1.2.3 gNMI](#3123-gnmi) + * [3.2 SONiC Management Framework Components](#32-sonic-management-framework-components) + * [3.2.1 Build time components](#321-build-time-components) + * [3.2.1.1 Yang to OpenAPI converter](#3211-yang-to-openapi-converter) + * [3.2.1.1.1 Overview](#32111-overview) + * [3.2.1.1.2 Supported HTTP verbs](#32112-supported-http-verbs) + * [3.2.1.1.3 Supported Data Nodes](#32113-supported-data-nodes) + * [3.2.1.1.4 Data Type Details](#32114-data-type-details) + * [3.2.1.1.5 Special Handling](#32115-special-handling) + * [3.2.1.1.6 Future enhancements](#32115-future-enhancements) + * [3.2.1.2 OpenAPI generator](#3212-OpenAPI-generator) + * [3.2.1.3 YGOT generator](#3213-YGOT-generator) + * [3.2.1.4 pyang compiler](#3214-pyang-compiler) + * [3.2.2 Run time components](#322-run-time-components) + * [3.2.2.1 CLI](#3221-cli) + * [3.2.2.2 REST Client SDK](#3222-REST-client-sdk) + * [3.2.2.3 gNMI Client](#3223-gnmi-client) + * [3.2.2.4 REST server](#3224-REST-server) + * [3.2.2.4.1 Transport options](#32241-transport-options) + * [3.2.2.4.2 Translib linking](#32242-Translib-linking) + * [3.2.2.4.3 Media Types](#32243-media-types) + * [3.2.2.4.4 Payload Validations](#32244-payload-validations) + * [3.2.2.4.5 Concurrency](#32245-concurrency) + * [3.2.2.4.6 API Versioning](#32246-api-versioning) + * [3.2.2.4.7 RESTCONF Entity-tag](#32247-restconf-entity-tag) + * [3.2.2.4.8 RESTCONF Discovery](#32248-restconf-discovery) + * [3.2.2.4.9 RESTCONF Query Parameters](#32249-restconf-query-parameters) + * [3.2.2.4.10 RESTCONF Operations](#322410-restconf-operations) + * [3.2.2.4.11 RESTCONF Notifications](#322411-restconf-notifications) + * [3.2.2.4.12 Authentication](#322412-authentication) + * [3.2.2.4.13 Error Response](#322413-error-response) + * [3.2.2.4.14 DB Schema](#322414-db-schema) + * [3.2.2.4.15 API Documentation](#322415-api-documentation) + * [3.2.2.5 gNMI server](#3225-gnmi-server) + * [3.2.2.5.1 Files changed/added](#32251-files-changed/added) + * [3.2.2.5.2 Sample Requests](#32253-sample-requests) + * [3.2.2.6 Translib](#3226-Translib) + * [3.2.2.6.1 App Interface](#32261-app-interface) + * [3.2.2.6.2 Translib Request Handler](#32262-Translib-request-handler) + * [3.2.2.6.3 YGOT request binder](#32263-YGOT-request-binder) + * [3.2.2.6.4 YANG Model Versioning](#32264-yang-model-versioning) + * [3.2.2.6.5 DB access layer](#32265-db-access-layer) + * [3.2.2.7 Transformer](#3227-transformer) + * [3.2.2.7.1 Components](#32271-components) + * [3.2.2.7.2 Design](#32272-design) + * [3.2.2.7.3 Process](#32273-process) + * [3.2.2.7.4 Common App](#32274-common-app) + * [3.2.2.7.5 YANG Extensions](#32275-yang-extensions) + * [3.2.2.7.6 Public Functions](#32276-public-functions) + * [3.2.2.7.7 Overloaded Modules](#32277-overloaded-modules) + * [3.2.2.7.8 Utilities](#32278-utilities) + * [3.2.2.8 Config Validation Library (CVL)](#3228-config-validation-library-cvl) + * [3.2.2.8.1 Architecture](#32281-architecture) + * [3.2.2.8.2 Validation types](#32282-validation-types) + * [3.2.2.8.3 CVL APIs](#32283-cvl-apis) + * [3.2.2.9 Redis DB](#3229-redis-db) + * [3.2.2.10 Non DB data provider](#32210-non-db-data-provider) +* [4 Flow Diagrams](#4-flow-diagrams) + * [4.1 REST SET flow](#41-REST-set-flow) + * [4.2 REST GET flow](#42-REST-get-flow) + * [4.3 Translib Initialization flow](#43-Translib-initialization-flow) + * [4.4 gNMI flow](#44-gNMI-flow) + * [4.5 CVL flow](#45-CVL-flow) +* [5 Developer Work flow](#5-Developer-Work-flow) + * [5.1 Developer work flow for custom (SONiC/CVL) YANG](#51-Developer-work-flow-for-custom-SONiCCVL-YANG) + * [5.1.1 Define Config Validation YANG schema](#511-Define-Config-Validation-YANG-schema) + * [5.1.2 Generation of REST server stubs and Client SDKs for YANG based APIs](#512-Generation-of-REST-server-stubs-and-Client-SDKs-for-YANG-based-APIs) + * [5.1.3 Config Translation App (Go language)](#513-Config-Translation-App-Go-language) + * [5.1.4 IS CLI](#514-IS-CLI) + * [5.1.5 gNMI](#515-gNMI) + * [5.2 Developer work flow for standard (OpenConfig/IETF) YANG](#52-Developer-work-flow-for-standard-OpenConfigIETF-YANG) + * [5.2.1 Identify the standard YANG module for the feature for northbound APIs](#521-Identify-the-standard-YANG-module-for-the-feature-for-northbound-APIs) + * [5.2.2 Define the Redis schema for the new feature. (not applicable for legacy/existing feature)](#522-Define-the-Redis-schema-for-the-new-feature-not-applicable-for-legacyexisting-feature) + * [5.2.3 Define Config Validation YANG schema](#523-Define-Config-Validation-YANG-schema) + * [5.2.4 Generation of REST server stubs and Client SDKs for YANG based APIs](#524-Generation-of-REST-server-stubs-and-Client-SDKs-for-YANG-based-APIs) + * [5.2.5 Config Translation App (Go language)](#525-Config-Translation-App-Go-language) + * [5.2.6 IS CLI](#526-IS-CLI) + * [5.2.7 gNMI](#527-gNMI) +* [6 Error Handling](#6-error-handling) +* [7 Serviceability and Debug](#7-serviceability-and-debug) +* [8 Warm Boot Support](#8-warm-boot-support) +* [9 Scalability](#9-scalability) +* [10 Unit Test](#10-unit-test) +* [11 Appendix A](#11-appendix-a) +* [12 Appendix B](#11-appendix-b) + + +## List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision + +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:-----------------------:|-----------------------------------| +| 0.1 | 06/13/2019 | Anand Kumar Subramanian | Initial version | +| 0.2 | 07/05/2019 | Prabhu Sreenivasan | Added gNMI, CLI content from DELL | +| 0.3 | 08/05/2019 | Senthil Kumar Ganesan | Updated gNMI content | +| 0.4 | 08/07/2019 | Arun Barboza | Clarifications on Table CAS | +| 0.5 | 08/07/2019 | Anand Kumar Subramanian | Translib Subscribe support | +| 0.6 | 08/08/2019 | Kwangsuk Kim | Updated Developer Workflow and CLI sections | +| 0.7 | 08/09/2019 | Partha Dutta | Updated Basic Approach under Design Overview | +| 0.8 | 08/15/2019 | Anand Kumar Subramanian | Addressed review comments | +| 0.9 | 08/19/2019 | Partha Dutta | Addressed review comments related to CVL | +| 0.10 | 09/25/2019 | Kwangsuk Kim | Updated Transformer section | +| 0.11 | 09/30/2019 | Partha Dutta | Updated as per SONiC YANG guideline | +| 0.12 | 10/19/2019 | Senthil Kumar Ganesan | Added Appendix B | +| 0.13 | 11/27/2019 | Anand Kumar Subramanian | Added new APIs in translib | +| 0.14 | 12/03/2019 | Sachin Holla | RESTCONF yang library and other enhancements | +| 0.15 | 12/19/2019 | Partha Dutta | Added new CVL API, platform and custom validation details | +| 0.16 | 04/08/2020 | Sachin Holla | API versioning enhancement | +| 0.17 | 04/08/2020 | Mohammed Faraaz | OpenAPI 3.0 enhancements | +| 0.18 | 04/09/2020 | Kwangsuk Kim | Updated CLI and Transformer enhancement | + +## About this Manual + +This document provides general information about the Management framework feature implementation in SONiC. + +## Scope + +This document describes the high level design of Management framework feature. + +## Definition/Abbreviation + +### Table 1: Abbreviations + +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| CVL | Config Validation Library | +| NBI | North Bound Interface | +| ABNF | Augmented Backus-Naur Form | +| YANG | Yet Another Next Generation | +| JSON | Java Script Object Notation | +| XML | eXtensible Markup Language | +| gNMI | gRPC Network Management Interface | +| YGOT | YANG Go Tools | + +## 1 Feature Overview + +Management framework is a SONiC application which is responsible for providing various common North Bound Interfaces (NBIs) for the purposes of managing configuration and status on SONiC switches. The application manages coordination of NBI’s to provide a coherent way to validate, apply and show configuration. + +### 1.1 Requirements + +* Must provide support for: + + 1. Standard [YANG](https://tools.ietf.org/html/rfc7950) models (e.g. OpenConfig, IETF, IEEE) + 2. Custom YANG models ([SONiC YANG](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md)) + 3. Industry-standard CLI / Cisco like CLI + +* Must provide support for [OpenAPI spec](https://OpenAPI.io/specification/) to generate REST server side code +* Must provide support for NBIs such as: + + 1. CLI + 2. gNMI + 3. REST/RESTCONF + +* Must support the following security features: + + 1. Certificate-based authentication + 2. User/password based authentication + 3. Role based authorization + +* Ease of use for developer workflow + + 1. Specify data model and auto-generate as much as possible from there + +* Must support Validation and Error Handling - data model, platform capability/scale, dynamic resources +* SNMP integration in SONiC is left for future study + +### 1.2 Design Overview + +Management framework makes use of the translation library (Translib) written in golang to convert the data models exposed to the management clients into the Redis ABNF schema format. Supported management servers can make use of the Translib to convert the incoming payload to SONiC ABNF schema and vice versa depending on the incoming request. Translib will cater to the needs of REST and gNMI servers. Later the Translib can be enhanced to support other management servers if needed. This framework will support both standard and custom YANG models for communication with the corresponding management servers. Management framework will also take care of maintaining data consistency, when writes are performed from two different management servers at the same time. Management framework will provide a mechanism to authenticate and authorize any incoming requests. Management framework will also take care of validating the requests before writing them into the Redis DB. Config Validation Library is used for syntactic and semantic validation of ABNF JSON based on YANG derived from Redis ABNF schema. + +#### 1.2.1 Basic Approach + +* Management framework takes comprehensive approach catering: + * Standard based YANG Models and custom YANG + * Open API spec + * Industry standard CLI + * Config Validation +* REST server, gNMI server, App module and Translib - all in Go +* Translation by using the Translib Library and application specific modules +* Marshalling and unmarshalling using YGOT +* Redis updated using CAS(Check-and-Set) trans. (No locking, No rollback) +* Config Validation by using YANG model from ABNF schema +* CLI with Klish framework + +#### 1.2.2 Container + +The management framework is designed to run in a single container named “sonic-mgmt-framework”. The container includes the REST server linked with Translib, and CLI process. +The gNMI support requires the gNMI server which is provided as a part of sonic-telemetry container. We would like to rename this container as the sonic-gnmi container as now it can perform configurations as well through the gNMI server. +Will introduce a new container sonic-mgmt-common to host the common code that is used both in the mgmt-framework and sonic-telemetry container. This new repo will compile into static libraries that will be used in the other two repos. This way sonic-telemetry repo can be compiled without the mgmt-framework being present in the code base. + +## 2 Functionality + +### 2.1 Target Deployment Use Cases + +1. Industry Standard CLI which will use REST client to talk to the corresponding servers to send and receive data. +2. REST client through which the user can perform POST, PUT, PATCH, DELETE, GET operations on the supported YANG paths. +3. gNMI client with support for capabilities, get, set, and subscribe based on the supported YANG models. + +## 3 Design + +### 3.1 Overview + +The SONiC management framework comprises two workflows: + +1. Build time flow +2. Run time flow + +as show in the architecture diagram below. + +![Management Framework Architecture diagram](images/Mgmt_Frmk_Arch.jpg) + +#### 3.1.1 Build time flow + +The Developer starts by defining the desired management objects and the access APIs to provide for the target application. This can be done in one of the two ways: - +1) A YANG data model +2) An OpenAPI spec + +This can be an independent choice on an application by application basis. However note that using YANG allows for richer data modelling, and therefore superior data validation. + +1. In case of YANG, if the developer chooses standard YANG model (Openconfig, IETF etc.), a separate SONiC YANG model has to be written based on Redis ABNF schema for validating Redis configuration and transformer hints should be written in a deviation file for standard YANG model to Redis DB coversion and vice versa (refer to [3.2.2.7 Transformer](#3227-transformer) for details). However, if custom SONiC YANG model is written based on guidelines, CVL YANG is automatically derived from it and the same is used for validation purpose and there is no need of writing any deviation file for transformer hints. Based on the given YANG model as input, the pyang compiler generates the corresponding OpenAPI spec which is in turn given to the OpenAPI generator to generate the REST client SDK and REST server stubs in golang. The YANG data model is also provided to the [YGOT](https://github.com/openconfig/YGOT) generator to create the YGOT bindings. These are used on the interface between Translib and the selected App module. Specifically, Translib populates the binding structures based upon the incoming server payload, and the App module processes the structure accordingly. Additionally, a YANG annotation file must also be provided, for data models that do not map directly to the SONiC YANG structure. The requests in this case will be populated into the YGOT structures and passed to App module for conversion. The App module uses the YANG annotations to help convert and map YANG objects to DB objects and vice-versa. + +2. In case of OpenAPI spec, it is directly given to the [OpenAPI](https://OpenAPI.io) generator to generate the REST client SDK and REST server stubs in golang. In this case the REST server takes care of validating the incoming request to be OpenAPI compliant before giving the same to Translib. There is no YANG, and therefore no YGOT bindings are generated or processed, and so the Translib infra will invoke the App module functions with the path and the raw JSON for App modules to convert. For configuration validation purpose, SONiC YANG model has to be written based on Redis ABNF schema. + +#### 3.1.2 Run time flow + +##### 3.1.2.1 CLI + +1. CLI uses the KLISH framework to provide a CLI shell. The CLI request is converted to a corresponding REST client request using c, and is sent to the REST server. +2. The OpenAPI generated REST server handles all the REST requests from CLI and invokes a common handler for all the create, update, replace, delete and get operations along with path and payload. This common handler converts all the requests into Translib arguments and invokes the corresponding Translib provided APIs. +3. Translib API then calls the corresponding handler defined in Common App module, subsequently invokes Transformer functions with YGOT structured request to perform model to model transformation, i.e. Openconfig Model to SONiC model, to generate the request DB map. +4. Common App handlers pass the request DB map to DB Access module to access REDIS Database, process the request DB map. The response to GET operation is passed to Transformer to perform reverse transformation to Openconfig Model, and wired out to CLI as JSON response. +5. Further processing of CLI commands will be handled by Translib componenets that will be discussed in more detail in the later sections. + +##### 3.1.2.2 REST + +1. REST client will use the OpenAPI generated client SDK to send the request to the REST server. +2. From then on the flow is similar to the one seen in the CLI. + +##### 3.1.2.3 gNMI + +GNMI service defines a gRPC-based protocol for the modification and retrieval of configuration from a target device, as well as the control and generation of telemetry streams from a target device to a data collection system. Refer [GNMI spec](https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md) + +![GNMI Service High level diagram (Proposed)](images/GNMI_Server.png) + +1. Existing SONiC telemetry framework has been extended to support the new GNMI services. +2. All 4 GNMI services are supported: Get, Set, Capabilities and Subscribe. +3. A new transl data client is added to process the incoming YANG-based request (either standard or proprietary) +4. The new transl data client relies on Translib infra provided to translate, get, set of the YANG objects. + +More details on the GNMI server, Client and workflow provided later in the document. + +### 3.2 SONiC Management Framework Components + +Management framework components can be classified into + +1. Build time components +2. Run time components + +#### 3.2.1 Build time components + +Following are the build time components of the management framework + +1. Pyang compiler (for YANG to OpenAPI conversion) +2. OpenAPI generator +3. YGOT generator +4. pyang compiler (for YANG to YIN conversion) + +##### 3.2.1.1 YANG to OpenAPI converter + +##### 3.2.1.1.1 Overview + +Open source Python-based YANG parser called pyang is used for YANG parsing and building a Python object dictionary. A custom plugin is developed to translate this Python object dictionary into an OpenAPI spec. The OpenAPI specs are [3.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.3.md#schemaObject) complaint. + +URI format and payload is RESTCONF complaint and is based on the [RFC8040](https://tools.ietf.org/html/rfc8040). The Request and Response body is in JSON format in this release. + +##### 3.2.1.1.2 Supported HTTP verbs + +Following table lists the HTTP methods generated for different types of YANG nodes. + + YANG node type | HTTP methods +------------------------|----------------- +Configuration data | POST, PUT, PATCH, DELETE, GET, HEAD +Non configuration data | GET, HEAD +YANG RPC | POST + +##### 3.2.1.1.3 Supported Data Nodes + +For each of the below-listed Data keywords nodes in the YANG model, the OpenAPI (path) will be generated + +* Container +* List +* Leaf +* Leaf-list +* Rpc + +##### 3.2.1.1.4 Data Type Details + + YANG Type | OpenAPI Type +--------------|------------------ +int8 | Integer +int16 | Integer +int32 | Integer +int64 | Integer +uint8 | Integer +uint16 | Integer +uint32 | Integer +uint64 | Integer +decimal64 | Number +String | string +Enum | Enum +Identityref | String (Future can be Enum) +long | Integer +Boolean | Boolean +Binary | String with Format as Binary +bits | integer + +* All list keys will be made mandatory in the payload and URI +* YANG mandatory statements will be mapped to the required statement in OpenAPI +* Default values, Enums are mapped to Default and Enums statements of OpenAPI +* Since OpenAPI spec does not provide a way to define an unsigned types, a custom vendor extension x-yang-type is introduced which will contain the actual yang data type of a leaf/leaf-list. This vendor extension will be part of type object. + +Example below + +``` + /restconf/data/ietf-snmp:snmp/tlstm/cert-to-name={id}: + put: + tags: + - ietf-snmp + parameters: + - name: id + in: path + required: true + schema: + format: int32 + maximum: 4294967295 + minimum: 0 + type: integer + **x-yang-type: uint32** +``` +* As seen in above example all Integer types will have maximum and minimum fields. If the range statement is defined on the YANG node, then the value for maximum and minimum is derived from the range statement otherwise default range specified by YANG will be considered. And also a custom vendor extension called x-range will include a copy of range statement's value. If a type has one more range statements (may be indirectly using typedefs) all the range statements will be included as part of x-range extension separated by '|' character. + +Exampe below +``` + - format: int32 + **maximum: 2147483647** + **minimum: 1** + type: integer + **x-range: 1..2147483647 | 4..10** + x-yang-type: int32 +``` +* A string data types will have a vendor extension x-pattern whose value will be a regex. This field will only be present when the YANG node has it. Although OpenAPI 3.0 has a pattern field under schema object, but it is not used because the regex required for OpenAPI 3.0 is a W3C complaint regex but Openconfig models follow POSIX style regex. If a type has one more regex statements (may be indirectly using typedefs) all the regex statements will be included as part of x-pattern extension, they will be ANDed together. + +Example below +``` + - maxLength: 18446744073709551615 + minLength: 0 + type: string + **x-pattern: (([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?** + x-yang-type: string +``` + +* As seen in above example all String types will have maxLength and minLength fields. If the length statement is defined on the YANG node, then the value for maxLength and minLength is derived from the length statement otherwise default length specified by YANG will be considered. And also a custom vendor extension called x-length will include a copy of length statement's value. If a type has one more length statements (may be indirectly using typedefs) all the length statements will be included as part of x-length extension separated by '|' character. + +* An Union data type is supported through oneOf directive. oneOf is a JSON schema directive supported by openAPI 3.0. It takes an array of objects and at any given time only one of item can be used. + +Example below +``` + - name: security-model + in: path + required: true + schema: + oneOf: + - enum: + - v1 + - v2c + - usm + - tsm + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-yang-type: string + - format: int32 + maximum: 2147483647 + minimum: 1 + type: integer + x-range: 1..2147483647 + x-yang-type: int32 +``` + +##### 3.2.1.1.5 Special handling + +* A list and leaf-list will have maxItems and minItems properties set if a YANG node defines max-elements and min-elements statements. + +Example below +``` + security-model: + type: array + items: + oneOf: + - enum: + - v1 + - v2c + - usm + - tsm + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-yang-type: string + - format: int32 + maximum: 2147483647 + minimum: -2147483648 + type: integer + x-range: 1..2147483647 + x-yang-type: int32 + minItems: 1 +``` +* A Choice-case statement is supported using oneOf directive. + +Example below +``` + ietf-snmp:auth: + type: object + properties: {} + **oneOf:** + - type: object + properties: + md5: + type: object + properties: + key: + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-pattern: ([0-9a-fA-F]{2}(:[0-9a-fA-F]{2})*)? + x-yang-type: string + required: + - key + - type: object + properties: + sha: + type: object + properties: + key: + maxLength: 18446744073709551615 + minLength: 0 + type: string + x-pattern: ([0-9a-fA-F]{2}(:[0-9a-fA-F]{2})*)? + x-yang-type: string + required: + - key +``` + +##### 3.2.1.1.6 Future enhancements + +* Support for additional YANG actions and notifications(if required). +* Support for RESTCONF query parameters such as depth, filter, etc +* Convert Yang identities in to enums + +##### 3.2.1.2 OpenAPI generator + +OpenAPI-codegen tool (github.com/OpenAPITools/openapi-generator) is used to generate REST server and client code from the OpenAPI definitions. It consumes the OpenAPI definitions generated from YANG files and any other manually written OpenAPI definition files. + +REST server is generated in Go language. Customized OpenAPI-codegen templates are used to make each server stub invoke a common request handler function. The common request handler will invoke Translib APIs to service the request. + +REST client is generated in Python language. Client applications can generate the REST client in any language using standard OpenAPI-codegen tool. + +##### 3.2.1.3 YGOT generator + +YGOT generator generates Go binding structures for the management YANG. The generated Go binding structures are consumed by the Translib to validate the incoming payload and help in conversion of Redis data to management YANG specific JSON output payload. + +##### 3.2.1.4 pyang compiler + +Open source pyang tool is used to compile YANG models and generate output file in required format. For example SONiC YANG model is compiled and YIN schema file is generated for validation purpose. Similarly, after compiling YANG model, OpenAPI spec file is generated for generating REST client SDK and server code using OpenAPI-codegen. + +#### 3.2.2 Run time components + +Following are the run time components in the management framework + +1. CLI +2. REST Client SDK +3. REST server +4. gNMI Client +5. gNMI server +6. Translib +7. Config Validation Library (CVL) +8. Redis DB +9. Non DB data provider + +##### 3.2.2.1 CLI + +Open source Klish is integrated into sonic-mgmt-framework container to provide the command line interface tool to perform more efficient network operations more efficiently in SONiC. Klish will provide the core functionality of command parsing, syntax validation, command help and command auto-completion. The following diagram shows how the CLI commands are built, processed, and executed. + +![CLI components Interaction Diagram](images/cli_interactions.jpg) + +1. CLI command input from user +2. Klish invokes the actioner defined in command tag +3. Actioner invokes either sub shell or the built-in methods to make a REST client API call using C based libcurl or Python Requests package to the REST server. +4. Receive response from OpenAPI clientAPI and pass it to renderer scripts. +5. Renderer scripts processes the JSON response from REST Client and optionally formats the output using a Jinja template +6. CLI output is rendered to the terminal. + +###### 3.2.2.1.1 CLI components + +CLI consists of the following components. + +* **Open source Klish** - CLI parser framework to support Command Line Interface Shell +* **XML files** to define CLI command line options and actions + * Klish uses XML to define CLI commands to build the command tree. Klish provides modular CLI tree configurations to build command trees from multiple XML files. XML elements can be defined with macros and entity references, which are then preprocessed by utility scripts to generate the expanded XML files that are ultimately used by Klish. +* **Actioner** - Python scripts or built-in methods defined as a command `action`, to form the request body and invoke the OpenAPI client API +* **Renderer** - Python scripts defined with Jinja templates. Receives the JSON response from Swagger API and use the jinja2 template file to render and format CLI output. +* **Preprocess scripts** - Validates XML files and applies some processing from a developer-friendly form into a "raw" form that is compatible with Klish. + +###### 3.2.2.1.2 Preprocessing XML files + +Multiple scripts are executed at build time to preprocess special XML tags/attributes - MACRO substitution, adding pipe processing, etc. - and generate target XML files in a form that is consumable by Klish. + +The XML files are also validated as part of compilation. `xmllint` is used to validate all the processed XML files after macro substitution and pipe processing against the detailed schema defined in `sonic-clish.xsd`. Once the XML files are fully validated and preprocessed, the target XML files are generated in the folder `${workspace}/sonic-mgmt-framework/build/cli/target/command-tree`. + +The following preprocessing scripts are introduced: +* `klish_ins_def_cmd.py` - append the "exit" and "end" commands to the views of the Klish XML files +* `klish_insert_pipe.py` - extend every show and get COMMAND with pipe option +* `klish_platform_features_process.sh` - validate all platform XML files. Generate the entity.XML files. +* `klish_replace_macro.py` – perform macro substitution on the Klish XML files + +###### 3.2.2.1.3 MACROs + +There are some CLI commands that can have the same set of options, where the set of XML tags would need to be repeated in all those CLI commands. Macros can be used to avoid such repetitions and to keep the options in one place, so that it is possible to make a reference to a macro in multiple command definitions. There are cases where we may need to use variations in the values of these macro options. In such cases, it is also possible to pass those XML attributes as an argument to macros and substitute those values inside the macro definition. The macro definition is referred as the ```` and ```` tags. `klish_replace_macro.py` is used to process macros at the compile time to expand the references to the target XML files. +The macro definition files are located in the folder `${workspace}/sonic-mgmt-framework/src/CLI/clitree/macro`. + +Example: + +Before macro substitution: + +```XML + + + + + ... + +``` + +After macro substitution: + +```XML + + + + + + + + + ... + +``` + +###### 3.2.2.1.4 ENTITY +XML files can include an ENTITY that refers to a predefined value. Entity is typically used to define platform specific values and processed by `klish_platform_features_process.sh` to prepend the ENTITY values to the target XML files. By default, there is a default file called `platform_dummy.XML` that defines a platform default ENTITY list. Note that the platform specific is not supported yet. + +Example: `platform_dummy.XML` + +```XML + + START_PORT_ID + MAX_PORT_IDma + START_SUB_PORT_ID + MAX_SUB_PORT_ID + MAX_MTU + +``` + +The ENTITY name can be referenced in command definitions. For example, PTYPE for RANGE_MTU, used for interface commands: + +```XML + +``` + +###### 3.2.2.1.5 Actioner + +The Actioner is used to convert CLI commands to the corresponding OpenAPI client requests on the target resources. It is defined with the `` tag in the XML file. There are three different methods available in the Actioner: sub-shell, builtin clish_restcl and clish_pyobj. CLI builtin functions "clish_pyobj" and "clish_restcl" can be applied to reduce time take to execute a command by eliminating the sub-shell interpreter overhead. + +1) Spawn a sub-shell to interpret Python scripts +Klish spawns a sub-shell to interpret the Python scripts defined in a command's `` tag. +The sub-shell runs the wrapper script `sonic_cli_.py` +``` + sonic_cli_.py [parameters . . .] +``` +The `sonic_cli_.py` has a dispatch function to call a OpenAPI client method with parameters passed from user input. + +Example: +```XML + + + + + + if test "${direction-switch}" = "in"; then + Python $SONIC_CLI_ROOT/target/sonic-cli.py post_list_base_interfaces_interface ${access-list-name} ACL_IPV4 ${iface} ingress + else + Python $SONIC_CLI_ROOT/target/sonic-cli.py post_list_base_interfaces_interface ${access-list-name} ACL_IPV4 ${iface} egress + fi + + + ... +``` + +2) Invoke the built-in function, clish_restcl, to use libcurl to make REST client call +This builtin uses libcurl to connect to the REST server +Format of ACTION tag argument: +oper= url= body={..} +oper can be PUT/POST/PATCH/DELETE +body is optional. +oper GET is not supported as we currently don't handle rendering using jinja templates. +Example: +``` +oper=PATCH url=/restconf/data/openconfig-interfaces:interfaces/interface=Vlan${vlan-id}/config body={"openconfig-interfaces:config": {"name": "Vlan${vlan-id}"}} +``` + +3) Invoke the built-in function, clish_pyobj, to use embedding Python to make REST client call +This builtin uses embedded python approach. + +Format of ACTION tag argument: + +( is the actioner script without the extension. The name cannot include "-") + +Example: +``` +sonic_cli_if patch_openconfig_interfaces_interfaces_interface_config Vlan${vlan-id} +``` + +###### 3.2.2.1.6 Renderer scripts + +The actioner script receives the JSON output from the OpenAPI client API and invokes the renderer script. The renderer script will send the JSON response to the jinja2 template file to parse the response and generate the CLI output. + +Example: "show acl" + +``` +{% set acl_sets = acl_out['openconfig_aclacl']['acl_sets']['acl_set'] %} + {% for acl_set in acl_sets %} + Name: {{ acl_set['state']['description'] }} + {% endfor %} +``` + +###### 3.2.2.1.7 Workflow (to add a new CLI) + +The following steps are to be followed when a new CLI is to be added. +1. Create an XML file that defines CLI command and parameters that the command requires. +2. Define the CLI help string to be displayed and datatype for the parameters. New parameter types (PTYPES), macros, and entities can be defined and used in the XML files. Valid XML tags are defined in the `sonic-clish.xsd` file. +3. Add the actioner to `` tag to run the wrapper script with the OpenAPI client method name and parameters +4. Add the code to the wrapper script to construct the payload in `generate_body()` and handle the response +5. For ‘show’ commands, create a Jinja template to format the output + + +##### 3.2.2.2 REST Client SDK + +Framework provides OpenAPI-codegen generated Python client SDK. Developers can generate client SDK code in other programming languages from the OpenAPI definitions as required. + +Client applications can use OpenAPI generated client SDK or any other REST client tool to communicate with the REST server. + +##### 3.2.2.3 gNMI Client + +SONiC Telemetry service provides the gNMI server, while the client must be provided by the user. gNMI is typically used as a programmatic interface and therefore it is typically called directly from programming language environments using their respective gRPC libraries (https://github.com/grpc/grpc). For testing and scripting purposes, several CLI programs are provided as well. + +GNMI clients developed by google and modified by JipanYANG.(github.com/jipanYANG/gnxi/gnmi_get, github.com/jipanYANG/gnxi/gnmi_set) +was taken and further modified to support new features. gnmi_cli from openconfig (https://github.com/openconfig/gnmi/tree/master/cmd/gnmi_cli) is also taken for testing subscribe and capabilities operations. Finally, a new client was developed gnoi_client, for test gNOI RPC operations. + +Note: Although the gRPC protocol allows for many encodings to be used, our usage is restricted to JSON_IETF encoding. + +Supported RPC Operations: +------------------------- +- Get: Get one or more paths and have value(s) returned in a GetResponse. +- Set: Update, replace or delete objects + + Update: List of one or more objects to update + + Replace: List of one or objects to replace existing objects, any unspecified fields wil be defaulted. + + Delete: List of one or more object paths to delete +- Capabilities: Return gNMI version and list of supported models and model versions. + + A gNMI Extension field has been added as well to return the SONiC model "bundle version". +- Subscribe: + + Subscribe to paths using either streaming or poll, or once based subscription, with either full current state or updated values only. + * Once: Get single subscription message. + * Poll: Get one subscription message for each poll request from the client. + * Stream: Get one subscription message for each object update (called ON_CHANGE mode), or at each sample interval if using sample mode. target_defined uses the values pre-configured for that particular object. Not all paths support ON_CHANGE mode due to performance considerations, while all paths will support sample mode and therefore target defined as well since it will default to sample if ON_CHANGE is not supported. + + +Example Client Operations: +-------------------------- +Using opensource clients, these are example client operations. The .json test payload files are available here: https://github.com/project-arlo/sonic-mgmt-common/tree/master/src/Translib/test + +Get: +---- +`./gnmi_get -xpath /openconfig-acl:acl/interfaces -target_addr 127.0.0.1:8080 -alsologtostderr -insecure true -pretty` + +Set: +---- +Replace: +-------- + `./gnmi_set -replace /openconfig-acl:acl/:@./test/01_create_MyACL1_MyACL2.json -target_addr 127.0.0.1:8080 -alsologtostderr -insecure true -pretty` +Delete: +------- + `./gnmi_set -delete /openconfig-acl:acl/ -target_addr 127.0.0.1:8080 -insecure` + +Subscribe: +---------- +Streaming sample based: +----------------------- +`./gnmi_cli -insecure -logtostderr -address 127.0.0.1:8080 -query_type s -streaming_sample_interval 3 -streaming_type SAMPLE -q /openconfig-acl:acl/ -v 0 -target YANG` + +Poll based: +----------- +`./gnmi_cli -insecure -logtostderr -address 127.0.0.1:8080 -query_type p -polling_interval 1s -count 5 -q /openconfig-acl:acl/ -v 0 -target YANG` + +Once based: +----------- +`./gnmi_cli -insecure -logtostderr -address 127.0.0.1:8080 -query_type o -q /openconfig-acl:acl/ -v 0 -target YANG` + + +gNOI: +-------------- +gNOI (gRPC Network Operations Interface) extends the gNMI server, adding new custom RPC's to execute management functions on the switch. + +gNOI Clear Neighbor: +------------------- +``` +gnoi_client -module Sonic -rpc clearNeighbors -jsonin '{"sonic-neighbor:input": {"force": true, "ip": "4.4.4.1"}}' -insecure +``` + +##### 3.2.2.4 REST Server + +The management REST server is a HTTP server implemented in Go language. +It supports following operations: + +* RESTCONF APIs for YANG data +* REST APIs for manual OpenAPI definitions + +###### 3.2.2.4.1 Transport options + +REST server supports only HTTPS transport and listens on default port 443. +Server port can be changed through an entry in ConfigDB REST_SERVER table. + +By default a temporary self signed certificate is used as TLS server certificate. +It can be overridden by specifiying a valid TLS private key and certificate file +paths through the REST_SERVER table. + +REST_SERVER table schema is described in [DB Schema](#322414-db-schema) section. + +###### 3.2.2.4.2 Translib linking + +REST server will statically link with Translib. For each REST request, the server +invokes Translib API which then invokes appropriate App module to process the request. +Below is the mapping of HTTP operations to Translib APIs: + + HTTP Method | Translib API | Request data | Response data +-------------|------------------|---------------|--------------- + GET | translib.Get | path | status, payload + POST | translib.Create | path, payload | status + POST (for YANG RPC) | translib.Action | path, payload | status, payload + PATCH | translib.Update | path, payload | status + PUT | translib.Replace | path, payload | status + DELETE | translib.Delete | path | status + HEAD | translib.Get | path | status, (payload ignored) + OPTIONS | - | - | - + +More details about Translib APIs are in section [3.2.2.6](#3_2_2_6-Translib). + +REST OPTIONS requests do not invoke any Translib API. They return list of supported HTTP methods +for requested path in the "Allow" response header. If PATCH method was supported, the response +will also include an "Accept-Patch" header with the value "application/yang-data+json". + +###### 3.2.2.4.3 Media Types + +YANG defined RESTCONF APIs support **application/yang-data+json** media type. +Request and response payloads follow [RFC7951](https://tools.ietf.org/html/rfc7951) +defined encoding rules. Media type **application/yang-data+xml** is not supported +in first release. + +OpenAPI defined REST APIs can use any media type depending on App module +implementation. However content type negotiation is not supported by REST server. +A REST API should not be designed to consume or produce multiuple content types. +OpenAPI definition for each REST API should have maximun one cone media type in +its "consumes" and "produces" statements. + +###### 3.2.2.4.4 Payload Validations + +REST server does not validate request payload for YANG defined RESTCONF APIs. +Payload will be validated automatically in lower layers when it gets loaded +into YGOT bindings. + +For OpenAPI defined REST APIs the REST server will provide limited payload +validation. JSON request payloads (content type **application/json**) will be +validated against the schema defined in OpenAPI. Response data and non-JSON +request data are not validated by the REST server - this is left to the App module. + +###### 3.2.2.4.5 Concurrency + +REST server will accept concurrent requests. Translib provides appropriate locking mechanism - parallel reads and sequential writes. + +###### 3.2.2.4.6 API Versioning + +REST server will allow clients to specify API version through a custom HTTP header **Accept-Version**. + + Accept-Version: 1.0.3 + +Version text should be in **MAJOR.MINOR.PATCH** format. REST server will extract version text from +the request header and pass it to the Translib API as **ClientVersion** argument. +Section [3.2.2.6.4.2](#322642-version-checks) explains Translib version checking logic. + +Version checks are bypassed if Accept-Version header is not present in the request. + +For YANG defined RESTCONF APIs, the server's version can be discovered through the standard YANG +module library API "GET /restconf/data/ietf-yang-library:modules-state/module-set-id". +Client should not pass Accept-Version header for this API! This is safe since module-set-id +is a standard API defined by [RFC7895](https://tools.ietf.org/html/rfc7895) for this very purpose. + +API Versioning is not supported for manual OpenAPI definitions in this release. +Accept-Version header value will be ignored even if specified. + +###### 3.2.2.4.7 RESTCONF Entity-tag + +REST server does not maintain entity-tag and last-modified timestamp for configuration data nodes. +GET and HEAD responses does not include "ETag" and "Last-Modified" headers. + +[RFC7232](https://tools.ietf.org/html/rfc7232) style HTTP conditional requests is also not supported. +REST server ignores "If-Match", "If-Modified-Since" like conditional request headers. + +###### 3.2.2.4.8 RESTCONF Discovery + +REST server supports following APIs for clients to discover various RESTCONF protocol features +as described in [RFC8040](https://tools.ietf.org/html/rfc8040). + +Method | Path | Purpose +-------|------------------------------------------------|--------------------------- +GET | /.well-known/host-meta | RESTCONF root path +GET | /restconf/yang-library-version | YANG library version +GET | /restconf/data/ietf-yang-library:modules-state | RFC7895 YANG module library +GET | /restconf/data/ietf-restconf-monitoring:restconf-state/capabilities | RESTCONF Capabilities +GET | /models/yang/{filename} | YANG download + +###### 3.2.2.4.8.1 RESTCONF root + +Server supports RESTCONF root path discovery through "GET /.well-known/host-meta" API +as described in [RFC8040, section 3.1](https://tools.ietf.org/html/rfc8040#page-18). +RESTCONF root path is "/restconf". + +###### 3.2.2.4.8.2 Yang library version + +REST server supports "GET /restconf/yang-library-version" API to advertise YANG library version. +Response will indicate version "2016-06-21" as described in [RFC8040, section 3.3.3](https://tools.ietf.org/html/rfc8040#section-3.3.3). +This advertises that the server supports [RFC7895](https://tools.ietf.org/html/rfc7895) compliant +YANG library operations. + +###### 3.2.2.4.8.3 YANG module library + +REST server allows clients to discover and download all YANG modules supported by the server +via "GET /restconf/data/ietf-yang-library:modules-state" API. Response data includes YANG module +information as per [RFC7895](https://tools.ietf.org/html/rfc7895) requirements. + +REST server allows clients to download the YANG files via "GET /models/yang/{filename}" API. +YANG module library response includes full download URL for every YANG module entry. + +Note - RFC7895 has been recently obsoleted by [RFC8525](https://tools.ietf.org/html/rfc8525). +It supports YANG library with multi data stores and data store schemas. However these features +are not available in SONiC. Hence REST server does not implement RFC8525 YANG library APIs (for now). + +###### 3.2.2.4.8.4 RESTCONF capabilities + +REST server supports "GET /restconf/data/ietf-restconf-monitoring:restconf-state/capabilities" API to +advertise its capabilities as described in [RFC8040, section 9.1](https://tools.ietf.org/html/rfc8040#section-9.1). +REsponse includes below mentioned capability information. + + urn:ietf:params:restconf:capability:defaults:1.0?basic-mode=report-all + +###### 3.2.2.4.9 RESTCONF Query Parameters + +RESTCONF Query Parameters will be supported in future release. All query parameters will be ignored by REST server in this release. + +###### 3.2.2.4.10 RESTCONF Operations + +REST server supports invoking YANG RPCs through "POST /restconf/operations/{rpc_name}" API +as described in [RFC8040, section 4.4.2](https://tools.ietf.org/html/rfc8040#section-4.4.2). +Operations modeled through YANG v1.1 action statement are not supported in this release. + +YGOT binding objects are not available for YANG RPC input and output data model. +Hence payload validation is not performed by REST server or Translib for these APIs. +App modules will receive raw JSON data. + +###### 3.2.2.4.11 RESTCONF Notifications + +RESTCONF Notification are not supported by framework. Clients can use gNMI for monitoring and notifications. + +###### 3.2.2.4.12 Authentication + +REST server supporsts following authentication modes. + +* HTTP Basic authentication (username/password authentication) +* HTTP Bearer token authentication with JSON Web Token (JWT) format. +* TLS Certificate authentication +* Any combination of above 3 modes +* No authentication + +Details are in [SONiC RBAC HLD](https://github.com/project-arlo/SONiC/blob/master/doc/aaa/SONiC%20RBAC%20HLD.md). + +By default HTTP Baisc and bearer token authentication modes are enabled. +It can be overridden through ConfigDB [REST_SERVER table](#322414-db-schema) entry. + +###### 3.2.2.4.13 Error Response + +REST server sends back HTTP client error (4xx) or server error (5xx) status when request processing +fails. Response status and payload will be as per RESTCONF specifications - [RCF8040, section7](https://tools.ietf.org/html/rfc8040#page-73). +Error response data will be a JSON with below structure. Response Content-Type will be +"application/yang-data+json". + + +---- errors + +---- error* + +---- error-type "protocol" or "application" + +---- error-tag string + +---- error-app-tag? string + +---- error-path? xpath + +---- error-message? string + +Note: REST server will not populate error-app-tag and error-path fields in this release. It can be +enhanced in a future release. A sample error response: + + { + "ietf-restconf:errors" : { + "error" : [ + { + "error-type" : "application", + "error-tag" : "invalid-value", + "error-message" : "VLAN 100 not found" + } + ] + } + } + +**error-type** can be either "protocol" or "application", indicating the origin of the error. +RESTCONF defines two more error-type enums "transport" and "rpc"; they are not used by REST server. + +**error-tag** indicates nature of error as described in [RFC8040, section 7](https://tools.ietf.org/html/rfc8040#page-74). + +**error-message** field carries a human friendly error message that can be displayed to the end +user. This is an optional field; system error do not include error-message, or have generic +messages like "Internal error". App module developer should use human friendly messages while +returning application errors. In case of CVL constraint violation the REST server will pick +the error message from the YANG "error-message" statement of CVL schema YANG. + +Table below lists possible error conditions with response status and data returned by REST server. + +Method | Error condition | Status | error-type | error-tag | error-message +--------|--------------------------|--------|-------------|------------------|---------------------- +*any* | Incorrect request data | 400 | protocol | invalid-value | +*write* | Bad content-type | 415 | protocol | invalid-value | Unsupported content-type +*write* | OpenAPI schema validation fails | 400 | protocol| invalid-value | Content not as per schema +*write* | YGOT schema validation fails | 400 | protocol| invalid-value | *YGOT returned message* +*any* | Invalid user credentials | 401 | protocol | access-denied | Authentication failed +*write* | User is not an admin | 403 | protocol | access-denied | Authorization failed +*write* | Translib commit failure | 409 | protocol | in-use | +*any* | Bad Accept-Version value | 400 | protocol | invalid-value | Invalid Accept-Version +*any* | Version ckeck fails | 400 | protocol | operation-not-supported | Unsupported client version *X.Y.Z* +*any* | Unknown HTTP server failure | 500 | protocol | operation-failed | Internal error +*any* | Not supported by App module | 405 | application | operation-not-supported | *App module returned message* +*any* | Incorrect payload | 400 | application | invalid-value | *App module returned message* +*any* | Resource not found | 404 | application | invalid-value | *App module returned message* +POST | Resource exists | 409 | application | resource-denied | *App module returned message* +*any* | Unknown error in Translib | 500 | application | operation-failed | Internal error +*any* | Unknown App module failure | 500 | application | operation-failed | *App module returned message* +*any* | CVL constraint failure | 500 | application | invalid-value | *error-message defined in SONiC YANG* + + +###### 3.2.2.4.14 DB Schema + +A new table "REST_SERVER" is introduced in ConfigDB for maintaining REST server configurations. + + key = REST_SERVER:default ; REST server configurations. + ;field = value + port = 1*5DIGIT ; server port - defaults to 443 + client_auth = "none"/"password"/"jwt"/"cert" + ; Client authentication mode. + ; none: No authentication, all clients + ; are allowed. Should be used only + ; for debugging. + ; password: HTTP Basic authentication. + ; jwt : HTTP Bearer Token authentication with + ; JSON Web Token format. + ; cert: Certificate based authentication. + ; Requires REST_SERVER['certs']['ca_crt'] configuration. + ; Any combination of "password", "jwt" and "cert" modes can be + ; enabled by specifying a comma separated values. + ; Eg: "password,jwt" enables both password and jwt modes. + log_level = DIGIT ; Verbosity for glog.V logs + + + key = REST_SERVER:certs ; Server certificate configurations + ;field = value + server_crt = STRING ; Path to TLS certificate file + server_key = STRING ; Path to TLS private key file + ca_crt = STRING ; Path to the CA certificate to be used for + ; client certificate validation. + +###### 3.2.2.4.15 API Documentation + +REST server will provide [OpenAPI UI](https://github.com/OpenAPI-api/OpenAPI-ui) based online +documentation and test UI for all REST APIs it supports. Documentation can be accessed by launching +URL **https://REST_SERVER_IP/ui** in a browser. This page will list all supported OpenAPI +definition files (both YANG generated and manual) along with link to open OpenAPI UI for them. + + +##### 3.2.2.5 gNMI server + +1. gNMI server is part of the telemetry process that supports telemtry as well as gNMI. +2. The gRPC server opens a TCP port and allows only valid mutually authenticated TLS connections, which requires valid Client, server and CA Certificates be installed as well a properly configured DNS. Multiple simultaneous connections are allowed to gNMI server. +3. The gNMI Agent uses the db client, as well as the non-db client to access and modify data directly in the Redis DB. +4. The Translib client is used to provide alternative models of access such as Openconfig models as opposed to the native Redis schema, as long as the Translib supports these models. Translib offers bidirectional translation between the native Redis model and the desired north bound model, as well as notifications/updates on these model objects to support telemetry and asynchronous updates, alarms and events. Translib should also provide information about what models it supports so that information can be returned in gNMI Capabilities response. +5. The gNMI server defines the four RPC functions as required by the gNMI Specification: Get, Set, Capabilities and Subscribe. +6. Since the db, non-db and Translib clients offer the functionality to support these functions, gNMI only has to translate the paths and object payloads into the correct parameters for the client calls and package the results back into the response gNMI objects to return to the gNMI Client, which is a straightforward operation, since no additional processing of the data is expected to be done in the gNMI server itself. When new models are added to Translib, no additional work should be required to support them in gNMI server. +7. All operations in a Set request are processed in a single transaction that will either succeed or fail as one operation. The db, non-db and Translib clients must support a Bulk operation in order to achieve the transactional behavior. gNMI server then must use this Bulk operation for Set requests. +8. Subscribe operations: Once, Poll and Stream require that the gRPC connection remain open until the subscription is completed. This means many connections must be supported. Subscribe offers several options, such as only sending object updates (not the whole object) which requires support form the db clients. Subscribe also allows for periodic sampling defined by the client. This must be handled in the gNMI agent itself. This requires a timer for each subscribe connection of this type in order to periodically poll the db client and return the result in a Subscribe Response. These timers should be destroyed when the subscription gRPC connection is closed. + +###### 3.2.2.5.1 Files changed/added: + + |-- gnmi_server + | |-- client_subscribe.go + | |-- server.go ------------------- MODIFIED (Handles creation of transl_data_client for GET/SET/CAPABILITY) + | |-- server_test.go + |-- sonic_data_client + | |-- db_client.go ---------------- MODIFIED (Common interface Stub code for new functions as all data clients implement common interface functions) + | |-- non_db_client.go ------------ MODIFIED (Common interface Stub code for new functions as all data clients implement common interface functions) + | |-- transl_data_client.go ------- ADDED (Specific processing for GET/SET/CAPABILITY for transl data clients) + | |-- trie.go + | |-- virtual_db.go + | + |-- transl_utils -------------------- ADDED + |-- transl_utils.go ------------- ADDED (Layer for invoking Translib APIs) + +###### 3.2.2.5.2 Sample Requests + +go run gnmi_get.go -xpath /openconfig-acl:acl/acl-sets/acl-set[name=MyACL4][type=ACL_IPV4]/acl-entries/acl-entry[sequence-id=1] -target_addr 10.130.84.34:8081 -alsologtostderr -insecure true -pretty + +go run gnmi_set.go -replace /openconfig-acl:acl/acl-sets/acl-set[name=MyACL4][type=ACL_IPV4]/acl-entries/acl-entry=2/actions/config:@openconfig.JSON -target_addr 10.130.84.34:8081 -alsologtostderr -insecure true -pretty + +go run gnmi_capabilities.go -target_addr 10.130.84.34:8081 -alsologtostderr -insecure true -pretty + +##### 3.2.2.6 Translib + +Translib is a library that adapts management server requests to SONiC data providers and vice versa. Translib exposes the following APIs for the management servers to consume. + + func Create(req SetRequest) (SetResponse, error) + func Update(req SetRequest) (SetResponse, error) + func Replace(req SetRequest) (SetResponse, error) + func Delete(req SetRequest) (SetResponse, error) + func Get(req GetRequest) (GetResponse, error) + func Action(req ActionRequest) (ActionResponse, error) + func Bulk(req BulkRequest) (BulkResponse, error) + func Subscribe(req SubscribeRequest) ([]*IsSubscribeResponse, error) + func IsSubscribeSupported(req IsSubscribeRequest) ([]*IsSubscribeResponse, error) + func GetModels() ([]ModelData, error) + + Translib Structures: + type ErrSource int + + const ( + ProtoErr ErrSource = iota + AppErr + ) + + type SetRequest struct { + Path string + Payload []byte + User string + ClientVersion Version + } + + type SetResponse struct { + ErrSrc ErrSource + Err error + } + + type GetRequest struct { + Path string + User string + ClientVersion Version + } + + type GetResponse struct { + Payload []byte + ErrSrc ErrSource + } + + type ActionRequest struct { + Path string + Payload []byte + User string + ClientVersion Version + } + + type ActionResponse struct { + Payload []byte + ErrSrc ErrSource + } + + type BulkRequest struct { + DeleteRequest []SetRequest + ReplaceRequest []SetRequest + UpdateRequest []SetRequest + CreateRequest []SetRequest + User string + ClientVersion Version + } + + type BulkResponse struct { + DeleteResponse []SetResponse + ReplaceResponse []SetResponse + UpdateResponse []SetResponse + CreateResponse []SetResponse + } + + type SubscribeRequest struct { + Paths []string + Q *queue.PriorityQueue + Stop chan struct{} + User string + ClientVersion Version + } + + type SubscribeResponse struct { + Path string + Payload []byte + Timestamp int64 + SyncComplete bool + IsTerminated bool + } + + type NotificationType int + + const ( + Sample NotificationType = iota + OnChange + ) + + type IsSubscribeRequest struct { + Paths []string + User string + } + + type IsSubscribeResponse struct { + Path string + IsOnChangeSupported bool + MinInterval int + Err error + PreferredType NotificationType + } + + type ModelData struct { + Name string + Org string + Ver string + } + + type Version struct { + Major uint32 + Minor uint32 + Patch uint32 + } + +Translib has the following sub modules to help in the translation of data + +1. App Interface +2. Translib Request Handlers +3. YGOT request binder +4. DB access layer +5. App Modules +6. Transformer + +###### 3.2.2.6.1 App Interface + +App Interface helps in identifying the App module responsible for servicing the incoming request. It provides the following APIs for the App modules to register themselves with the App interface during the initialization of the app modules. + + func Register(path string, appInfo *AppInfo) error + This method can be used by any App module to register itself with the Translib infra. + Input Parameters: + path - base path of the model that this App module services + appInfo - This contains the reflect types of the App module structure that needs to be instantiated for each request, corresponding YGOT structure reflect type to instantiate the corresponding YGOT structure and boolean indicating if this is native App module to differentiate between OpenAPI spec servicing App module and the YANG serving App module. + Returns: + error - error string + func AddModel(model *gnmi.ModelData) error + This method can be used to register the models that the App module supports with the Translib infra. + Input Parameters: + model - Filled ModelData structure containing the Name, Organisation and version of the model that is being supported. + Returns: + error - error string + + App Interface Structures: + //Structure containing App module information + type AppInfo struct { + AppType reflect.Type + YGOTRootType reflect.Type + IsNative bool + tablesToWatch []*db.TableSpec + } + + Example Usages: + func init () { + log.Info("Init called for ACL module") + err := register("/openconfig-acl:acl", + &appInfo{appType: reflect.TypeOf(AclApp{}), + ygotRootType: reflect.TypeOf(ocbinds.OpenconfigAcl_Acl{}), + isNative: false, + tablesToWatch: []*db.TableSpec{&db.TableSpec{Name: ACL_TABLE}, &db.TableSpec{Name: RULE_TABLE}}}) + + if err != nil { + log.Fatal("Register ACL App module with App Interface failed with error=", err) + } + + err = appinterface.AddModel(&gnmi.ModelData{Name:"openconfig-acl", + Organization:"OpenConfig working group", + Version:"1.0.2"}) + if err != nil { + log.Fatal("Adding model data to appinterface failed with error=", err) + } + } + + type AclApp struct { + path string + YGOTRoot *YGOT.GoStruct + YGOTTarget *interface{} + } + +Translib request handlers use the App interface to get all the App module information depending on the incoming path as part of the requests. + +###### 3.2.2.6.2 Translib Request Handler + +These are the handlers for the APIs exposed by the Translib. Whenever a request lands in the request handler, the handler uses the App interface to get the App module that can process the request based on the incoming path. It then uses the YGOT binder module, if needed, to convert the incoming path and payload from the request into YGOT structures. The filled YGOT structures are given to the App Modules for processing. The Translib also interacts with the DB access layer to start, commit and abort a transaction. + +###### 3.2.2.6.3 YGOT request binder + +The YGOT request binder module uses the YGOT tools to perform the unmarshalling and validation. YGOT (YANG Go Tools) is an open source tool and it has collection of Go utilities which are used to + + 1. Generate a set of Go structures for bindings for the given YANG modules at build time + 2. Unmarshall the given request into the Go structure objects. These objects follows the same hierarchical structure defined in the YANG model, and it's simply a data instance tree of the given request, but represented using the generated Go structures + 3. Validate the contents of the Go structures against the YANG model (e.g., validating range and regular expression constraints). + 4. Render the Go structure objects to an output format - such as JSON. + +This RequestBinder module exposes the below mentioned APIs which will be used to unmarshall the request into Go structure objects, and validate the request + + func getRequestBinder(uri *string, payload *[]byte, opcode int, appRootNodeType *reflect.Type) *requestBinder + This method is used to create the requestBinder object which keeps the given request information such as uri, payload, App module root type, and unmarshall the same into object bindings + Input parameters: + uri - path of the target object in the request. + payload - payload content of given the request and the type is byte array + opcode - type of the operation (CREATE, DELETE, UPDATE, REPLACE) of the given request, and the type is enum + appRootNodeType - pointer to the reflect.Type object of the App module root node's YGOT structure object + Returns: + requestBinder - pointer to the requestBinder object instance + + func (binder *requestBinder) unMarshall() (*YGOT.GoStruct, *interface{}, error) + This method is be used to unmarshall the request into Go structure objects, and validates the request against YANG model schema + Returns: + YGOT.GoStruct - root Go structure object of type Device. + interface{} - pointer to the interface type of the Go structure object instance of the given target path + error - error object to describe the error if the unmarshalling fails, otherwise nil + +Utilities methods: +These utilities methods provides below mentioned common operations on the YGOT structure which are needed by the App module + + func getParentNode(targetUri *string, deviceObj *ocbinds.Device) (*interface{}, *YANG.Entry, error) + This method is used to get parent object of the given target object's uri path + Input parameters: + targetUri - path of the target URI + deviceObj - pointer to the base root object Device + Returns + interface{} - pointer to the parent object of the given target object's URI path + YANG.Entry - pointer to the YANG schema of the parent object + error - error object to describe the error if this methods fails to return the parent object, otherwise nil + + func getNodeName(targetUri *string, deviceObj *ocbinds.Device) (string, error) + This method is used to get the YANG node name of the given target object's uri path. + Input parameters: + targetUri - path of the target URI + deviceObj - pointer to the base root object Device + Returns: + string - YANG node name of the given target object + error - error object to describe the error if this methods fails to return the parent object, otherwise nil + + func getObjectFieldName(targetUri *string, deviceObj *ocbinds.Device, YGOTTarget *interface{}) (string, error) + This method is used to get the go structure object field name of the given target object. + Input parameters: + targetUri - path of the target URI + deviceObj - pointer to the base root object Device + YGOTTarget - pointer to the interface type of the target object. + Returns: + string - object field name of the given target object + error - error object to describe the error if this methods fails to perform the desired operation, otherwise nil + +###### 3.2.2.6.4 YANG Model Versioning + +###### 3.2.2.6.4.1 Version Management + +Translib maintains a "YANG bundle version" which is the collective version number for all the YANG +modules deployed in the Management Framework Service. This version is maintained in a configuration +file. It uses **MAJOR.MINOR.PATCH** syntax as per [Sematic Versioning](https://semver.org) specification. +Developer will have to update the bundle version number when he makes any YANG change. + +**Major version** is fixed to 1 in this release. +It will be incremented in future releases when any YANG model is changed in a non backward +compatible manner. Following are the candidates: + +* Delete, rename or relocate data node +* Change list key attributes +* Change data type of a node to an incompatible type +* Change leafref target + +**Minor version** is incremented if the YANG change modifies the API in a backward +compatible way. Patch version is reset to 0. +Candidate YANG changes for this category are: + +* Add new YANG module +* Add new YANG data nodes +* Mark a YANG data node as deprecated +* Change data type of a node to a compatible type +* Add new enum or identity + +**Patch version** is incremented for cosmetic fixes that do not change YANG API. +Candidate YANG changes for this category are: + +* Change description, beautification. +* Expand pattern or range of a node to wider set. +* Change must expression to accept more cases. +* Error message or error tag changes. + +###### 3.2.2.6.4.2 Version Checks + +Version check ensures that translib processes the requests only from compatible clients. +All translib APIs accept an optional **ClientVersion** argument. +Clients can pass the YANG bundle version that they are capable of handling through this argument. +If specified, request will be processed only if below criteria is satisfied. + + Base_Version <= ClientVersion <= YANG_Bundle_Version + +Base Version is **1.0.0** for current release. +It will change in a future release when major version is incremented. + +Version check is bypassed if client does not send its version number in the request. +However some requests may succeed and some may fail due to input or schema mismatch. + +###### 3.2.2.6.5 DB access layer + +The DB access layer implements a wrapper over the [go-redis](https://github.com/go-redis/redis) package +enhancing the functionality in the following ways: + + * Provide a sonic-py-swsssdk like API in Go + * Enable support for concurrent access via Redis CAS (Check-And-Set) + transactions. + * Invoke the CVL for validation before write operations to the Redis DB + +The APIs are broadly classified into the following areas: + + * Initialization/Close: NewDB(), DeleteDB() + * Read : GetEntry(), GetKeys(), GetKeysPattern(), GetTable() + * Write : SetEntry(), CreateEntry(), ModEntry(), DeleteEntry() + DeleteEntryField() + * Transactions : StartTx(), CommitTx(), AbortTx(), AppendWatchTx() + * Map : GetMap(), GetMapAll() + * Subscriptions : SubscribeDB(), UnsubscribeDB() + * Publish : Publish() + +Detail Method Signature: + Please refer to the code for the detailed method signatures. + +Concurrent Access via Redis CAS transactions: + + Upto 4 levels of concurrent write access support. + + 1. Table based watch keys (Recommended): + At App module registration, the set of Tables that are to be managed by + the module are provided. External (i.e. non-Management-Framework) + applications may choose to watch and set these same table keys to detect + and prevent concurrent write access. The format of the table key is + "CONFIG_DB_UPDATED_". (Eg: CONFIG_DB_UPDATED_ACL_TABLE) + + 2. Row based watch keys: + For every transaction, the App module provides a list of keys that it + would need exclusive access to for the transaction to succeed. Hence, + this is more complex for the app modules to implement. The external + applications need not be aware of table based keys. However, the + concurrent modification of yet to be created keys (i.e. keys which are + not in the DB, but might be created by a concurrent application) may not + be detected. + + 3. A combination of 1. and 2.: + More complex, but easier concurrent write access detection. + + 4. None: + For applications not needing concurrent write access protections. + + +DB access layer, Redis, CVL Interaction: + + DB access | PySWSSSDK API | RedisDB Call at | CVL Call at + | | at CommitTx | invocation + ----------------|------------------|-------------------|-------------------- + SetEntry(k,v) | set_entry(k,v) | HMSET(fields in v)|If HGETALL=no entry + | | HDEL(fields !in v | ValidateEditConfig + | | but in | (OP_CREATE) + | | previous HGETALL)| + | | |Else + | | | ValidateEditConfig( + | | | {OP_UPDATE, + | | | DEL_FIELDS}) + ----------------|------------------|-------------------|-------------------- + CreateEntry(k,v)| none | HMSET(fields in v)| ValidateEditConfig( + | | | OP_CREATE) + ----------------|------------------|-------------------|-------------------- + ModEntry(k,v) | mod_entry(k,v) | HMSET(fields in v)| ValidateEditConfig( + | | | OP_UPDATE) + ----------------|------------------|-------------------|-------------------- + DeleteEntry(k,v)|set,mod_entry(k,0)| DEL | ValidateEditConfig( + | | | OP_DELETE) + ----------------|------------------|-------------------|-------------------- + DeleteEntryField| none | HDEL(fields) | ValidateEditConfig( + (k,v) | | | DEL_FIELDS) + +Notes: + 1. SetEntry(), CreateEntry(), ModEntry() with empty fields in v, will map to + DeleteEntry(). [ There may be an option to disable this in the future. ] + + +##### 3.2.2.7 Transformer + +Transformer provides the underlying infrastructure for developers to translate data from YANG to ABNF/Redis schema and vice versa, using YANG extensions annotated to YANG paths to provide translation methods. At run time, the YANG extensions are mapped to an in-memory Transformer Spec that provides two-way mapping between YANG and ABNF/Redis schema for Transformer to perform data translation while processing SET/GET operations. + +In case that SONiC YANG modules are used by NBI applications, the Transformer performs 1:1 mapping between a YANG object and a SONiC DB object without a need to write special translation codes. If the openconfig YANGs are used by NBI applications, you may need special handling to translate data between YANG and ABNF schema. In such case, you can annotate YANG extensions and write callbacks to perform translations where required. + +For special handling, a developer needs to provide: +1. An annotation file to define YANG extensions on YANG paths where translation required +e.g. In openconfig-acl.yang, ACL FORWARDING_ACTION "ACCEPT" mapped to "FORWARD", ACL sequence id ‘1’ mapped to ‘RULE_1’ in Redis DB etc. +2. Transformer callbacks to perform translation + +###### 3.2.2.7.1 Components + +Transformer consists of the following components and data: +* **Transformer Spec:** a collection of translation hints +* **Spec Builder:** loads YANG and annotation files to dynamically build YANG schema tree and Transformer Spec. +* **Transformer Core:** perform main transformer tasks, i.e. encode/decode YGOT, traverse the payload, lookup Transformer spec, call Transformer methods, construct the results, error reporting etc. +* **Built-in Default Transformer method:** perform static translation +* **Overloaded Transformer methods:** callback functions invoked by Transformer core to perform complex translation with developer supplied translation logic +* **Ouput framer:** aggregate the translated pieces returned from default and overloaded methods to construct the output payload +* **Method overloader:** dynamically lookup and invoke the overloaded transformer methods during data translation +* **YANG schema tree:** provides the Transformer with the schema information that can be accessed by Transformer to get node information, like default values, parent/descendant nodes, etc. + +![Transformer Components](images/transformer_components_v1.png) + +###### 3.2.2.7.2 Design + +Requests from Northbound Interfaces (NBI) are processed by Translib public APIs - Create, Replace, Update, Delete, (CRUD) and Get - that call a specific method on the common app module. The common app calls Transformer APIs to translate the request, then use the translated data to proceed to DB/CVL layer to set or get data in Redis DB. The **common app** as a default application module generically handles both SET (CRUD) and GET, and Subscribe requests with Transformer. Note that a specific app module can be registered to the Translib to handle the requests if needed. + +![Transformer Design](images/transformer_design.PNG) + +At Transformer init, it loads YANG modules pertaining to the applications. Transformer parses YANG modules with the extensions to dynamically build an in-memory schema tree and transformer spec. + +Below structure is defined for the transformer spec: + +```YANG +type yangXpathInfo struct { + yangDataType string + tableName *string + xfmrTbl *string + childTable []string + dbEntry *yang.Entry + yangEntry *yang.Entry + keyXpath map[int]*[]string + delim string + fieldName string + xfmrFunc string + xfmrField string + xfmrPost string + validateFunc string + rpcFunc string + xfmrKey string + keyName *string + dbIndex db.DBNum + keyLevel int + isKey bool + defVal string + hasChildSubTree bool +} +``` + +When a request lands at the common app in the form of a YGOT structure from the Translib request handler, the request is passed to Transformer that decodes the YGOT structure to read the request payload and look up the spec to get translation hints. Note that the Transformer cannot be used for OpenAPI spec. The Transformer Spec is structured with a two-way mapping to allow Transformer to map YANG-based data to ABNF data and vice-versa via reverse lookup. The reverse mapping is used to populate the data read from Redis DB to YANG structure for the response to get operation. + +Transformer has a built-in default transformer method to perform static, simple translation from YANG to ABNF or vice versa. It performs simple mapping - e.g. a direct name/value mapping, table/key/field name - which can be customized by a YANG extension. + +Additionally, for more complex translations of non-ABNF YANG models, i.e. OpenConfig models, Transformer also allows developers to overload the default method by specifying a callback fucntion in YANG extensions, to perform translations with developer-supplied translation codes as callback functions. Transformer dynamically invokes those functions instead of using default method. Each transformer callback must be defined to support two-way translation, i.e, YangToDb_ and DbToYang_, which are invoked by Transformer core. + +###### 3.2.2.7.3 Process + +CRUD requests (configuration) are processed via the following steps: + +1. App module calls transformer, passing it a YGOT populated Go structure to translate YANG to ABNF +2. App module calls CVL API to get all depenent table list to watch tables, and get the ordered table list +3. Transformer allocates buffer with 3-dimensional map: `[table-name][key-values][attributes]` +4. Transformer decodes YGOT structure and traverses the incoming request to get the YANG node name +5. Transformer looks up the Transformer Spec to check if a translation hint exists for the given path +6. If no spec or hint is found, the name and value are copied as-is +7. If a hint is found, check the hint to perform the action, either simple data translation or invoke external callbacks +8. Repeat steps 4 through 7 until traversal is completed +9. Invoke any annotated post-Transformer functions +10. Transformer aggregates the results to returns to App module +11. App module proceeds to update DB to ensure DB update in the order learnt from step 2 + +GET requests are processed via the following steps: +1. App module asks the transformer to translate the URL to the keyspec to the query target + ```YANG + type KeySpec struct { + DbNum db.DBNum + Ts db.TableSpec + Key db.Key + Child []KeySpec + IgnoreParentKey bool + } + ``` +2. Transformer proceeds to traverse the DB with the keyspec to get the results +3. Transformer translate the results from ABNF to YANG, with default transformer method or callbacks +4. Transformer aggregate the translated results, return to the App module to unmarshall the JSON payload + +###### 3.2.2.7.4 Common App + +The Common App is a default app that handles the GET/SET/Subscribe requests for SONiC or OpenConfig YANG modules unless an app module is registered to Translib. + +Here is a diagram to show how the common app supports SET(CRUD)/GET requests. + +![sequence diagram_for_set](images/crud_v1.png) + +![sequence_diagram_for_get](images/get_v1.png) + +If a request is associated with multiple tables, the common app module processes the DB updates in the table order learned from CVL layer. +e.g. in case that the sonic-acl.yang used by NBI and the payload with CREATE operation has a data including both ACL_TABLE and ACL_RULE, the common app updates the CONFIG-DB to create an ACL TABLE instance, followed by ACL_RULE entry creation in this order. DELETE operates in the reverse order, i.e. ACL_RULE followed by ACL_TABLE. + +###### 3.2.2.7.5 YANG Extensions + +The translation hints are defined as YANG extensions to support simple table/field name mapping or more complex data translation by external callbacks. + +---------- + +1. `sonic-ext:table-name [string]`: +Map a YANG container/list to TABLE name, processed by the default transformer method. Argument is a table name statically mapped to the given YANG container or list node. +The table-name is inherited to all descendant nodes unless another one is defined. + +2. `sonic-ext:field-name [string]`: +Map a YANG leafy - leaf or leaf-list - node to FIELD name, processed by the default transformer method + +3. `sonic-ext:key-delimiter [string]`: +Override the default key delimiters used in Redis DB, processed by the default transformer method. +Default delimiters are used by Transformer unless the extension is defined - CONFIG_DB: "|", APPL_DB: ":", ASIC_DB: "|", COUNTERS_DB: ":", FLEX_COUNTER_DB: "|", STATE_DB: "|" + +4. `sonic-ext:key-name [string]`: +Fixed key name, used for YANG container mapped to TABLE with a fixed key, processed by the default transformer method. Used to define a fixed key, mainly for container mapped to TABLE key +e.g. Redis can have a hash “STP|GLOBAL” +```YANG +container global + sonic-ext:table-name “STP” + sonic-ext:key-name “GLOBAL” +``` +5. `sonic-ext:key-transformer [function]`: +Overloading default method with a callback to generate DB keys(s), used when the key values in a YANG list are different from ones in DB TABLE. +A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function* + +6. `sonic-ext:field-transformer [function]`: +Overloading default method with a callback to generate FIELD value, used when the leaf/leaf-list values defined in a YANG list are different from the field values in DB. +A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function* + +7. `sonic-ext:subtree-transformer [function]`: +Overloading default method with a callback for the current subtree, allows the sub-tree transformer to take full control of translation. Note that, if any other extensions, e.g. table-name etc., are annotated to the nodes on the subtree, they are not effective. +The subtree-transformer is inherited to all descendant nodes unless another one is defined, i.e. the scope of subtree-transformer callback is limited to the current and descendant nodes along the YANG path until a new subtree transformer is annotated. +A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function* + +8. `sonic-ext:db-name [string]`: +DB name to access data – “APPL_DB”, “ASIC_DB”, “COUNTERS_DB”, “CONFIG_DB”, “FLEX_COUNTER_DB”, “STATE_DB”. The default db-name is CONFIG_DB, Used for GET operation to non CONFIG_DB, applicable only to SONiC YANG. Processed by Transformer core to traverse database. +The db-name is inherited to all descendant nodes unless another one. Must be defined with the table-name + +9. `sonic-ext:post-transformer [function]`: +A special hook to update the DB requests right before passing to common-app, analogous to the postponed YangToDB subtree callback that is invoked at the very end by the Transformer. +Used to add/update additional data to the maps returned from Transformer before passing to common-app, e.g. add a default acl rule +Note that the post-transformer can be annotated only to the top-level container(s) within each module, and called once for the given node during translation + +10. `sonic-ext:table-transformer [function]`: +Dynamically map a YANG container/list to TABLE name(s), allows the table-transformer to map a YANG list/container to table names. +Used to dynamically map a YANG list/container to table names based on URI and payload. +The table-transformer is inherited to all descendant nodes unless another one is defined + +11. `sonic-ext:get-validate [function]`: +A special hook to validate YANG nodes, to populate data read from database, allows developers to instruct Transformer to choose a YANG node among multiple nodes, while constructing the response payload. +Typically used to check the “when” condition to validate YANG node among multiple nodes to choose only valid nodes from sibling nodes. + +---------- + + +Note that the key-transformer, field-transformer and subtree-transformer have a pair of callbacks associated with 2 way translation using a prefix - **YangToDB***function*, **DbToYang***function*. It is not mandatory to implement both functions. E.g. if you need a translation for GET operation only, you can implement only **DbToYang***function*. + +The template annotation file can be generated and used by the developers to define extensions to the yang paths as needed to translate data between YANG and ABNF format. Refer to the 3.2.2.7.8 Utilities. + +Here is the general guide you can check to find which extensions can be annotated in implementing your model. +```YANG +1) If the translation is simple mapping between YANG container/list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter +2) If the translation requires a complex translation with your codes, consider the following transformer extensions - key-transformer, field-transformer, subtree-transformer to take a control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope +3) If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-transformer to dynamically choose tables based on URI/payload +4) In Get operation access to non CONFIG_DB, you can use the db-name extension +5) In Get operation, you can annotate the subtree-transformer on the node to implement your own data access and translation with DbToYangxxx function +6) In case of mapping a container to TABLE/KET, you can use the key-name along with the table-name extension +``` + +###### 3.2.2.7.6 Public Functions + +`XlateToDb()` and `GetAndXlateFromDb` are used by the common app to request translations. + +```go +func XlateToDb(path string, opcode int, d *db.DB, yg *ygot.GoStruct, yt *interface{}) (map[string]map[string]db.Value, error) {} + +func GetAndXlateFromDB(xpath string, uri *ygot.GoStruct, dbs [db.MaxDB]*db.DB) ([]byte, error) {} +``` + +###### 3.2.2.7.7 Overloaded Methods + +The function prototypes for external transformer callbacks are defined in the following- + +```go +type XfmrParams struct { + d *db.DB + dbs [db.MaxDB]*db.DB + curDb db.DBNum + ygRoot *ygot.GoStruct + uri string + requestUri string //original uri using which a curl/NBI request is made + oper int + key string + dbDataMap *map[db.DBNum]map[string]map[string]db.Value + subOpDataMap map[int]*RedisDbMap // used to add an in-flight data with a sub-op + param interface{} + txCache *sync.Map + skipOrdTblChk *bool +} + +/** + * KeyXfmrYangToDb type is defined to use for conversion of Yang key to DB Key + * Transformer function definition. + * Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath + * Return: Database keys to access db entry, error + **/ +type KeyXfmrYangToDb func (inParams XfmrParams) (string, error) +/** + * KeyXfmrDbToYang type is defined to use for conversion of DB key to Yang key + * Transformer function definition. + * Param: XfmrParams structure having Database info, operation, Database keys to access db entry + * Return: multi dimensional map to hold the yang key attributes of complete xpath, error + **/ +type KeyXfmrDbToYang func (inParams XfmrParams) (map[string]interface{}, error) + +/** + * FieldXfmrYangToDb type is defined to use for conversion of yang Field to DB field + * Transformer function definition. + * Param: Database info, YgotRoot, operation, Xpath + * Return: multi dimensional map to hold the DB data, error + **/ +type FieldXfmrYangToDb func (inParams XfmrParams) (map[string]string, error) +/** + * FieldXfmrDbtoYang type is defined to use for conversion of DB field to Yang field + * Transformer function definition. + * Param: XfmrParams structure having Database info, operation, DB data in multidimensional map, output param YgotRoot + * Return: error + **/ +type FieldXfmrDbtoYang func (inParams XfmrParams) (map[string]interface{}, error) + +/** + * SubTreeXfmrYangToDb type is defined to use for handling the yang subtree to DB + * Transformer function definition. + * Param: XfmrParams structure having Database info, YgotRoot, operation, Xpath + * Return: multi dimensional map to hold the DB data, error + **/ +type SubTreeXfmrYangToDb func (inParams XfmrParams) (map[string]map[string]db.Value, error) +/** + * SubTreeXfmrDbToYang type is defined to use for handling the DB to Yang subtree + * Transformer function definition. + * Param : XfmrParams structure having Database pointers, current db, operation, DB data in multidimensional map, output param YgotRoot, uri + * Return : error + **/ +type SubTreeXfmrDbToYang func (inParams XfmrParams) (error) +/** + * ValidateCallpoint is used to validate a YANG node during data translation back to YANG as a response to GET + * Param : XfmrParams structure having Database pointers, current db, operation, DB data in multidimensional map, output param YgotRoot, uri + * Return : bool + **/ +type ValidateCallpoint func (inParams XfmrParams) (bool) +/** + * RpcCallpoint is used to invoke a callback for action + * Param : []byte input payload, dbi indices + * Return : []byte output payload, error + **/ +type RpcCallpoint func (body []byte, dbs [db.MaxDB]*db.DB) ([]byte, error) +/** + * PostXfmrFunc type is defined to use for handling any default handling operations required as part of the CREATE + * Transformer function definition. + * Param: XfmrParams structure having database pointers, current db, operation, DB data in multidimensional map, YgotRoot, uri + * Return: multi dimensional map to hold the DB data, error + **/ +type PostXfmrFunc func (inParams XfmrParams) (map[string]map[string]db.Value, error) +/** + * TableXfmrFunc type is defined to use for table transformer function for dynamic derviation of redis table. + * Param: XfmrParams structure having database pointers, current db, operation, DB data in multidimensional map, YgotRoot, uri + * Return: List of table names, error + **/ +type TableXfmrFunc func (inParams XfmrParams) ([]string, error) + +``` + +###### 3.2.2.7.8 Utilities + +The goyang package is extended to generate the template annotation file for any input yang files. A new output format type "annotate" can be used to generate the template annotation file.The goyang usage is as below: + +``` +Usage: goyang [-?] [--format FORMAT] [--ignore-circdep] [--path DIR[,DIR...]] [--trace TRACEFILE] [FORMAT OPTIONS] [SOURCE] [...] + -?, --help display help + --format=FORMAT + format to display: annotate, tree, types + --ignore-circdep + ignore circular dependencies between submodules + --path=DIR[,DIR...] + comma separated list of directories to add to search path + --trace=TRACEFILE + write trace into to TRACEFILE + +Formats: + annotate - generate template file for yang annotations + + tree - display in a tree format + + types - display found types + --types_debug display debug information + --types_verbose + include base information +``` +The $(SONIC_MGMT_FRAMEWORK)/gopkgs/bin is added to the PATH to run the goyang binary. + +For example: + +``` +goyang --format=annotate --path=/path/to/yang/models openconfig-acl.yang > openconfig-acl-annot.yang + +Sample output: +module openconfig-acl-annot { + + yang-version "1" + + namespace "http://openconfig.net/yang/annotation"; + prefix "oc-acl-annot" + + import openconfig-packet-match { prefix oc-pkt-match } + import openconfig-interfaces { prefix oc-if } + import openconfig-yang-types { prefix oc-yang } + import openconfig-extensions { prefix oc-ext } + + deviation oc-acl:openconfig-acl { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state/oc-acl:counter-capability { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set { + deviate add { + } + } + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set/oc-acl:type { + deviate add { + } + } +... +... + deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:config { + deviate add { + } + } +} +``` + + +##### 3.2.2.8 Config Validation Library (CVL) + +Config Validation Library (CVL) is an independent library to validate ABNF schema based SONiC (Redis) configuration. This library can be used by component like [Cfg-gen](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-config-engine/sonic-cfggen), Translib, [ZTP](https://github.com/Azure/SONiC/blob/master/doc/ztp/ztp.md) etc. to validate SONiC configuration data before it is written to Redis DB. Ideally, any component importing config_db.json file into Redis DB can invoke CVL API to validate the configuration. + +CVL uses SONiC YANG models written based on ABNF schema along with various constraints. These native YANG models are simple and have a very close mapping to the associated ABNF schema. Custom YANG extensions (annotations) are used for custom validation purpose. Specific YANG extensions (rather metadata) are used to translate ABNF data to YANG data. In the context of CVL these YANG models are called CVL YANG models and generated from SONiC YANG during build time. Opensource [libyang](https://github.com/CESNET/libyang) library is used to perform YANG data validation. + +SONiC YANG can be used as Northbound YANG for management interface by adding other data definitions such as state data (read only data), RPC or Notification as needed.Such YANG models are called SONiC NBI YANG models. Since CVL validates configuration data only, these data definition statements are ignored by CVL. During build time CVL YANG is actually generated from SONiC NBI YANG models with the help of CVL specific pyang plugin. + +CVL supports multiple user-defined Redis database instances based on the [Multi DB instance HLD](https://github.com/Azure/SONiC/blob/master/doc/database/multi_database_instances.md) specification. During CVL initialization time, the DB configuration file is read from the predefined location and details are stored in internal data structure. CVL implements internal API for connecting to a Redis DB instance using the details stored in its internal data structure. + +###### 3.2.2.8.1 Architecture + +![CVL architecture](images/CVL_Arch.jpg) + +1. During build time, developer writes SONiC YANG schema based on ABNF schema following [SONiC YANG Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) and adds metadata and constraints as needed. Custom YANG extensions are defined for this purpose. +2. The YANG models are compiled using Pyang compiler. CVL specific YIN schema files are derived from SONiC YANG with the help of CVL specific pyang plugin. Finally, generated YIN files are packaged in the build. +3. During boot up/initialization sequence, YIN schemas generated from SONiC YANG models are parsed and schema tree is build using libyang API. +4. Application calls CVL APIs to validate the configuration. In case of Translib library DB Access layer calls appropriate CVL APIs to validate the configuration. +5. ABNF JSON goes through a translator and YANG data is generated. Metadata embedded in the YANG schema are used to help this translation process. +6. Then YANG data is fed to libyang for performing syntax validation first. If error occurs, CVL returns appropriate error code and details to application without proceeding further. +7. If syntax validation is successful, CVL uses dependent data from translated YANG data or if needed, fetches the dependent data from Redis DB. Dependent data refers to the data needed to validate constraint expressed in YANG syntax such as 'leafref', 'must', 'when' etc. +8. Finally translated YANG data and dependent data are merged and fed to libyang for performing semantics validation. If error occurs, CVL returns appropriate error code and details to application, else success is returned. +9. Platform validation is specific syntax and semantics validation only performed with the help of dynamic platform data as input. + +###### 3.2.2.8.2 Validation types + +Config Validator does Syntactic, Semantic validation and Platform Validation as per YIN schema with metadata. + +###### 3.2.2.8.2.1 Syntactic Validation + +Following are some of the syntactic validations supported by the config validation library + +* Basic data type +* Enum +* Ranges +* Pattern matching +* Check for mandatory field +* Check for default field +* Check for number of keys are their types +* Check for table size etc. + +###### 3.2.2.8.2.2 Semantic Validation + +* Check for key reference existence in other table +* Check any conditions between fields within same table +* Check any conditions between fields across different table + +###### 3.2.2.8.2.3 Platform specific validation + +There can be two types of platform constraint validation + +###### 3.2.2.8.2.3.1 Static Platform Constraint Validation + +* Platform constraints (range, enum, ‘must’/’when’ expression etc.) are expressed in SONiC YANG deviation model for each feature. + +Example of 'deviation' : + +``` +module sonic-acl-deviation { + ...... + ...... + deviation /sacl:sonic-acl/sacl:ACL_TABLE/sacl:ACL_TABLE_LIST { + deviate add { + max-elements 3; + } + } + + deviation /sacl:sonic-acl/sacl:ACL_RULE/sacl:ACL_RULE_LIST { + deviate add { + max-elements 768; + } + } +} +``` + +* All deviation models are compiled along with corresponding CVL YANG model and are placed in platform specific schema folder. At runtime based on detected platform from provisioned “DEVICE_METADATA:platform” field, deviation files are applied. + +* Here is the sample folder structure for platform specific deviation files. +``` + models/yang/sonic/platform/ + |-- accton_as5712 + | |--- sonic-acl-deviation.yang + |-- quanta_ix8 + |--- sonic-acl-deviation.yang +``` + +###### 3.2.2.8.2.3.2 Dynamic Platform Constraint Validation + +###### 3.2.2.8.2.3.2.1 Platform data is available in Redis DB table. + +* Based on Redis DB, platform specific YANG data defintion can be added in SONiC YANG models. Constraints like ‘must’ or ‘when’ are used in feature YANG by cross-referencing platform YANG models. + +###### 3.2.2.8.2.3.2.2 Platform data is available through APIs + +* If constraints cannot be expressed using YANG syntax or platform data is available through feature/component API (APIs exposed by a feature to query platform specific constants, resource limitation etc.), custom validation needs to be hooked up in SONiC YANG model through custom YANG extension. +* Feature developer implements the custom validation functions with functional validation code. The validation function may call feature/component API and fetch required parameter for checking constraints. +* Based on YANG extension syntax, CVL will call the appropriate custom validation function along with YANG instance data to be validated. Below is the custom validation context structure definition. + +``` + //Custom validation context passed to custom validation function + type CustValidationCtxt struct { + ReqData []CVLEditConfigData //All request data + CurCfg *CVLEditConfigData //Current request data for which validation should be done + YNodeName string //YANG node name + YNodeVal string //YANG node value, leaf-list will have "," separated value + YCur *xmlquery.Node //YANG data tree + RClient *redis.Client //Redis client + } +``` + +###### 3.2.2.8.3 CVL APIs + +``` + //Structure for key and data in API + type CVLEditConfigData struct { + VType CVLValidateType //Validation type + VOp CVLOperation //Operation type + Key string //Key format : "PORT|Ethernet4" + Data map[string]string //Value : {"alias": "40GE0/28", "mtu" : 9100, "admin_status": down} + } + + /* CVL Error Structure. */ + type CVLErrorInfo struct { + TableName string /* Table having error */ + ErrCode CVLRetCode /* Error Code describing type of error. */ + Keys []string /* Keys of the Table having error. */ + Value string /* Field Value throwing error */ + Field string /* Field Name throwing error . */ + Msg string /* Detailed error message. */ + ConstraintErrMsg string /* Constraint error message. */ + } + + /* Structure for dependent entry to be deleted */ + type CVLDepDataForDelete struct { + RefKey string /* Ref Key which is getting deleted */ + Entry map[string]map[string]string /* Entry or field which should be deleted as a result */ + } + + /* Maintain time stats for call to ValidateEditConfig(). */ + type ValidationTimeStats struct { + Hits uint /* Total number of times ValidateEditConfig() called */ + Time time.Duration /* Total time spent in ValidateEditConfig() */ + Peak time.Duration /* Highest time spent in ValidateEditConfig() */ + } + + /* Validation type */ + type CVLValidateType uint + const ( + VALIDATE_NONE CVLValidateType = iota //Data is used as dependent data + VALIDATE_SYNTAX //Syntax is checked and data is used as dependent data + VALIDATE_SEMANTICS //Semantics is checked + VALIDATE_ALL //Syntax and Semantics are checked + ) + + /* Validation operation */ + type CVLOperation uint + const ( + OP_NONE CVLOperation = 0 //Used to just validate the config without any operation + OP_CREATE = 1 << 0//For Create operation + OP_UPDATE = 1 << 1//For Update operation + OP_DELETE = 1 << 2//For Delete operation + ) + + /* Error code */ + type CVLRetCode int + const ( + CVL_SUCCESS CVLRetCode = iota + CVL_SYNTAX_ERROR /* Generic syntax error */ + CVL_SEMANTIC_ERROR /* Generic semantic error */ + CVL_ERROR /* Generic error */ + CVL_SYNTAX_MISSING_FIELD /* Missing field */ + CVL_SYNTAX_INVALID_FIELD /* Invalid Field */ + CVL_SYNTAX_INVALID_INPUT_DATA /*Invalid Input Data */ + CVL_SYNTAX_MULTIPLE_INSTANCE /* Multiple Field Instances */ + CVL_SYNTAX_DUPLICATE /* Duplicate Fields */ + CVL_SYNTAX_ENUM_INVALID /* Invalid enum value */ + CVL_SYNTAX_ENUM_INVALID_NAME /* Invalid enum name */ + CVL_SYNTAX_ENUM_WHITESPACE /* Enum name with leading/trailing whitespaces */ + CVL_SYNTAX_OUT_OF_RANGE /* Value out of range/length/pattern (data) */ + CVL_SYNTAX_MINIMUM_INVALID /* min-elements constraint not honored */ + CVL_SYNTAX_MAXIMUM_INVALID /* max-elements constraint not honored */ + CVL_SEMANTIC_DEPENDENT_DATA_MISSING /* Dependent Data is missing */ + CVL_SEMANTIC_MANDATORY_DATA_MISSING /* Mandatory Data is missing */ + CVL_SEMANTIC_KEY_ALREADY_EXIST /* Key already existing. */ + CVL_SEMANTIC_KEY_NOT_EXIST /* Key is missing. */ + CVL_SEMANTIC_KEY_DUPLICATE /* Duplicate key. */ + CVL_SEMANTIC_KEY_INVALID /* Invaid key */ + CVL_NOT_IMPLEMENTED /* Not implemented */ + CVL_INTERNAL_UNKNOWN /*Internal unknown error */ + CVL_FAILURE /* Generic failure */ + ) +``` + +1. func Initialize() CVLRetCode + - Initialize the library only once, subsequent calls does not affect once library is already initialized . This automatically called when if ‘cvl’ package is imported. + +2. func Finish() + - Clean up the library resources. This should ideally be called when no more validation is needed or process is about to exit. + +3. func (c *CVL) ValidateConfig(jsonData string) CVLRetCode + - Just validates json buffer containing multiple row instances of the same table, data instance from different tables. All dependency are provided in the payload. This is useful for bulk data validation. + +4. func (c *CVL) ValidateEditConfig(cfgData []CVLEditConfigData) (cvlErr CVLErrorInfo, ret CVLRetCode) + - Validates the JSON data for create/update/delete operation. Syntax or Semantics Validation can be done separately or together. Related data should be given as dependent data for validation to be successful. + +5. func (c *CVL) ValidateKeys(key []string) CVLRetCode + - Just validates the key and checks if it exists in the DB. It checks whether the key value is following schema format. Key should have table name as prefix. + +6. func (c *CVL) ValidateFields(key string, field string, value string) CVLRetCode + - Just validates the field:value pair in table. Key should have table name as prefix. + +7. func (c *CVL) SortDepTables(inTableList []string) ([]string, CVLRetCode) + - Sort the list of given tables as per their dependency imposed by leafref. + +8. func (c *CVL) GetOrderedTables(yangModule string) ([]string, CVLRetCode) + - Get the sorted list of tables in a given YANG module based on leafref relation. + +9. func (c *CVL) GetDepTables(yangModule string, tableName string) ([]string, CVLRetCode) + - Get the list of dependent tables for a given table in a YANG module. + +10. func (c *CVL) GetDepDataForDelete(redisKey string) ([]CVLDepDataForDelete) + - Get the dependent data (Redis keys) to be deleted or modified for a given entry getting deleted. + +11. func GetValidationTimeStats() (ValidationTimeStats) + - Provides statistics details of time spent in ValidateEditConfig(). + +12. func ClearValidationTimeStats() + - Clears statistics details of time spent in ValidateEditConfig(). + +##### 3.2.2.9 Redis DB + +Please see [3.2.2.6.5 DB access layer](#32265-db-access-layer) + +##### 3.2.2.10 Non DB data provider + +Currently, it is up to each App module to perform the proprietary access +mechanism for the app specific configuration. + +## 4 Flow Diagrams + +### 4.1 REST SET flow + +![REST SET flow](images/write.jpg) + +1. REST client can send any of the write commands such as POST, PUT, PATCH or DELETE and it will be handled by the REST Gateway. +2. All handlers in the REST gateway will invoke a command request handler. +3. Authentication and authorization of the commands are done here. +4. Request handler invokes one of the write APIs exposed by the Translib. +5. Translib infra populates the YGOT structure with the payload of the request and performs a syntactic validation +6. Translib acquires the write lock (mutex lock) to avoid another write happening from the same process at the same time. +7. Translib infra gets the App module corresponding to the incoming URI. +8. Translib infra calls the initialize function of the App module with the YGOT structures, path and payload. +9. App module caches the incoming data into the app structure. +10. App module calls Transformer function to translate the request from cached YGOT structures into Redis ABNF format. It also gets all the keys that will be affected as part of this request. +11. App module returns the list of keys that it wants to keep a watch on along with the status. +12. Translib infra invokes the start transaction request exposed by the DB access layer. +13. DB access layer performs a WATCH of all the keys in the Redis DB. If any of these keys are modified externally then the EXEC call in step 26 will fail. +14. Status being returned from Redis. +15. Status being returned from DB access layer. +16. Translib then invokes the processWrite API on the App module. +17. App modules perform writes of the translated data to the DB access layer. +18. DB access layer validates the writes using CVL and then caches them. +19. Status being returned from DB access layer. +20. Status being returned from App module. +21. Translib infra invokes the commit transaction on the DB access layer. +22. DB access layer first invokes MULTI request on the Redis DB indicating there are multiple writes coming in, so commit everything together. All writes succeed or nothing succeeds. +23. Status returned from Redis. +24. Pipeline of all the cached writes are executed from the DB access layer. +25. Status retuned from Redis. +26. EXEC call is made to the Redis DB. Here if the call fails, it indicates that one of the keys that we watched has changed and none of the writes will go into the Redis DB. +27. Status returned from Redis DB. +28. Status returned from DB access layer. +29. Write lock acquired in Step 6 is released. +30. Status returned from the Translib infra. +31. REST Status returned from the Request handler. +32. REST response is sent by the REST gateway to the REST client. + +### 4.2 REST GET flow + +![REST GET flow](images/read.jpg) + +1. REST GET request from the REST client is sent to the REST Gateway. +2. REST Gateway invokes a common request handler. +3. Authentication of the incoming request is performed. +4. Request handler calls the Translib exposed GET API with the uri of the request. +5. Translib infra gets the App module corresponding to the incoming uri. +6. Translib infra calls the initialize function of the App module with the YGOT structures and path. App module caches them. +7. Status retuned from App module. +8. App module queries Transformer to translate the path to the Redis keys that need to be queried. +9. Status returned from App module. +10. Translib infra calls the processGet function on the App module +11. App modules calls read APIs exposed by the DB access layer to read data from the Redis DB. +12. Data is read from the Redis DB is returned to the App module +13. App module fills the YGOT structure with the data from the Redis DB and validated the filled YGOT structure for the syntax. +14. App module converts the YGOT structures to JSON format. +15. IETF JSON payload is returned to the Translib infra. +16. IETF JSON payload is returned to the request handler. +17. Response is returned to REST gateway. +18. REST response is returned to the REST client from the REST gateway. + +### 4.3 Translib Initialization flow + +![Translib Initialization flow](images/Init.jpg) + +1. App module 1 `init` is invoked +2. App module 1 calls `Register` function exposed by Translib infra to register itself with the Translib. +3. App module 2 `init` is invoked +4. App module 2 calls `Register` function exposed by Translib infra to register itself with the Translib. +5. App module N `init` is invoked +6. App module N calls `Register` function exposed by Translib infra to register itself with the Translib. + +This way multiple app modules initialize with the Translib infra during boot up. + +### 4.4 gNMI flow + +![GNMI flow](images/GNMI_flow.jpg) + +1. GNMI requests land in their respective GET/SET handlers which then redirect the requests to corresponding data clients. +2. If user does not provide target field then by default the request lands to the transl_data_client. +3. Next, the transl_data_client provides higher level abstraction along with collating the responses for multiple paths. +4. Transl Utils layer invokes Translib API's which in turn invoke App-Module API's and data is retrieved and modified in/from Redis Db/non-DB as required. + +### 4.5 CVL flow + +![CVL flow](images/CVL_flow.jpg) + +Above is the sequence diagram explaining the CVL steps. Note that interaction between DB Access layer and Redis including transactions is not shown here for brevity. + +1. REST/GNMI invokes one of the write APIs exposed by the Translib. +2. Translib infra populates the YGOT structure with the payload of the request and performs a syntactic validation. +3. Translib acquires the write lock (mutex lock) to avoid another write happening from the same process at the same time. +4. Translib infra gets the App module corresponding to the incoming uri. +5. Translib infra calls the initialize function of the App module with the YGOT structures, path and payload. +6. App module calls Transformer to translate the request from cached YGOT structure into Redis ABNF format. It also gets all the keys that will be affected as part of this request. +7. App modules returns the list of keys that it wants to keep a watch on along with the status. +8. Translib infra invokes the start transaction request exposed by the DB access layer. +9. Status being returned from DB access layer. +10. Translib then invokes the processWrite API on the App module. +11. App modules perform writes of the translated data to the DB access layer. +12. DB access layer calls validateWrite for CREATE/UPDATE/DELETE operation. It is called with keys and Redis/ABNF payload. +13. validateSyntax() feeds Redis data to translator internally which produces YANG XML. This is fed to libyang for validating the syntax. +14. If it is successful, control goes to next step, else error is returned to DB access layer. The next step is to ensure that keys are present in Redis DB for Update/Delete operation. But keys should not be present for Create operation. +15. Status is returned after checking keys. +16. CVL gets dependent data from incoming Redis payload. For example if ACL_TABLE and ACL_RULE is getting created in a single request. +17. Otherwise dependent should be present in Redis DB, query is sent to Redis to fetch it. +18. Redis returns response to the query. +19. Finally request data and dependent is merged and validateSemantics() is called. +20. If above step is successful, success is returned or else failure is returned with error details. +21. DB Access layer forwards the status response to App mpdule. +22. App module forwards the status response to Translib infra. +23. Translib infra invokes the commit transaction on the DB access layer. +24. Status is returned from DB access layer after performing commit operation. +25. Write lock acquired in Step 3 is released. +26. Final response is returned from the Translib infra to REST/GNMI. + +## 5 Developer Work flow +Developer work flow differs for standard YANG (IETF/OpenConfig) vs proprietary YANG used for a feature. When a standards-based YANG model is chosen for a new feature, the associated Redis DB design should take the design of this model into account - the closer the mapping between these, then the less translation logic is required in the Management path. This simplifies the work flow as translation intelligence can be avoided as both Redis schema and NB YANG schema are aligned. + +Where the YANG does not map directly to the Redis DB, the management framework provides mechanisms to represent complex translations via developer written custom functions. + +SONiC YANG should always be developed for a new feature, and should be purely based on the schema of config objects in DB. + +For the case where developer prefers to write a non-standard YANG model for a new or existing SONIC feature, the YANG should be written aligned to Redis schema such that the same YANG can be used for both northbound and CVL. This simplifies the developer work flow (explained in section 5.1) + + +### 5.1 Developer work flow for non Standards-based SONiC YANG + + Typical steps for writing a non-standard YANG for management framework are given below. +- Write the SONiC YANG based upon the Redis DB design. +- Add 'must','when' expressions to capture the dependency between config objects. +- Add required non-config, notification and RPC objects to the YANG. +- Add meta data for transformer. + +#### 5.1.1 Define SONiC YANG schema +Redis schema needs to be expressed in SONiC proprietary YANG model with all data types and constraints. Appropriate custom YANG extensions need to be used for expressing metadata. The YANG model is used by Config Validation Library(CVL) to provide automatic syntactic and semantic validation and Translib for Northbound management interface. + +Custom validation code needs to be written if some of the constraints cannot be expressed in YANG syntax. + +Please refer to [SONiC YANG Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) for detiailed guidelines on writing SONiC YANG. + +#### 5.1.2 Generation of REST server stubs and Client SDKs for YANG based APIs + +* Place the main YANG modules under sonic-mgmt-common/models/yang directory. + * By placing YANG module in this directory, on the next build, OpenAPI YAML (OpenAPI spec) is generated for the YANG. + * If there is YANG which is augmenting the main YANG module, this augmenting YANG should also be placed in sonic-mgmt-common/models/yang directory itself. +* Place all dependent YANG modules which are imported into the main YANG module such as submodules or YANGs which define typedefs, etc under sonic-mgmt-common/models/yang/common directory. + * By placing YANG module in this directory, OpenAPI YAML (OpenAPI spec) is not generated for the YANG modules, but the YANGs placed under sonic-mgmt-common/models/yang can utilize or refer to types, and other YANG constraints from the YANG modules present in this directory. + * Example: ietf-inet-types.yang which mainly has typedefs used by other YANG models and generally we won't prefer having a YAML for this YANG, this type of YANG files can be placed under sonic-mgmt-common/models/yang/common. +* Generation of REST server stubs and client SDKs will automatically happen when make command is executed as part of the build. + +#### 5.1.3 Config Translation App (Go language) +Config Translation App consists of two parts - Transformer and App module. They translate the data in Northbound API schema to the native Redis schema (refer to section 5.1.1) and vice versa. All Northbound API services like REST, GNMI, NETCONF invoke this App to read and write data through Translib API calls. + +Key features: + +* Go language. +* YANG to Redis and vice-versa data translation is handled by Transformer. The data translation happens based on the YANG model developed as per section 5.1.1. +* The processing of data is taken care by App module + * App consumes/produces YANG data through [YGOT](https://github.com/openconfig/YGOT) structures. + * Framework provides Go language APIs for Redis DB access. APIs are similar to existing Python APIs defined in sonic-py-swsssdk repo. + * The Transformer converts between these formats. + + * For read operation + * App module receives the YANG path to read in a Go struct + * App module uses the Transformer to get a DB entry(s) reference from this path + * App module reads the Redis entry(s) + * App module uses the Transformer to convert the returned DB object(s) to a Go struct + * App module returns this to TransLib + * For write operations + * App module receives the target YANG path and data in a Go struct + * App module uses the Transformer to translates the Go struct into appropriate Redis calls + * DB Access layer takes care of DB transactions - write everything or none + +* REST server provides a test UI for quick UT of App modules. This UI lists all REST APIs for a YANG and provides option to try them out. +* Pytest automation integration can make use of direct REST calls or CLI (which also makes use of REST APIs internally). Framework generates REST client SDK to facilitate direct REST API calls. + +#### 5.1.4 IS CLI +IS CLI is achieved using KLISH framework. + +* CLI tree is expressed in the XML file with node data types and hierarchy along with different modes. +* Actioner handler needs to be hooked up in XML for corresponding CLI syntax. Actioner handler should be developed to call client SDK APIs (i.e one action handler might need to call multiple client SDK APIs.) +* Show command output formatting is achieved using [Jinja](http://jinja.pocoo.org/) templates. So, the developer needs to check if an existing template can be used or new template needs to be written. + +#### 5.1.5 gNMI +There is no specific steps required for gNMI. + + +### 5.2 Developer work flow for standard-based (e.g. OpenConfig/IETF) YANG + +#### 5.2.1 Identify the standard YANG module for the feature for northbound APIs. +The Developer starts by selecting the standard YANG to be the basis for the Management data model. For SONiC, OpenConfig is the preferred source. However, in the absence of a suitable model there, other standards can be considered (e.g. IETF, IEEE, OEM de-facto standards). + +The SONiC feature implementation may not exactly match the chosen model - there may be parts of the YANG that are not implemented by (or have limitations in) the feature, or the Developer may wish to expose feature objects that are not covered by the YANG. In such cases, the Developer should extend the model accordingly. The preferred method is to write a deviation file and then use modification statements there (e.g. deviate, augment) accordingly. + +#### 5.2.2 Define the Redis schema for the new feature. (not applicable for legacy/existing feature) +The Redis DB design should take the YANG design into account, and try to stay as close to it as possible. This simplifies the translation processes within the Management implementation. Where this is not possible or appropriate, custom translation code must be provided. + + +#### 5.2.3 Define SONiC YANG schema +Redis schema needs to be expressed in SONiC YANG model with all data types and constraints. Appropriate custom YANG extensions need to be used for expressing this metadata. The YANG model is used by Config Validation Library(CVL)to provide automatic syntactic and semantic validation. + +Custom validation code needs to be written if some of the constraints cannot be expressed in YANG syntax. + +Please refer to [SONiC YANG Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) for detiailed guidelines on writing SONiC YANG. + +#### 5.2.4 Generation of REST server stubs and Client SDKs for YANG based APIs + +* Place the main YANG modules under sonic-mgmt-common/models/yang directory. + * By placing YANG module in this directory, YAML (OpenAPI spec) is generated for the YANG. + * If there is YANG which is augmenting the main YANG module, this augmenting YANG should also be placed in sonic-mgmt-common/models/yang directory itself. +* Place all dependent YANG modules such as submodules or YANGs which define typedefs, etc under sonic-mgmt-common/models/yang/common directory. + * By placing YANG module in this directory, YAML (OpenAPI spec) is not generated for the YANG modules, but the YANGs placed under sonic-mgmt-common/models/yang can utilize or refer to types, and other YANG constraints from the YANG modules present in this directory. + * Example: ietf-inet-types.yang which mainly has typedefs used by other YANG models and generally we won't prefer having a YAML for this YANG, this type of YANG files can be placed under sonic-mgmt-common/models/yang/common. +* Generation of REST-server stubs and client SDKs will automatically happen when make command is executed as part of the build. + + +#### 5.2.5 Config Translation App (Go language) +Config Translation App consists of two parts - Transformer and App module. They translate the data in Northbound API schema (defined in step#1) to the native Redis schema (defined in step#2) and vice versa. All Northbound API services like REST, GNMI, NETCONF invoke this App to read and write data. + +Key features: + +* Go language. +* YANG to Redis and vice-versa data translation is handled by Transformer. In order to facilitate data translation, the developer needs to provide just the YANG file for the data model + * YANG file for the data model + * Optionally, a YANG annotation file (refer to [3.2.2.7.8 Utilities](#32278-utilities)) to define translation hints to map YANG objects to DB objects. These translation hints are external callbacks for performing complex translation whereas simple translations are handled by Transformer's built-in methods. The annotation file is also placed in `sonic-mgmt-common/models/yang` + * Code to define the translation callbacks, in `sonic-mgmt-common/src/translib/transformer` +* The processing of data is taken care by App module + * App consumes/produces YANG data through [YGOT](https://github.com/openconfig/YGOT) structures. + * Framework provides Go language APIs for Redis DB access. APIs are similar to existing Python APIs defined in sonic-py-swsssdk repo. + * For read operation + * App module receives the YANG path to read in a Go struct + * App module uses the Transformer to get a DB entry(s) reference from this path + * App module reads the Redis entry(s) + * App module uses the Transformer to convert the returned DB object(s) to a Go struct + * App module returns this to TransLib + * For write operations + * App module receives the target YANG path and data as YGOT tree + * App module translates the YGOT tree data into appropriate Redis calls using reference from Transformer + * App module also handles additional complex logic like transaction ordering or checking for dependencies + * Translation Framework takes care of DB transactions - write everything or none +* REST server provides a test UI for quick UT of translation app. This UI lists all REST APIs for a YANG and provide option to try them out. REST server invokes Translation Apps. +* Spytest automation integration can make use of direct REST calls or CLI (which also makes use of REST internally - step#5). Framework generates REST client SDK to facilitate direct REST calls. + + +#### 5.2.6 IS CLI +IS CLI is achieved using KLISH framework and steps are same as in SONiC YANG model. Please refer to section [5.1.4 IS CLI](#514-IS-CLI). + +#### 5.2.7 gNMI +There is no specific steps required for gNMI. + + +## 6 Error Handling + +Validation is done at both north bound interface and against database schema. Appropriate error code is returned for invalid configuration. +All application errors are logged into syslog. + +## 7 Serviceability and Debug + +1. Detailed syslog messages to help trace a failure. +2. Debug commands will be added when debug framework becomes available. +3. CPU profiling enable/disable with SIGUR1 signal. + +## 8 Warm Boot Support + +Management Framework does not disrupt data plane traffic during warmboot. No special handling required for warmboot. + +## 9 Scalability + +Manageability framework will be scalable to handle huge payloads conforming to the standard/SONiC yang models. + +## 10 Unit Test + +#### GNMI +1. Verify that gnmi_get is working at Toplevel module +2. Verify thet gnmi_get is working for each ACL Table +3. Verify gnmi_get working for each ACL Rule: +4. Verify that gnmi_get is working for all ACL interfaces +5. Verify that gnmi_get is working for each ACL interface name +6. Verify that gnmi_get fails for non-existent ACL name and type +7. Verify that TopLevel node can be deleted +8. Verify that a particular ACL Table can be deleted +9. Verify that ACL rule can be deleted +10. Verify that ACL table can be created +11. Verify that ACL rule can be created +12. Verify that ACL binding can be created +13. Verify that creating rule on non existent ACL gives error +14. Verify that giving invalid interface number is payload gives error. +15. Verify that GNMI capabalities is returning correctly. + +#### Request Binder (YGOT) +1. create a YGOT object binding for the uri ends with container +2. create a YGOT object binding for the uri ends with leaf +3. create a YGOT object binding for the uri ends with list +4. create a YGOT object binding for the uri ends with leaf-list +5. create a YGOT object binding for the uri which has keys +6. create a YGOT object binding for the uri which has keys and ends with list with keys +7. validate the uri which has the correct number of keys +8. validate the uri which has the invalid node name +9. validate the uri which has the invalid key value +10. validate the uri which has the incorrect number of keys +11. validate the uri which has the invalid leaf value +12. validate the payload which has the incorrect number of keys +13. validate the payload which has the invalid node name +14. validate the payload which has the invalid leaf value +15. validate the uri and the payload with the "CREATE" operation +16. validate the uri and the payload with the "UPDATE" operation +17. validate the uri and the payload with the "DELETE" operation +18. validate the uri and the payload with the "REPLACE" operation +19. validate the getNodeName method for LIST node +20. validate the getNodeName method for leaf node +21. validate the getNodeName method for leaf-list node +22. validate the getParentNode method for LIST node +23. validate the getParentNode method for leaf node +24. validate the getParentNode method for leaf-list node +25. validate the getObjectFieldName method for LIST node +26. validate the getObjectFieldName method for leaf node +27. validate the getObjectFieldName method for leaf-list node + +#### DB access layer +1. Create, and close a DB connection. (NewDB(), DeleteDB()) +2. Get an entry (GetEntry()) +3. Set an entry without Transaction (SetEntry()) +4. Delete an entry without Transaction (DeleteEntry()) +5. Get a Table (GetTable()) +6. Set an entry with Transaction (StartTx(), SetEntry(), CommitTx()) +7. Delete an entry with Transaction (StartTx(), DeleteEntry(), CommitTx()) +8. Abort Transaction. (StartTx(), DeleteEntry(), AbortTx()) +9. Get multiple keys (GetKeys()) +10. Delete multiple keys (DeleteKeys()) +11. Delete Table (DeleteTable()) +12. Set an entry with Transaction using WatchKeys Check-And-Set(CAS) +13. Set an entry with Transaction using Table CAS +14. Set an entry with Transaction using WatchKeys, and Table CAS +15. Set an entry with Transaction with empty WatchKeys, and Table CAS +16. Negative Test(NT): Fail a Transaction using WatchKeys CAS +17. NT: Fail a Transaction using Table CAS +18. NT: Abort an Transaction with empty WatchKeys/Table CAS +19. NT: Check V logs, Error logs +20. NT: GetEntry() EntryNotExist. + +#### ACL app (via REST) +1. Verify that if no ACL and Rules configured, top level GET request should return empty response +2. Verify that bulk request for ACLs, multiple Rules within each ACLs and interface bindings are getting created with POST request at top level +3. Verify that all ACLs and Rules and interface bindings are shown with top level GET request +5. Verify that GET returns all Rules for single ACL +6. Verify that GET returns Rules details for single Rule +7. Verify that GET returns all interfaces at top level ACL-interfaces +8. Verify that GET returns one interface binding +9. Verify that single or multiple new Rule(s) can be added to existing ACL using POST/PATCH request +10. Verify that single or mutiple new ACLs can be added using POST/PATCH request +11. Verify that single or multiple new interface bindings can be added to existing ACL using POST/PATCH request +12. Verify that single Rule is deleted from an ACL with DELETE request +13. Verify that single ACL along with all its Rules and bindings are deleted with DELETE request +14. Verify that single interface binding is deleted with DELETE request +15. Verify that all ACLs and Rules and interface bindings are deleted with top level DELETE request +16. Verify that CVL throws error is ACL is created with name and type same as existing ACL with POST request +17. Verify that CVL throws error is RULE is created with SeqId, ACL name and type same as existing Rule with POST request +18. Verify that GET returns error for non exising ACL or Rule +19. Verify that CVL returns errors on creating rule under non existent ACL using POST request +20. Verify that CVL returns error on giving invalid interface number in payload during binding creation + +#### CVL +1. Check if CVL validation passes when data is given as JSON file +2. Check if CVL Validation passes for Tables with repeated keys like QUEUE,WRED_PROFILE and SCHEDULER +3. Check if CVL throws error when bad schema is passed +4. Check if debug trace level is changed as per updated conf file on receiving SIGUSR2 +5. Check must constraint for DELETE throws failure if condition fails, (if acl is a bound to port, deleting the acl rule throws error due to must constraint) +6. Check if CVL Validation passes when data has cascaded leafref dependency (Vlan Member->Vlan->Port) +7. Check if Proper Error Tag is returned when must condition is not satisfied +8. Check if CVL Validation passes if Redis is loaded with dependent data for UPDATE operation. +9. Check is CVL Error is returned when any mandatory node is not provided. +10. Check if CVL validation passes when global cache is updated for PORT Table for "must" expressions. +11. Check if CVL is able to validate JSON data given in JSON file for VLAN , ACL models +12. Check if CVL initialization is successful +13. Check if CVL is able to validate JSON data given in string format for CABLE LENGTH +14. Check if CVL failure is returned if input JSON data has incorrect key +15. Check if CVL is returning CVL_SUCCESS for Create operation if Dependent Data is present in Redis +16. Check if CVL is returning CVL_FAILURE for Create operation with invalid field for CABLE_LENGTH . +17. Check is CVL Error is returned for any invalid field in leaf +18. Check is Valid CVL_SUCCESS is returned for Valid field for ACL_TABLE when data is given in Test Structure +19. Check is Valid CVL_SUCCESS is returned for Valid field for ACL_RULE where Dependent data is provided in same session +20. Check if CVL is returning CVL_FAILURE for Create operation with invalid Enum vaue +21. Check if CVL validation fails when incorrect IP address prefix is provided. +22. Check is CVL validation fails when incorrect IP address is provided. +23. Check is CVL validation fails when out of bound are provided. +24. Check is CVL validation fails when invalid IP protocol +25. Check is CVL validation fails when out of range values are provided. +26. Check if CVL validation fails when incorrect key name is provided . +27. Check if CVL validation passes is any allowed special character is list name. +28. Check if CVL validation fails when key names contains junk characters. +29. Check if CVL validation fails when additional extra node is provided +30. Check is CVL validation passes when JSON data is given as buffer for DEVICE METEADATA +31. Check if CVL validation fails when key name does not contain separators. +32. Check if CVL validation fails when one of the keys is missing for Create operation +33. Check if CVL validation fails when there are no keys between separators for Create operation +34. Check if CVL validation fails when missing dependent data is provided for Update operation in same transaction +35. Check if CVL validation fails when missing dependent data is provided for Create operation in same transaction. +36. Check if CVL validation fails when there are no keys between separators for DELETE operation +37. Check if CVL validation fails when there are no keys between separators for UPDATE operation +38. Check if CVL validation fails when invalid key separators are provided for Delete operation +39. Check if CVL validation fails if UPDATE operation is given with invalid enum value +40. Check if CVL validation fails if UPDATE operation is given with invalid key containing missing keys +41. Check if CVL validation passes with dependent data present in same transaction for DELETE operation. +42. Check if CVL validation fails if DELETE operation is given with missing key for DELETE operation. +43. Check if CVL validation fails if UPDATE operation is given with missing key +44. Check if CVL validation fails when an existing key is provided in CREATE operation +45. Check if CVL validation passes for INTERFACE table +46. Check if CVL validation fails when configuration not satisfying must constraint is provided +47. Check if CVL validation passes when Redis has valid dependent data for UPDATE operation +48. Check if CVL validation fails when two different sequences are passed(Create and Update is same transaction) +49. Check if CVL validation fails for UPDATE operation when Redis does not have dependent data. +50. Check if CVL validation passes with valid dependent data given for CREATE operation. +51. Check if CVL validation fails when user tries to delete non existent key +52. Check if CVL Validation passes if Cache contains dependent data populated in same sessions but separate transaction. +53. Check if CVL Validation passes if Cache data dependent data that is populated across sessions +54. Check if CVL Validation fails when incorrect dependent Data is provided for CREATE operation +55. Check if CVL validation passes when valid dependent data is provided for CREATE operation +56. Check if Proper Error Tag is returned when must condition is not satisfied in "range" +57. Check if Proper Error Tag is returned when must condition is not satisfied in "length" +58. Check if Proper Error Tag is returned when must condition is not satisfied in "pattern" +59. Check if DELETE fails when ACL Table is tried to Rule or when DELETE tries to delete TABLE with non-empty leafref +60. Check if validation fails when non-existent dependent data is provided. +61. Check if CVL validation fails when DELETE tries to delete leafref of another table(delete ACL table referenced by ACL rule) +62. Check if CVL Validation fails when unrelated chained dependent data is given. +63. Check if CVL Validation fails when VLAN range is out of bound and proper error message is returned +64. Check if Logs are printed as per configuration in log configuration file. +65. Check if DELETE operation is performed on single field +66. Check if CVL validation passes when valid dependent data is provided using a JSON file. +67. Check if CVL validation is passed when when delete is performed on Table and then connected leafref +68. Check if CVL validation is passes when JSON data can be given in file format +69. Check if CVL Finish operation is successful +70. Check if CVL validation passes when Entry can be deleted and created in same transaction +71. Check if CVL validation passes when two UPDATE operation are given +72. Check if CVL validation passes when 'leafref' points to a key in another table having multiple keys. +73. Check if CVL validation passes when 'leafref' points to non-key in another table. +74. Check if CVL validation passes when 'leafref' points to a key which is drived in predicate from another table in cascaded fashion. +75. Check if CVL validation passes when 'must' condition involves checking with fields having default value which are not provided in request a data. +76. Check if CVL validation passes when 'must' condition has predicate field/key value derived from another table using another predicate in cascaded fashion. +77. Check if CVL validation passes for 'when' condition present within a leaf/leaf-list. +78. Check if CVL validation passes for 'when' condition present in choice/case node. +79. Check if CVL validation passes if 'max-elements' is present in a YANG list. +80. Check if CVL can sort the list of given tables as per their dependency imposed by leafref. +81. Check if CVL can return sorted list of tables in a given YANG module based on leafref relation. +82. Check if CVL can return the list of dependent tables for a given table in a YANG module. +83. Check if CVL can return the dependent data(Redis keys) to be deleted or modified for a given entry getting deleted. + +## 11 Appendix A + +Following are the list of Open source tools used in Management framework + +1. [Gorilla/mux](https://github.com/gorilla/mux) +2. [Go datastructures](https://github.com/Workiva/go-datastructures/tree/master/queue) +3. [OpenAPI](https://OpenAPI.io) +4. [gNMI client](https://github.com/jipanyang/gnxi) +5. [goyang](https://github.com/openconfig/goyang) +6. [YGOT](https://github.com/openconfig/ygot/ygot) +7. [GO Redis](https://github.com/go-redis/redis) +8. [Logging](https://github.com/google/glog) +9. [Profiling](https://github.com/pkg/profile) +10. [Validation](https://gopkg.in/go-playground/validator.v9) +11. [JSON query](https://github.com/antchfx/jsonquery) +12. [XML query](https://github.com/antchfx/xmlquery) +13. [Sorting](https://github.com/facette/natsort) +14. [pyangbind](https://github.com/robshakir/pyangbind) +15. [libYang](https://github.com/CESNET/libyang) + + +## 12 Appendix B + +Following are the list of Open source libraries used in telemetry container. +Always refer to the [Makefile](https://github.com/Azure/sonic-telemetry/blob/master/Makefile) for the sonic-telemetry container for current package list. + + +1. [GRPC](https://google.golang.org/grpc) +2. [GNMI](https://github.com/openconfig/gnmi/proto/gnmi) +3. [Protobuf](https://github.com/golang/protobuf/proto) +4. [goyang](https://github.com/openconfig/goyang) +5. [GO Redis](https://github.com/go-redis/redis) +6. [YGOT](https://github.com/openconfig/ygot/ygot) +7. [Logging](https://github.com/google/glog) +8. [GO Context](https://golang.org/x/net/context) +9. [Credentials](https://google.golang.org/grpc/credentials) +10. [Validation](https://gopkg.in/go-playground/validator.v9) +11. [GNXI utils](https://github.com/google/gnxi/utils) +12. [Gorilla/mux](https://github.com/gorilla/mux) +13. [jipanyang/xpath](https://github.com/jipanyang/gnxi/utils/xpath) +14. [c9s/procinfo](https://github.com/c9s/goprocinfo/linux) +15. [Workkiva/Queue](https://github.com/Workiva/go-datastructures/queue) +16. [jipanyang/gnmi client](https://github.com/jipanyang/gnmi/client/gnmi) +17. [xeipuuv/gojsonschema](https://github.com/xeipuuv/gojsonschema) diff --git a/doc/mgmt/REST_DOC_GEN_HLD.md b/doc/mgmt/REST_DOC_GEN_HLD.md new file mode 100644 index 0000000000..dd43fbc2a6 --- /dev/null +++ b/doc/mgmt/REST_DOC_GEN_HLD.md @@ -0,0 +1,98 @@ +# The RESTCONF API document generator design +# High Level Design Document +### Rev 0.1 + +# Table of Contents + * [Revision](#revision) + * [About this Manual](#about-this-manual) + * [Scope](#scope) + * [Definitions/Abbreviation](#definitionsabbreviation) + * [Introduction](#introduction) + + * [1 Overview](#1-overview) + * [1.1 Design Diagram](#11-system-chart) + * [1.2 Modules description](#12-modules-description) + * [1.2.1 Pyang](#121-pyang) + * [1.2.2 OpenAPI plugin](#122-openapi-plugin) + * [1.2.3 Markdown formatter](#123-markdown-formatter) + * [2 Requirements](#2-requirements) + * [3 Document Generation Approach](#3-document-generation-approach) + * [3.1 Auto Generated document](#31-auto-generated-document) + * [3.2 Manually written template document](#32-manually-written-template-document) + * [3.3 Final Document](#33-final-document) + +###### Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 01/08/2020 | Mohammed Faraaz | Initial version | + +# About this Manual +This document provides general information about the design of the [RESTCONF 8040](https://tools.ietf.org/html/rfc8040) API document generator and the REST document syntax. + +# Scope +This document describes only the high level design of the [RESTCONF 8040](https://tools.ietf.org/html/rfc8040) API document related components. It is intending to provide high level details about Pyang and its associated OpenAPI plugin. + +# Introduction +The Existing OpenAPI plugin for Pyang is enhanced to generate a document for RESTCONF API, it uses the same Utils and tools used by OpenAPI Plugin. The generated document will be in markdown(.md) format. + +# Definitions/Abbreviation + +| Definitions/Abbreviation | Description | +|--------------------------|--------------------------------------------| +| REST | Representational state transfer | +| API | Application Programmable Interface | + +# 1 Overview + +## 1.1 Design + +Following diagram describes a RESTCONF API document generator design + +![Design](https://docs.google.com/drawings/d/1mIejWXE6kqnxcN933_VMDKgtIeyobT0sw-w0DErkRHQ/edit?folder=0AM8ib1ydVn2cUk9PVA) + +Note: +Temporary using G-link above will change it actual file name after review. + +## 1.2 Modules description + +### 1.2.1 Pyang + +Pyang is an open source Yang Parser written in Python, It parses the yang modules and provides yang data in a programmable object (python objects) format, this data is passed on to the OpenAPI plugin for OpenAPI spec and RESTCONF document generation. + +### 1.2.2 OpenAPI Plugin + +This is an extension plugin written on top of Pyang parser, it walks over the generated Yang data (present in a form of Python object) and generates a yang tree along with required metadata for the generation of OpenAPI spec (swagger spec 2.0) and a RESTCONF document for each yang module in a markdown (.md) format. + +### 1.2.3 Markdown Formatter + +It is an extension/module inside OpenAPI plugin, it walks over the yang tree produced and uses the same Utils and metadata generated by OpenAPI generator and generates a REST document in a markdown(.md) Format. + +# 2 Requirements + +* Tool shall generate RESTCONF compatible URIs and payload. +* Tool shall output the Request and Response payload for the URIs in a JSON format. +* Tool shall fetch the Description from the yang and describe the URIs in a generated document +* Tool shall list and describe the Parameters which will be part of URIs +* Tool shall generate a RESTCONF API document in a markdown(.md) format. +* Tool shall as part of build and can be turned off using an option in Makefile. +* Tool shall generate documents for both Standard and Sonic yang models. + +# 3 Document Generation Approach + +## 3.1 Auto Generated document + +The RESTCONF document generation tool will generate one markdown (.md) file per Yang module. +These auto-generated documents will be placed under sonic-mgmt-framework/build/restconf_md directory. +The auto-generated documents will have following sections +* Configuration APIs - Describes Configuration URIs i.e. config:true Yang statement. +* Operational-state APIs - Describes Operational-state URIs i.e. config:false Yang statement. +* Operations APIs - Describes RPCs defined in yang model. + +## 3.2 Manually written template document + +The base document known as RESTCONF_documentation_index.template is a manually written jinja based markdown document which contains a static sections describing RESTCONF protocol itself, authentication, supported media types, RESTCONF capabilities, Response messages, protocol and transport requirements. This is a checked-in file. + +## 3.3 Final Document + +The Final RESTCONF document called RESTCONF_documentation_index.md is auto-generated during build time which contains contents of static file described in above [3.2](#32-manually-written-template-document) and the hyperlinks to the RESTCONF document for individual Yang models generated in section [3.1](#31-auto-Generated-document). + diff --git a/doc/mgmt/SONiC Management Framework Show Techsupport HLD.md b/doc/mgmt/SONiC Management Framework Show Techsupport HLD.md new file mode 100644 index 0000000000..83d2aa0323 --- /dev/null +++ b/doc/mgmt/SONiC Management Framework Show Techsupport HLD.md @@ -0,0 +1,298 @@ +# Show techsupport +Diagnostic information aggregated presentation +# High Level Design Document +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 10/06/2019 | Kerry Meyer | Initial version | + +# About this Manual +This document provides general information about presentation of aggregated diagnostic information for the SONiC subsystem via the Management Framework infrastructure. +# Scope +This document describes the high level design for the "show techsupport" command implementation under the control of the Management Framework infrastructure. It is intended to cover the general approach and method for providing a flexible collection of diagnostic information items and a mechanism for adding new items to the list of information to be provided. It also considers the basic mechanisms to be used for the various types of information to be aggregated. It does not address specific details for collection of all supported classes of information. + +# Definition/Abbreviation + +# 1 Feature Overview + +Provide Management Framework functionality to process the "show techsupport" command: + - Create an aggregated file containing the information items needed for analysis and diagnosis of problems occurring during switch operations. + - Support reduction of aggregated log file information via an optional "--since" parameter specifying the desired logging start time. + +NOTE: The underlying feature for which this Management Framework feature provides "front end" client interfaces is unchanged by the addition of these interfaces. (The "since " option available through these interfaces, however, is restricted to the IETF/YANG date/time format.) Please refer to the following document for a description of the "show techsupport" base feature: + +https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md#troubleshooting-commands + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide a Management Framework based interface for the "show tech-support" command. + +### 1.1.2 Configuration and Management Requirements +Provide the ability to invoke the command via the following client interfaces: + + - Management Framework CLI (same syntax as the existing Click-based + API except for tighter restriction of the "DateTime" format to + conform with the Yang/IETF DateTime standard) + - REST API + - gNOI + +(See Section 3 for additional details.) + +### 1.1.3 Scalability Requirements + +Time and storage space constraints: The large number of information items collected and the potentially large size of some of the items (e.g. interface information display in a large system) present an exposure to the risk of long processing times and significant demands on disk storage space. The Management Framework interface invokes the same command used for the Click-based interface. It adds no significant additional overhead or processing time. The storage space requirements are unchanged. +### 1.1.4 Warm Boot Requirements +N/A +## 1.2 Design Overview +### 1.2.1 Basic Approach +This feature will be implemented using the Management Framework infrastructure supplemented with customized access mechanisms for handling "non-DB" data items. + +### 1.2.2 Container +The user interface (front end) portion of this feature is implemented within the Management Framework container. + +### 1.2.3 SAI Overview +N/A (non-hardware feature) + +# 2 Functionality +## 2.1 Target Deployment Use Cases +This feature provides a quick and simple mechanism for network administrators or other personnel with no detailed knowledge of switch internal details to gather an extensive set of information items by using a single command. These items provide critical information to help development and sustaining engineering teams in the analysis and debugging of problems encountered in deployment and test environments. +## 2.2 Functional Description +The set of items to be gathered for a given software release is defined by the development team. It is specified in a way that enables run-time access to the desired set of information items to be collected. The definition of the set of information items to be collected includes specification of the access function to be used for each item in the list to gather the information, format it as needed, and pack it into the output file. The location of the resulting output file is provided to the requesting client at the completion of command execution. + +The output file name has the following form: + +` +/var/dump/sonic_dump_sonic_YYYYMMDD_HHMMSS.tar.gz +` + +Example: + +`/var/dump/sonic_dump_sonic_20191118_221625.tar.gz +` +See section 3.6.2.2 for an explanation of the output file name format. + +To view the contents of the file, the user must copy it to a local file in the client file system. If the file is to be extracted within the directory to which it is copied, the directory should have at least 50 MB of available space. To extract the file inside of the directory to which it has been copied while displaying a list of output files, the following command can be used: + +` +tar xvzf filename.tar.gz +` +The files are extracted to a directory tree, organized based on the type of information contained in the files. Example file categories for which sub-directories are provided in the output file tree include: + +- log files ("log" directory ) +- Linux configuration files ("etc" directory) +- generic application "dump" output ("dump" directory) +- network hardware driver information ("sai" directory) +- detailed information on various processes ("proc" directory). + +To extract the file contents to an alternate location, the following form of the "tar" command can be used: + +` + tar xvzf filename.tar.gz -C /path/to/destination/directory +` +Some of the larger "extracted" files will be compressed in gzip format. This includes log files and core files and also includes other files containing a large amount of output (e.g. a dump of all BGP tables). These files have a ".gz" file type. They can be extracted using: + +` +gunzip +` + + +# 3 Design +## 3.1 Overview +The "show techsupport" command causes invocation of an RPC sent from the management framework to a process in the host to cause collection of a list of flexibly defined sets of diagnostic information (information "items"). The collected list of items is stored in a compressed "tar" file with a unique name. The command output provides the location of the resulting compressed tar file. + +The "since" option can be used, if desired, to restrict the time scope for log files and core files to be collected. This option is passed to the host process for use during invocation of the applicable information gathering sub-functions. + +## 3.2 DB Changes +N/A +### 3.2.1 CONFIG DB +### 3.2.2 APP DB +### 3.2.3 STATE DB +### 3.2.4 ASIC DB +### 3.2.5 COUNTER DB + +## 3.3 Switch State Service Design +N/A +### 3.3.1 Orchestration Agent +### 3.3.2 Other Process +The "show techsupport" feature requires RPC support in a process running within the host context. The host process handling the RPC is responsible for dispatching "show techsupport" requests from the management framework container to trigger allocation of an output file, gathering and packing of the required information into the output file, and sending a response to the management framework RPC agent to specify the name and path of the output file. + +## 3.4 SyncD +N/A + +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models +The following Sonic Yang model is used for implementation of this feature: + +```module: sonic-show-techsupport + + rpcs: + +---x sonic-show-techsupport-info + +---w input + | +---w date? yang:date-and-time + +--ro output + +--ro output-filename? string +``` + + + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands +N/A +#### 3.6.2.2 Show Commands + +Command syntax summary: + +` +show techsupport [since ] +` + +Command Description: + +Gather information for troubleshooting. Display the name of a file containing the resulting group of collected information items in a compressed "tar" file. + + +Syntax Description: + +| Keyword | Description | +|:--------------|:----------- | +| since | This option uses a text string containing the desired starting Date/Time for collected log files and core files. The format of the Date/Time in the string is defined by the Yang/IETF date-and-time specification (REF http://www.netconfcentral.org/modules/ietf-yang-types, based on http://www.ietf.org/rfc/rfc6020.txt). If "since is specified, this value is passed to the host process for use during invocation of the applicable log/core file gathering sub-functions.| + +Command Mode: User EXEC + +Output format example and summary: + +``` +Example: + +Output stored in: /var/dump/sonic_dump_sonic_20191008_082312.tar.gz + +-------------------------------------------------- + +Output file name sub-fields are defined a follows: + +- YYYY = Year +- MM = Month (numeric) +- DD = Day of the Month +- HH = hour of the current time (based on execution of the Linux "**date**" command) at the start of command execution +- MM = minute of the current time (based on execution of the Linux "**date**" command) at the start of command execution +- SS = second of the current time (based on execution of the Linux "**date**" command) at the start of command execution +``` + +Command execution example (basic command): + +``` +sonic# show techsupport + +Output stored in: /var/dump/sonic_dump_sonic_20191008_082312.tar.gz + +``` +Command execution Example (using the "since" keyword/subcommand): + +``` +sonic# show tech-support + since Collect logs and core files since a specified date/time + | Pipe through a command + + +sonic# show tech-support since + String date/time in the format: + + "YYYY-MM-DDTHH:MM:SS[.ddd...]Z" or + "YYYY-MM-DDTHH:MM:SS[.ddd...]+hh:mm" or + "YYYY-MM-DDTHH:MM:SS[.ddd...]-hh:mm" Where: + + YYYY = year, MM = month, DD = day, + T (required before time), + HH = hours, MM = minutes, SS = seconds, + .ddd... = decimal fraction of a second (e.g. ".323") + Z indicates zero offset from local time + +/- hh:mm indicates hour:minute offset from local time + +sonic# show tech-support since 2019-11-27T22:02:00Z +Output stored in: /var/dump/sonic_dump_sonic_20191127_220334.tar.gz +``` +Command execution example invocation via REST API: + +``` +REST request via CURL: + +curl -X POST "https://10.11.68.13/restconf/operations/sonic-show-techsupport:sonic-show-techsupport-info" -H "accept: application/yang-data+json" -H "Content-Type: application/yang-data+json" -d "{ \"sonic-show-techsupport:input\": { \"date\": \"2019-11-27T22:02:00.314+03:08\" }}" + +Request URL: + +https://10.11.68.13/restconf/operations/sonic-show-techsupport:sonic-show-techsupport-info + +Response Body: + +{ + "sonic-show-techsupport:output": { + "output-filename": "/var/dump/sonic_dump_sonic_20191128_013141.tar.gz" + } +} +``` + +Command execution example invocation via gNOI API: + +``` +root@sonic:/usr/sbin# ./gnoi_client -module Sonic -rpc showtechsupport -jsonin "{\"input\":{\"date\":\"2019-11-27T22:02:00Z\"}}" -insecure +Sonic ShowTechsupport +{"sonic-show-techsupport:output":{"output-filename":"/var/dump/sonic_dump_sonic_20191202_194856.tar.gz"}} +``` + +NOTE: See section 3.6.1 for a description of the limitations of the current implementation. A supplementary capability to transfer the tech support file and other diagnostic information files to the client via the Management Framework interface is highly desirable for a future release. + +#### 3.6.2.3 Debug Commands +N/A +#### 3.6.2.4 IS-CLI Compliance +The current implementation differs from the IS-CLI. + + Instead of dumping the technical support information to the output buffer, this implementation dumps the information to a compressed "tar" file and sends the name of the file to the output buffer. This implementation matches the current SONiC implementation. A supplementary capability to enable transfer of the specified file to the client is highly desirable for full functionality of this command when using a REST API or gNMI/gNOI client interface. Without this capability, it is necessary to open a shell on the host and use the SONiC host CLI interface to transfer the file. + +### 3.6.3 REST API Support +REST API support is provided. The REST API corresponds to the SONiC Yang model described in section 3.6.1. + +# 4 Flow Diagrams + +# 5 Error Handling +N/A + +# 6 Serviceability and Debug +Any errors encountered during execution of the "show tech-support" command that prevent retrieval or saving of information are reported in the command output at completion of the operation. + +# 7 Warm Boot Support +N/A + +# 8 Scalability +Refer to section 1.1.3 + +# 9 Unit Test + +| Case | Trigger | Result | +|:-----------|:--------|:-------| +| Basic command execution | Execute the "show techsupport" command with no parameters. | Confirm that the command is accepted without errors and a "result" file name is returned. Confirm that the result file contains the expected set of items. (Examine/expand the contents of the file to ensure that the top level directory tree is correct and that the number of sub-files within the tar file is correct.)| +"since" option (postive test case) | Execute the command with the "--since" TEXT option with a valid date string specifying a time near the end of one of the unfiltered output from the first test.| Same as the "Basic command execution" case. Additionally, confirm that the expected time filtering has occurred by examining one of the affected sub-files.| +"since" option (negative test case #1)|Execute the command with the "--since" TEXT option with an invalid date string.|Verify that an error is returned.| +"since" option (negative test case #2)|Execute the command with the "--since" TEXT option with no date string.|Verify that an error is returned.|Execute the command with the "--since" option with no date string.| Verify that an error is returned.| + + + + + +# 10 Internal Design Information +N/A diff --git a/doc/mgmt/SONiC-Management-Framework-Image-mgmt-HLD.md b/doc/mgmt/SONiC-Management-Framework-Image-mgmt-HLD.md new file mode 100644 index 0000000000..b2bfff5f0e --- /dev/null +++ b/doc/mgmt/SONiC-Management-Framework-Image-mgmt-HLD.md @@ -0,0 +1,250 @@ +# Image Management +Software upgrade and image installation support for SONiC +# High Level Design Document +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +| :--: | :--------: | :---------------: | ------------------ | +| 0.1 | 09/13/2019 | Arunsundar Kannan | Initial version | + +# About this Manual +This document provides general information about the Image management and installation feature implementation in SONiC. +# Scope +Covers northbound interface for the Image Management feature, as well as unit test cases. + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +| ---------------- | ------------------------------ | +| Image Management | Image installation, validation | + +# 1 Feature Overview + +Provide management framework capabilities to handle: + +- Image Installation +- List available images +- Removal of an available image +- Set default image for next boot + + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide management framework support to existing SONiC capabilities with respect to Image Management + +### 1.1.2 Configuration and Management Requirements +- CLI configuration and show commands +- REST API support +- gNMI Support + +### 1.1.3 Scalability Requirements +N/A +### 1.1.4 Warm Boot Requirements +N/A +## 1.2 Design Overview +### 1.2.1 Basic Approach +Implement Image Management support using translib in sonic-mgmt-framework. +### 1.2.2 Container +Management Container + +### 1.2.3 SAI Overview +N/A + +# 2 Functionality +## 2.1 Target Deployment Use Cases + +**image install** + +This command is used to install a new image on the alternate image partition. This command takes a path to an installable SONiC image or URL and installs the image. + +**show image-list** + +This command displays information about currently installed images. It displays a list of installed images, currently running image and image set to be loaded in next reboot. + +**image set-default** + +This command is be used to change the image which can be loaded by default in all the subsequent reboots. + +**image remove** + +This command is used to remove the unused SONiC image from the disk. Note that it's *not* allowed to remove currently running image. + +## 2.2 Functional Description +After recieving the request from the client, via an RPC, the REST server will transfer the control to processAction method in the app module(inside trasformer). This method will parse the target uri path and will branch to the corresponding function. These functions will call the python scripts in the host to perform image management related actions, like install, remove ..etc. The response from the output of the script is propagated back to processAction method and is converted to json. The json message is sent back to the client. + +# 3 Design +## 3.1 Overview +Enhancing the management framework backend code and transformer methods to add support for Image management + +## 3.2 DB Changes +State DB + +### 3.2.1 CONFIG DB + +N/A + +### 3.2.2 APP DB + +N/A + +### 3.2.3 STATE DB +The State DB is populated with details regarding the currently used image, image to be in the next boot and the list of available images. + + +### 3.2.4 ASIC DB + +N/A + +### 3.2.5 COUNTER DB + +N/A + +## 3.3 Switch State Service Design +### 3.3.1 Orchestration Agent + +N/A + +### 3.3.2 Other Process +N/A + +## 3.4 SyncD +N/A + +## 3.5 SAI +N/A + +## 3.6 User Interface +### 3.6.1 Data Models + +``` +module: sonic-image-management + +--rw sonic-image-management + +--rw IMAGE_GLOBAL + | +--rw IMAGE_GLOBAL_LIST* [img_key] + | +--rw img_key enumeration + | +--rw current? string + | +--rw next-boot? string + +--rw IMAGE_TABLE + +--rw IMAGE_TABLE_LIST* [image] + +--rw image string + + rpcs: + +---x image-install + | +---w input + | | +---w imagename? filename-uri-type + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string + +---x image-remove + | +---w input + | | +---w imagename? string + | +--ro output + | +--ro status? int32 + | +--ro status-detail? string + +---x image-default + +---w input + | +---w imagename? string + +--ro output + +--ro status? int32 + +--ro status-detail? string + +``` + +### 3.6.2 CLI +#### 3.6.2.1 Configuration Commands + +**image install** + +``` +sonic# image install https://sonic-jenkins.westus.cloudapp.azure.com/job/xxxx/job/buildimage-xxxx-all/xxx/artifact/target/sonic-xxxx.bin + +Done +``` + +**image set-default** + +``` +sonic# image set-default SONiC-OS-HEAD.XXXX +``` + +**image remove** + +``` +sonic# image remove SONiC-OS-HEAD.YYYY + +Image removed +``` + + + +#### 3.6.2.2 Show Commands + +**show image-list** + +``` +sonic# show image-list +Current: SONiC-OS-HEAD.XXXX +Next: SONiC-OS-HEAD.XXXX +Available: +SONiC-OS-HEAD.XXXX +SONiC-OS-HEAD.YYYY +``` + + +#### 3.6.2.3 Debug Commands + +N/A + +#### 3.6.2.4 IS-CLI Compliance + +N/A + +### 3.6.3 REST API Support +* get_sonic_image_management_sonic_image_management +* rpc_sonic_image_management_image_install +* rpc_sonic_image_management_image_remove +* rpc_sonic_image_management_image_default + +# 4 Flow Diagrams +N/A + +# 5 Error Handling + +TBD + +# 6 Serviceability and Debug + +TBD + +# 7 Warm Boot Support + +TBD + +# 8 Scalability +N/A + +# 9 Unit Test +List unit test cases added for this feature including warm boot. + +| Test Name | Test Description | +| :------ | :----- | +| Image install | Image installed successfully and an entry is added in grub.cfg | +| Image remove | Image is removed and the corresponding entry is removed from grub.cfg | +| Image set default | Image is set as zeroth entry(entry for the default image) in grub.cfg | +| Show image list | Image list shows all the entries present in grub.cfg | +# 10 Internal Design Information + diff --git a/doc/mgmt/SONiC_AliasFeature_HLD.md b/doc/mgmt/SONiC_AliasFeature_HLD.md new file mode 100644 index 0000000000..7436c2af7c --- /dev/null +++ b/doc/mgmt/SONiC_AliasFeature_HLD.md @@ -0,0 +1,411 @@ +# Interface-Naming Feature + +Implement alternative interface naming convention for ethernet interfaces via CLI/REST/gNMI in SONiC management framework. + +# High Level Design Document + +#### Rev 0.1 + +# Table of Contents + +- [List of Tables](#list-of-tables) +- [Revision](#revision) +- [About This Manual](#about-this-manual) +- [Scope](#scope) +- [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables + +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision + +| Rev | Date | Author | Change Description | +| ---- | ---------- | ------------ | ------------------ | +| 0.1 | 05/01/2020 | Justine Jose | Initial draft | +| 0.2 | 05/29/2020 | Justine Jose | v1.0 | | +| | | | | + +# About this Manual + +This document provides general information about interface-naming related syntax & overview for ethernet interfaces in break out & non-breakout mode. + +# Scope + +Covers general design for supporting interface-naming feature. Scope of interface-naming feature is restricted to SONiC management framework. SONiC CLICK based support is unaffected. Setttig interface-naming to standard in config_db.json is mentioned in the ZTP section. + +# Definition/Abbreviation + +### Table 1: Abbreviations + +# 1 Feature Overview + +The document covers various interfaces for interface-naming feature using SONiC management framework. Currently in SONiC, the ethernet interfaces are shown as a flat number. This flat interface numbering does not give mapping information about actual ports that are mapped to the interface number. By setting interface-naming to standard, user has better usability & mapping for actual port numbers. + +## 1.1 Requirements + +### 1.1.1 Functional Requirements + +Provide ability to have an alternative configuration for Ethernet Interface using SONiC management framework. + +### 1.1.2 Configuration and Management Requirements + +- Two ways to set the `interface-naming`. They are classified as `standard` and `native`. +- Implement interface-naming related config & show commands. +- REST & gNMI set/get support for interface-naming. + +### 1.1.3 Scalability Requirements + +### 1.1.4 Warm Boot Requirements + +## 1.2 Design Overview + +### 1.2.1 Basic Approach + +- By default system will boot in native interface-naming. Native interface-naming is where interfaces are displayed as Ethernet 1, Ethernet 4, Ethernet 8 etc. This also ensures that backwards compatibility is maintained. +- Once interface naming is set to standard, all subsequent CLI, REST & gNMI interfaces will use the alternative name for ethernet interface configuration & retrieval. +- Existing SSH sessions:- All the existing sessions where change in interface-naming mode is required will be notified to restart their existing sonic-cli sessions. This will be supported through `wall`command. + +### 1.2.2 Container + +### 1.2.3 SAI Overview + +# 2 Functionality + +## 2.1 Target Deployment Use Cases + +## 2.2 Functional Description + +# 3 Design + +## 3.1 Overview + +- For dynamically using the right string for ethernet interfaces (i.e Ethernet 4 in native mode & Eth 1/4 in standard mode), XML file has to be modified to use the correct PTYPE. `PHY_INTERFACE` has to be used as PTYPE for Ethernet Interface. +- When interface-naming is set to standard, ethernet interfaces will be represented as Eth[slot/port/breakout-port], where slot will start from 1 and port/breakout-port numbers will start from 1. Slot will always likely to be 1 in fixed pizza-box format, but may take different values in chassis format. This name format is fixed and will not have references to interface speed. For e.g Eth1/1 (in non breakout mode), Eth1/1/1 (in breakout mode). +- The standard interface name {e.g Eth1/1) is picked from platform.json file. All platforms need to be updated with this format for port name. +- For Ethernet interfaces, (i.e Ethernet 4 in native interface-naming & Eth 1/4 in standard interface-naming), XML file has to be modified to use the correct PTYPE. `PHY_INTERFACE` has to be used as PTYPE for Ethernet interface. +Example: +``` + +``` + +- Annotations to be added for interface-naming + + What App-owners need to do? + + *1. Add “value-transformer” annotation for the leaves & keys that need special handling, in the sonic-yang annot file.* + *2. Write value-transformer callback api* + + Actions that Xfmr-infra will do: + *1. Process dbDataMap & invoke value-transformer, if present.* + + + **Use case:** + +“value-transformer” (callback_xfmr) annotated for a leaf(leaf-A) in a sonic-yang(assume sonic-A.yang), will be invoked for that leaf(leaf-A). + +If leaf-A in sonic-A.yang is referred from multiple other yang-leaves(assume leaf-B in sonic-B.yang & leaf-C in sonic-C.yang has leaf-ref to leaf-A in sonic-A.yang), then the “callback_xfmr” will be inherited & invoked for these leaves(leaf-B & C) as well, without any explicit annotation(i.e. no “value-transformer” annotation is needed for leaf-B or leaf-C). + + User can override inherited value-transformer, by annotating a new value-xfmr at that level. i.e. If an explicit value-transformer(“callback_NEW_xfmr”) is annotated for leaf-B in sonic-B.yang(which has leaf-ref to leaf-A), then “callback_NEW_xfmr” will be invoked for leaf-B. + + + + E.g. For interface-naming feature, “/sonic-port/PORT/PORT_LIST/ifname” is annotated with value-transformer ("**alias_value_xfmr**"). This api("**alias_value_xfmr**") will be invoked for `/sonic-port/PORT/PORT_LIST/ifname`, as well as for all the yang-leaves, which has leaf-ref to this (like `/sonic-acl/ACL_TABLE/ACL_TABLE_LIST/ports`, `/sonic-udld/UDLD_PORT/UDLD_PORT_LIST/ifname`, `/sonic-mirror-session/MIRROR_SESSION/MIRROR_SESSION_LIST/dst_port` etc.). + + **Sample YANG annotation**: + +[sonic-port-annot.yang](https://github.com/project-arlo/sonic-mgmt-framework/blob/master/models/yang/annotations/sonic-port-annot.yang) + ``` + deviation /prt:sonic-port/prt:PORT/prt:PORT_LIST/prt:ifname { + deviate add { + sonic-ext:value-transformer "alias_value_xfmr"; + } + } + ``` + + **API signature:** + + ​ `func alias_value_xfmr(inParams XfmrDbParams) (string, error)` + + + + **XfmrDbParams contents** +``` + type XfmrDbParams struct { + db db.DBNum + oper int + tableName string + key string + field string + fieldYangType yang.TypeKind + value string + } +``` + + **Below is the data flow** + + ***CRU operation (OC & Sonic)***: + +| **Existing implementation** | **New implementation** | +| ------------------------------------------------------------ | :----------------------------------------------------------- | +| For any CRU request that reaches Xfmr, xfmr-infra will process payload & invoke overloaded functions (i.e fieldxfmr, subtreexfmr…), if present. It then creates dbDataMap and performs DB operations on dbDataMap | For any CRU that reaches Xfmr, Xfmr-infra will process payload & invoke any overloaded functions if present (i.e fieldxfmr, subtreexfmr…). It then creates dbDataMap & invokes value-transformer( when present), process dbDataMap and create a new processed-dbDataMap· DB operations are now performed on processed-dbDataMap. | + + + + ***GET operation(OC & Sonic)*** + +| **Existing implementation** | **New implementation** | +| ------------------------------------------------------------ | ------------------------------------------------------------ | +| For GET request that reaches Xfmr, Xfmr-infra will read data from db and create dbDataMap. It then invokes overloaded functions(fieldxfmr, subtreexfmr…) with dbDataMap. Fills the ygot tree with data from dbDataMap & merges it with ygot-tree from subtree-xfmr and sends response back. | For GET request that reaches Xfmr, xfmr-infra will read data from db and create dbDataMap. First value-xfmr (if present) is invoked, dbDataMap is processed and a new processed-dbDataMap is created. Overloaded functions(fieldxfmr, subtreexfmr…) with processed-dbDataMap will be invoked next. Fills the ygot tree with data from dbDataMap & merges it with ygot-tree from subtree-xfmr and sends response back. | + + **Sub-tree Transformer**: In cases where applications use subtree transformer, then need to use an API that will return the standard interface-naming string. The API will internally figure out if interface-naming is set to standard or native. + + Following are the APIs used: + These APIs are available in `translib/utils` package. + + **GetNativeNameFromUIName** + `func GetNativeNameFromUIName(ifName *string) *string` + + Retrieves native interface name from user input name, if interface-naming is set to standard. User input name can be standard or native. API will provide native interface name if the user input is standard interface name, otherwise it will return the native interface name passed to the API. If interface-naming is native, API will return the input string passed. + + **GetUINameFromNativeName** + `func GetUINameFromNativeName(ifName *string) *string` + + Retrieves standard interface name from user input name, if interface-naming is set to standard. User input name can be standard or native. API will provide standard interface name if the user input is native interface name, otherwise it will return the standard interface name passed to the API. If interface-naming is native, API will return the input string passed. + +## 3.2 DB Changes +DB references to ports are unchanged, native name will be used. + +### 3.2.1 CONFIG DB +Interface-naming configuration is stored in the `DEVICE_METADATA` table as an entry `intf_naming_mode` with value as `standard` or `native`. +``` +127.0.0.1:6379[4]> hgetall DEVICE_METADATA|localhost + 1) "hwsku" + 2) "Force10-S6000" + 3) "type" + 4) "LeafRouter" + 5) "hostname" + 6) "sonic" + 7) "platform" + 8) "x86_64-dell_s6000_s1220-r0" + 9) "mac" +10) "90:b1:1c:f4:ab:da" +11) "intf_naming_mode" +12) "standard" +``` + +#### 3.2.1.1 + +### 3.2.2 APP DB + +### 3.2.3 STATE DB + +### 3.2.4 ASIC DB + +### 3.2.5 COUNTER DB + +## 3.3 Switch State Service Design + +### 3.3.1 Orchestration Agent + +### 3.3.2 Other Process + +## 3.4 SyncD + +## 3.5 SAI + +## 3.6 User Interface + +### 3.6.1 Data Models +YANG model used for interface-naming handling +[sonic-device-metadata.yang](https://github.com/project-arlo/sonic-mgmt-framework/blob/master/models/yang/sonic/sonic-device-metadata.yang) + +Supported YANG objects and attributes are highlighted in green: +```diff +module: sonic-device-metadata + +--rw sonic-device-metadata + +--rw DEVICE_METADATA + +--rw DEVICE_METADATA_LIST* [name] + +--rw name string ++ +--rw intf_naming_mode? enumeration +``` +*Note*: intf_naming_mode takes enum value `native` or `standard`. + +### 3.6.2 CLI + +#### 3.6.2.1 Configuration Commands + +#### Interface naming configuration + +`[no] interface-naming standard` +*Note*: Interface naming can be set to standard or native using the above command in `CONFIG` mode. Once the mode is set, please logout and login to enter the right interface-naming session. +``` +sonic(config)# interface-naming standard +sonic(config)# no interface-naming standard +``` +#### Config commands when interface naming is standard +`E1/2`, `e1/2`, `E 1/2`, `e 1/2`, `Et 1/2`, `et1/2`, `Eth 1/2`, `Eth1/2` options are supported to get into standard interface naming. The parser converts all to `Eth1/2`. Native interface name is considered to be an invalid output. + +``` +sonic(config)# interface e1/2 +sonic(conf-if-Eth1/2)# +``` +``` +sonic(conf-if-Eth1/2)# ip address 2.2.2.2/24 +``` + +#### 3.6.2.2 Show Commands + +#### Display interface naming + +`show interface-naming` +``` +sonic(config)# interface-naming standard +sonic# show interface-naming +Interface naming is set to standard +``` +``` +sonic(config)# no interface-naming standard +sonic# show interface-naming +Interface naming is set to native +``` +#### Show commands when interface-naming is standard +``` +show vlan +Q: A - Access (Untagged), T - Tagged +NUM Status Q Ports +2 Inactive A Eth1/3 + T Eth1/5/2 + T Eth1/10 +``` +``` +sonic# show interface status +------------------------------------------------------------------------------------------------------------------- +Name Description Admin Oper Speed MTU Alternate Name +------------------------------------------------------------------------------------------------------------------- +Eth1/1 - down down 40000 9100 Ethernet0 +Eth1/2 - down down 40000 9100 Ethernet4 +... +... +``` +#### Show commands when interface-naming is native +``` +show vlan +Q: A - Access (Untagged), T - Tagged +NUM Status Q Ports +2 Inactive A Ethernet8 + T Ethernet18 + T Ethernet36 +``` +``` +sonic# show interface status +------------------------------------------------------------------------------------------------------------------- +Name Description Admin Oper Speed MTU Alternate Name +------------------------------------------------------------------------------------------------------------------- +Ethernet0 - down down 40000 9100 Eth1/1 +Ethernet4 - down down 40000 9100 Eth1/2 +... +... +``` + + +#### 3.6.2.3 Debug Commands + +#### 3.6.2.4 IS-CLI Compliance + +Cisco has no equivalent command. Port naming is always fixed. + +The following table maps SONiC CLI commands to corresponding IS-CLI commands. The compliance column identifies how the command comply to the IS-CLI syntax: + +- **IS-CLI drop-in replace** \u2013 meaning that it follows exactly the format of a pre-existing IS-CLI command. +- **IS-CLI-like** \u2013 meaning that the exact format of the IS-CLI command could not be followed, but the command is similar to other commands for IS-CLI (e.g. IS-CLI may not offer the exact option, but the command can be positioned is a similar manner as others for the related feature). +- **SONiC** - meaning that no IS-CLI-like command could be found, so the command is derived specifically for SONiC. + +| CLI Command | Compliance | IS-CLI Command (if applicable) | Link to the web site identifying the IS-CLI command (if applicable) | +| ----------- | ---------- | ------------------------------ | ------------------------------------------------------------ | +| | | | | + +### 3.6.3 REST API Support + +#### 3.6.3.1 + +##### Following REST operations will be supported + +**PATCH, PUT, DELETE and GET** + +- `​/sonic-device-metadata:sonic-device-metadata​/DEVICE_METADATA​/DEVICE_METADATA_LIST={name}​/intf_naming_mode` + +**REST query when interface-naming is set to Standard** +``` +curl -X GET "https:///restconf/data/openconfig-interfaces:interfaces/interface=Eth1%2F2" -H "accept: application/yang-data+json" -k -u "admin:admin" +``` +*Note: '/' part of interface name has to be replaced by %2F irrespective of whether the port is in breakout mode or not.* + + +# 4 Flow Diagrams + +# 5 Error Handling + +# 6 ZTP + +Setting interface-naming to standard in case of ZTP can be done by adding `intf_naming_mode` to DEVICE_METADATA table in the `config_db.json` file. Interface naming is set to native by default. +Example: +``` +DEVICE_METADATA": { + "localhost": { + "hostname": "sonic", + "hwsku": "Force10-S6000", + "mac": "90:b1:1c:f4:ab:da", + "platform": "x86_64-dell_s6000_s1220-r0", + "type": "LeafRouter", + "intf_naming_mode": "standard" + } + } +``` +# 7 Logging +SONiC backend uses native name irrespective of whether interface-naming is set to standard or native. + +# 8 Serviceability and Debug + +# 9 Warm Boot Support + +# 10 Scalability + +# 11 Unit Test and Automation + +The following test cases will be tested using CLI/REST/gNMI management interfaces. +#### Configuration and Show via CLI + +| Test Name | Test Description | +| :------ | :----- | +| Native interface-naming verification | Verify interface-naming is set to native | +| Set interface-naming to standard | Verify whether interface-naming is set to standard using show command | +| Standard interface-naming verification | Standard interface-naming can be verified using config and show commands involving physical interface whether standard interface-naming has taken effect | +| Set interface-naming to native | Verify whether interface-naming is set to native using show command | +| Native interface-naming verification | Native interface-naming can be verified using config and show commands involving physical interface whether native interface-naming has taken effect | +| Save and reload test | Save the config and reload the box, make sure that the system comes up with native interface-naming | + + +#### Configuration via gNMI + +Same as CLI configuration test, but using gNMI SET request + +#### Get configuration via gNMI + +Same as CLI show test, but using gNMI GET request, verify the JSON response. + +#### Configuration via REST + +Same as CLI configuration test, but using REST request + +#### Get configuration via REST + +Same as CLI configuration test, but using REST request + +#### Automation + +Spytest cases will be implemented for new CLI and APIs. diff --git a/doc/mgmt/SONiC_DNS_Support.md b/doc/mgmt/SONiC_DNS_Support.md new file mode 100644 index 0000000000..37e6b3d99b --- /dev/null +++ b/doc/mgmt/SONiC_DNS_Support.md @@ -0,0 +1,433 @@ +# Feature Name +Domain Name System (DNS) support + +# High Level Design Document + +#### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + +# List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 05/05 | Venkatesan Mahalingam | Initial version | +| 0.2 | 05/11 | Venkatesan Mahalingam | Added source interface support and addressed comments | + + +# About this Manual + +This document introduces the support of DNS in SONiC. + +# Scope + +This document covers the following, +1) Click commands to configure DNS server & Source interface +2) Click command to show DNS hosts +3) New tables in config DB to handle DNS configs +4) DNS table configuration and show based on OpenConfig YANG model +5) KLISH commands to configure DNS server & Source interface +6) KLISH commands to show DNS hosts +7) Backend support to add DNS configs in "/etc/resolv.conf" file & iptable rules to change the source-IP of the DNS query packets +8) Unit Testcases + +# Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| DNS | Domain Name System | + + + +# 1 Feature Overview + +DNS stands for Domain Name System. It is used to translate human readable domain names into machine readable IP addresses. + +With this feature, users will be able to configure DNS servers and source interface using SONiC Click commands and north bound interface (KLISH/REST/gNMI) provided by management framework module using OpenConfig models. + +Also, backend support to handle new config DB table events to populate nameservers in "/etc/resolv.conf" and add iptable rules to change the source IP (present on the source interface) of the DNS query that is being sent to the DNS server. + +## 1.1 Requirements + +### 1.1.1 Front end configuration and get capabilities + +#### 1.1.1.1 add/delete DNS server +This requirement is to add/delete DNS name server information in the Redis ConfigDB (DNS_SERVER) using Click and mgmt-framework. +The DNS server can be IPv4/IPv6 ipaddress and multiple DNS name servers can be configured. + +#### 1.1.1.2 add/delete DNS source interface +This requirement is to add/delete the global DNS source interface information in the Redis ConfigDB (DNS) using Click and mgmt-framework. + +Only one DNS source interface can be configured in the global DNS table. A new source interface will override the existing DNS source interface. + +#### 1.1.1.4 add/delete VRF name +No special handling is required in "/etc/resolv.conf" file to work on the management VRF (for this release). + +#### 1.1.1.5 Get DNS hosts +This displays the output of the DNS source interface and nameservers. + +### 1.1.2 Backend mechanisms to support configuration and get + +#### 1.1.2.1 add/delete DNS server +The creates or deletes a DNS server entry in the Redis ConfigDB. + +``` + "DNS_SERVER|10.11.0.1": { + "type": "hash", + "value": { + "NULL": "NULL" + } + }, + "DNS_SERVER|2001:aa:aa::a": { + "type": "hash", + "value": { + "NULL": "NULL" + } + }, + } +``` + +A change in the DNS_SERVER entry triggers hostcfgd to start the DNS configuration script, which in turn writes each DNS server to the resolv.conf and then restart the dns-config service. + +#### 1.1.2.2 add/delete DNS source interface + +This creates or deletes a global DNS source interface entry in the Redis ConfigDB. + +``` + "DNS|global": { + "type": "hash", + "value": { + "src_intf": "Loopback0" + } + } +``` + +A change in this entry triggers hostcfgd to add iptables SNAT rule incase of IPv4 address and ip6tables SNAT rule in IPv6 source address. + +Only one global DNS source interface is allowed. + +#### 1.1.2.4 get DNS hosts + +Transformer function issues get on DNS tables, parses the response and maps the outputs to the OpenConfig DNS states incase of mgmt-framework interface. + +Incase of Click commands, show commands directly fetches the information from DNS & DNS_SERVER tables and display to the user. + +### 1.1.3 Functional Requirements + +Provide management framework support to +- configure DNS name server +- configure DNS source interface + +### 1.1.4 Configuration and Management Requirements +- Click configuration and show commands +- CLI style configuration and show commands +- REST API support +- gNMI Support + +Details described in Section 3. + +### 1.1.6 Scalability Requirements + +### 1.1.7 Warm Boot Requirements + +## 1.2 Design Overview + +### 1.2.1 Basic Approach +- Implement Click commands using SONiC utilities. +- Implement DNS support using transformer in sonic-mgmt-framework. + +### 1.2.2 Container +The front end code change will be done in management-framework container including: +- XML file for the CLI +- Python script to handle CLI request (actioner) +- Jinja template to render CLI output (renderer) +- OpenConfig YANG model for DNS openconfig-system.yang +- SONiC DNS model for DNS based on Redis DB schema +- transformer functions to + * convert OpenConfig YANG model to SONiC YANG model for DNS related configurations + +### 1.2.3 SAI Overview + +# 2 Functionality + +## 2.1 Target Deployment Use Cases +Manage/configure DNS configurations via gNMI, REST and CLI interfaces + +## 2.2 Functional Description +Provide CLI, gNMI and REST support for DNS configurations. + +## 2.3 Backend change to support new configurations +Provide change in management framework and hostcfgd modules. + +# 3 Design + +## 3.1 Overview + +Enhancing the management framework backend code and transformer methods to add support for DNS. + +## 3.2 DB Changes + +### 3.2.1 CONFIG DB +``` +; DNS configuration attributes global to the system. Only one instance of the table exists in the system. +; Key +global_key = “global” ; DNS global configuration +; Attributes +src_intf = ifname ; source interface address for the outgoing DNS packets + +``` + +``` +; DNS name server configuration in the system. +; Key +server_key = IPAddress ; DNS server’s address +; Attributes +No attributes are introduced as part of this design. +``` + +### 3.2.2 APP DB + +### 3.2.3 STATE DB + +### 3.2.4 ASIC DB + +### 3.2.5 COUNTER DB + +## 3.3 Switch State Service Design + +### 3.3.1 Orchestration Agent + +### 3.3.2 Other Process + +## 3.4 SyncD + +## 3.5 SAI + +## 3.6 User Interface + +### 3.6.1 Data Models + +YANG models needed for DNS handling in the management framework: +1. **openconfig-system.yang** + +2. **sonic-system-dns.yang** + +Supported yang objects and attributes: +```diff + +module: openconfig-system + +--rw system + + +--rw dns + | +--rw config + | | +--rw search* oc-inet:domain-name + | | +--rw oc-sys-ext:source-intf? -> /oc-if:interfaces/interface/name + | +--ro state + | | +--ro search* oc-inet:domain-name + | | +--ro oc-sys-ext:source-intf? -> /oc-if:interfaces/interface/name + | +--rw servers + | | +--rw server* [address] + | | +--rw address -> ../config/address + | | +--rw config + | | | +--rw address? oc-inet:ip-address + | | | +--rw port? oc-inet:port-number + | | +--ro state + | | +--ro address? oc-inet:ip-address + | | +--ro port? oc-inet:port-number + +The above "source-intf" & "address" fields are supported in the scope of this work. + +module: sonic-system-dns + +--rw sonic-system-dns + +--rw DNS_SERVER + | +--rw DNS_SERVER_LIST* [ipaddress] + | +--rw ipaddress inet:ip-address + +--rw DNS + +--rw DNS_LIST* [type] + +--rw type enumeration ('global') + +--rw src_intf? union + +``` + +### 3.6.2 CLI + + +#### 3.6.2.1 Configuration Commands +All commands are executed in `configuration-view`: +``` +sonic# configure terminal +sonic(config)# +``` + +##### 3.6.2.1.1 Configure DNS server & source IP +``` +sonic(config)# ip name-server + source-intf Configure source interface to pick the source IP, used for the DNS query + A.B.C.D/A::B Domain name server + +sonic(config)# ip name-server source-intf + Ethernet Ethernet interface + Loopback Loopback interface + Management Management interface + PortChannel PortChannel interface + Vlan Vlan interface + +sonic(config)# ip name-server 10.11.0.1 + +sonic(config)# ip name-server 2001:aa:aa::a + +sonic(config)# ip name-server source-intf Loopback 0 + +``` + +##### 3.6.2.1.2 Delete DNS server & source IP + +``` +sonic(config)# no ip name-server + source-intf Configure source interface to pick the source IP, used for the DNS query + A.B.C.D/A::B Domain name server + +sonic(config)# no ip name-server 10.11.0.1 + +sonic(config)# no ip name-server 2001:aa:aa::a + +sonic(config)# no ip name-server source-intf + +``` +#### 3.6.2.2 Show dns hosts + +``` +sonic# show hosts +Source Interface : Loopback0 +Name servers are : 10.11.0.1, 2001:aa:aa::a +sonic# +``` + +#### 3.6.2.3 Debug Commands + +#### 3.6.2.4 IS-CLI Compliance + +### 3.6.3 REST API Support +``` +GET - Get existing DNS configuration information from CONFIG DB. + Get DNS peer states +POST - Add DNS configuration into CONFIG DB. +PATCH - Update existing DNS Configuration information in CONFIG DB. +DELETE - Delete a existing DNS configuration from CONFIG DB. +``` +### 3.6.4 Click command support +#### 3.6.4.1 Configuration support +Below click commands are supported for DNS configurations +oot@sonic:~# config dns +Usage: config dns [OPTIONS] COMMAND [ARGS]... + + DNS command line + +Options: + -?, -h, --help Show this message and exit. + +Commands: + add Specify a DNS name server + delete Delete a DNS name server + source_intf DNS source interface configurations +root@sonic:~# +root@sonic:~# config dns source_intf +Usage: config dns source_intf [OPTIONS] COMMAND [ARGS]... + + DNS source interface configurations + +Options: + -?, -h, --help Show this message and exit. + +Commands: + add DNS source interface add configuration + delete DNS source interface delete configuration +root@sonic:~# +``` +root@sonic:~# ping www.yahoo.com +ping: www.yahoo.com: Temporary failure in name resolution + +root@sonic:~# config dns add 10.11.0.1 + +root@sonic:~# ping www.yahoo.com +PING new-fp-shed.wg1.b.yahoo.com (98.137.246.8) 56(84) bytes of data. +64 bytes from media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8): icmp_seq=1 ttl=52 time=24.2 ms +64 bytes from media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8): icmp_seq=2 ttl=52 time=23.9 ms +64 bytes from media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8): icmp_seq=3 ttl=52 time=23.9 ms +^C +--- new-fp-shed.wg1.b.yahoo.com ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2001ms +rtt min/avg/max/mdev = 23.910/24.033/24.262/0.241 ms +root@sonic:~# +``` +#### 3.6.4.2 Show support +Below Click command is used to dump the DNS configurations +``` +root@sonic:~# show hosts +Source Interface : Loopback0 +Name servers are : 10.11.0.1, 2001:aa:aa::a +root@sonic:~# +``` +# 4 Flow Diagrams + +# 5 Error Handling + +# 6 Serviceability and Debug + +# 7 Warm Boot Support +This support is added in the hostcfgd and hence no explicit handling is needed. + +# 8 Scalability + +# 9 Unit Test + +The unit-test for this feature will include: +#### Configuration via CLI + +| Test Name | Test Description | +| :-------- | :----- | +| Configure DNS server | Verify DNS servers are present in the DNS_SERVER table (configDB) and the same is reflected in the resolv.conf file | +| Delete DNS server | Verify DNS servers are not present in the DNS_SERVER table (configDB) and the same is reflected in the resolv.conf file | +| Verify max DNS servers | Verify more than 6 servers are not allowed | +| Configure DNS source interface | Verify source interface is present in the DNS table (configDB) and DNS query packets are transmitted with the source IP present in source interface | +| Delete DNS source interface | Verify source interface is not present in the DNS table (configDB) and DNS query packets are not transmitted with the source IP present in source interface | +| Configure mgmt VRF | Verify that DNS query packets (from mgmt VRF) are transmitted based on nameservers present in resolv.conf file | +| Verify IPv4 DNS query is sent for ping to domain name | Verify whether DNS query is sent based on IPv4 source IP (based on source interface) & destination IP | +| Verify IPv6 DNS query is sent for ping to domain name | Verify whether DNS query is sent based on IPv6 source IP (based on source interface) & destination IP | +| show hosts | Verify source interface and nameservers are displayed correctly | + +#### Configuration via gNMI + +Same test as CLI configuration Test but using gNMI request. +Additional tests will be done to set DNS configuration at different levels of Yang models. + +#### Get configuration via gNMI + +Same as CLI show test but with gNMI request, will verify the JSON response is correct. +Additional tests will be done to get DNS configuration and DNS states at different levels of Yang models. + +#### Configuration via REST (POST/PUT/PATCH) + +Same test as CLI configuration Test but using REST POST request +Additional tests will be done to set DNS configuration at different levels of Yang models. + +**URIs for REST configurations:** + +Source interface - /restconf/data/openconfig-system:system/dns/config/openconfig-system-ext:source-intf +Name server - /restconf/data/openconfig-system:system/dns/servers/server[8.8.8.8]/config + +#### Get configuration via REST (GET) + +Same as CLI show test but with REST GET request, will verify the JSON response is correct. +Additional tests will be done to get DNS configuration and DNS states at different levels of Yang models. + + +# 10 Internal Design Information diff --git a/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md b/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md new file mode 100644 index 0000000000..6ff37a0535 --- /dev/null +++ b/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md @@ -0,0 +1,1129 @@ +# SONiC FRR-BGP Extended Unified Configuration Management Framework +## High Level Design Document +### Rev 0.9 + +## Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About This Manual](#about-this-manual) + * [Scope](#scope) + * [Definition/Abbreviation](#definitionabbreviation) + * [Table 1: Abbreviations](#table-1-abbreviations) + * [1 Feature Overview](#1-feature-overview) + * [1.1 Requirements](#11-requirements) + * [1.1.1 Functional Requirements](#111-functional-requirements) + * [1.1.2 Configuration and Management Requirements](#112-configuration-and-management-requirements) + * [1.1.3 Scalability Requirements](#113-scalability-requirements) + * [1.1.4 Warm Boot Requirements](#114-warmboot-requirements) + * [1.2 Design Overview](#12-design-overview) + * [1.2.1 Basic Approach](#121-basic-approach) + * [1.2.2 Container](#122-container) + * [2 Functionality](#2-functionality) + * [3 Design](#3-design) + * [3.1 Overview](#31-overview) + * [3.2 DB Changes](#32-db-changes) + * [3.3 SwSS Design](#33-swss-design) + * [3.4 SyncD](#34-syncd) + * [3.5 SAI](#35-sai) + * [3.6 User Interface](#36-user-interface) + * [3.6.1 Data Models](#361-data-models) + * [3.6.2 CLI](#362-cli) + * [3.6.2.1 Configuration Commands](#3621-configuration-command) + * [3.6.2.2 Show Commands](#3622-show-command) + * [3.6.2.3 Debug Commands](#3623-debug-command) + * [3.6.2.4 IS-CLI Compliance](#3624-is-cli-compliance) + * [3.6.3 REST API Support](#363-rest-api-support) + * [4 Flow Diagrams](#4-flow-diagrams) + * [4.1 Configuration Sequence](#41-configuration-sequence) + * [4.2 CLI Show Command Sequence](#42-cli-show-command-sequence) + * [4.2.1 CLI Show Sequence - config only](#421-cli-show-sequence-config-only) + * [4.2.2 CLI Show Sequence - State/Statistics](#422-cli-show-sequence-state-statistics) + * [4.3 REST Get Sequence](#43-rest-get-sequence) + * [4.3.1 REST Get Sequence - config only](#431-rest-get-sequence-config-only) + * [4.3.2 REST Get Sequence - State/Statistics](#432-rest-get-sequence-state-statistics) + * [5 Error Handling](#6-error-handling) + * [6 Serviceability and Debug](#7-serviceability-and-debug) + * [7 Warm Boot Support](#8-warm-boot-support) + * [8 Scalability](#9-scalability) + * [9 Unit Test](#10-unit-test) + * [APPENDIX](#APPENDIX) + + +## List of Tables +[Table 1: Abbreviations](#table-1-abbreviations) + +## Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | 09/25/2019 | Karthikeyan Arumugam | Initial version | +| 0.2 | 10/30/2019 | Venkatesan Mahalingam | Config DB schema changes | +| 0.3 | 11/20/2019 | Venkatesan Mahalingam | Added various fields in config DB | +| 0.4 | 12/18/2019 | Venkatesan Mahalingam | Addressed the comments on error handling and method of testing | +| 0.5 | 05/29/2020 | Venkatesan Mahalingam | Addressed the comments and added the new fields | +| 0.6 | 07/17/2020 | Eddy Lem | Update to new show bgp command syntax | +| 0.7 | 08/10/2020 | Venkatesan Mahalingam | Added a requirement to indicate that this document intends to cover non-template based FRR configurations i.e solely based on configurations from config DB | +| 0.8 | 01/28/2021 | Zhenhong Zhao & Venkatesan Mahalingam | Changed daemon name from bgpcfgd to frrcfgd and added some missed table and field descriptions | +| 0.9 | 03/05/2021 | Venkatesan Mahalingam | Added a field 'frr_mgmt_framework_config' field in DEVICE_METADATA config-DB schema to start frrcfgd daemon and table name change for PREFIX table | + +## About this Manual +This document provides general information about the implementation of Extended Unified Configuration and Management framework support for FRR-BGP feature in SONiC. +## Scope +This document describes the high level design of FRR-BGP Extended Unified Configuration and Management feature. + +## Definition/Abbreviation + +### Table 1: Abbreviations +| **Term** | **Meaning** | +|--------------------------|-------------------------------------| +| FRR | Free Range Routing Stack | +| CVL | Config Validation Library | +| VRF | Virtual Routing and Forwarding | +| RIB | Routing Information Base | +| PBR | Policy Based Routing | +| NBI | North Bound Interface | + + + +# 1 Feature Overview + +This feature extends and provides unified configuration and management capability for FRR-BGP features used in SONiC. This allows the user to configure & manage FRR-BGP using SONiC Management Framework with Open Config data models via REST, gNMI and provides access via SONiC Management Framework CLI. + +## 1.1 Requirements + + +### 1.1.1 Functional Requirements + + 1. Extend Unified mode for full FRR-BGP config and management in SONiC + 2. Ability to start frrcfgd (new FRR config deamon fully based on config-DB events) or bgpcfgd (template based legacy daemon) based on frr_mgmt_framework_config field with "true"/"false" in DEVICE_METADATA table + 3. Add frrcfgd daemon (sonic-buildimage/src/sonic-frr-mgmt-framework) and integrate with FRR-BGP for features supported in SONiC + 3. Support for retrieval of FRR-BGP state and statistics information + 4. For backward compatibility retain access to FRR UI (vtysh) for managing features that are NOT in conflict with SONiC features + 5. Configure FRR solely based on configurations from config DB (refer various config DB schemas given in this document) i.e not based on fixed/template based FRR configurations from frrcfgd with minimal configurations from config DB. + +### 1.1.2 Configuration and Management Requirements + +1. Support [Open Config data models for BGP](https://github.com/openconfig/public/tree/master/release/models/bgp) config and Management +2. Provide IS-CLI/gNMI/REST support for config and management of FRR-BGP features used in SONIC +3. Enhance with Custom YANG models for features used in BGP that are not supported via Open Config data model +4. Define ABNF schema for BGP features used in SONiC + +### 1.1.3 Scalability Requirements +N/A + +### 1.1.4 Warm Boot Requirements +As state and statistics information is retrieved from FRR-BGP on demand there is no Warm Boot specific requirements for this feature. + +## 1.2 Design Overview +SONiC FRR-BGP Extended Unified config and management capability makes use of Management framework to implement the backend and transformer methods to support Open Config data models for BGP and route policy feature. The backend converts the incoming request to Redis ABNF schema format and writes the configuration to Redis DB. Then from DB events, frrcfgd will configure FRR-BGP using FRR CLI commands. +It also uses management framework's transformer methods to do syntactic and semantic YANG validation of the requests using ABNF JSON before writing them into the Redis DB. + +### 1.2.1 Basic Approach + +* This enhancement takes comprehensive approach to support BGP features used in SONiC: + * Standard based YANG models and custom YANG models + * Open API spec + * Industry standard CLI + * Config Validation +* REST server, gNMI server, Transformer methods - all in Go +* Marshalling and unmarshalling using YGOT +* Redis updated using CAS(Check-and-Set) trans. (No locking, No rollback) +* Config Validation by using YANG model from ABNF schema + +### 1.2.2 Container + +There will be changes in following containers, +* sonic-mgmt-framework +* bgp + +### 1.2.3 SAI Overview +N/A - software feature + + +# 2 Functionality +## 2.1 Target Deployment Use Cases +Configure and manage FRR-BGP via gNMI, REST and CLI interfaces using SONiC Management Framework. + +## 2.2 Functional Description +Provide GNMI and REST support for config get/set, state get and statistics get, CLI config and show commands for FRR-BGP features used in SONiC. + + +# 3 Design +## 3.1 Overview +The extended unified config and management framework for FRR-BGP in SONiC is represented in below diagram. + +![FRR-BGP Unified Mgmt Framework](images/FRR-BGP-Unified-mgmt-frmwrk.png) + +1. Provide annotations for openconfig YANG to SONiC YANG objects mapping so that transformer and CVL will take care of processing the requests and updating the config DB. + +2. Provide transformer methods as per the annotations defined to take care of model specific logics and validations. + +3. Define SONiC YANG and Redis ABNF schema for the supported Open Config BGP models & objects. + +4. KLISH CLI and REST clients provide extensive BGP configurations and hence there should not be any need for BGP configurations via vtysh. + +5. In frrcfgd register for Redis DB events for the BGP and other related objects, so as to translate the Redis DB events to FRR-BGP CLI commands to configure FRR-BGP, similarly, the config daemon could configure other features like OSPF, BFD, static route..etc + +6. Update /usr/local/sonic/frrcfgd/bgpd.conf.j2 template for new FRR-BGP configurations supported in SONiC which will be used by sonic-cfggen to generate /etc/frr/bgpd.conf file. + +## 3.2 DB Changes +Following section describes the changes to DB. + +### 3.2.1 CONFIG DB + +Added new tables to configure following information: + + * BGP router & address family configurations + * BGP neighbor & address family configurations + * BGP peer group & address family configurations + * BGP listen prefix configuration for peer group + * BGP aggregate entries address family configuration + * BGP network address family configuration + * Route map configurations + * Route redistribute configurations + * Route policy sets (prefix/community/ext-community/as-path/neighbor/tag/nexthop) configurations + +Enhanced following table to configure additional attributes: + * DEVICE_METADATA + * BGP Neighbor table + +#### 3.2.1.1 BGP_GLOBALS +```JSON +;Defines BGP globals table +; +;Status: stable + +key = BGP_GLOBALS|vrf_name ; +vrf_name = 1\*15VCHAR ; VRF name should be "default" or prefixed with "Vrf" for user VRFs +router_id = \*IPv4prefix ; Router ID IPv4 address +local_asn = 1*10DIGIT ; Local ASN for the BGP instance +always_compare_med = "true" / "false" ; Allow comparing MED from different neighbors +load_balance_mp_relax = "true" / "false" ; Allow load sharing across routes that have different AS paths (but same length) +graceful_restart_enable = "true" / "false" ; Graceful restart status +gr_preserve_fw_state = "true" / "false" ; Sets F-bit indication that fib is preserved while doing Graceful Restart +gr_restart_time = 1*4DIGIT ; {1..3600 }; Set the time to wait to delete stale routes before a BGP open message is received +gr_stale_routes_time = 1*4DIGIT ; {1..3600 }; Set the max time to hold onto restarting peer's stale paths +external_compare_router_id = "true" / "false" ; Compare router-id for identical EBGP paths +ignore_as_path_length = "true" / "false" ; Ignore as-path length in selecting a route +log_nbr_state_changes = "true" / "false" ; Log neighbor up/down and reset reason +rr_cluster_id = 1*64VCHAR ; Route reflector cluster id +rr_allow_out_policy = "true" / "false" ; Router reflector allow outbound Policy +disable_ebgp_connected_rt_check = "true" / "false" ; Disable checking if nexthop is connected on ebgp sessions +fast_external_failover = "true" / "false" ; Immediately reset session if a link to a directly connected external peer goes down +network_import_check = "true" / "false" ; Check BGP network route exists in IGP +graceful_shutdown = "true" / "false" ; Graceful shutdown status +rr_clnt_to_clnt_reflection = "true" / "false" ; Configure client to client route reflection +max_dynamic_neighbors = 1*4DIGIT; {1..5000}; Maximum number of BGP Dynamic Neighbors that can be created +read_quanta = 1*2DIGIT; {1..10}; Indicates how many packets to read from peer socket per I/O cycle +write_quanta = 1*2DIGIT; {1..10}; Indicates how many packets to write to peer socket per run +coalesce_time = 1*10DIGIT; Subgroup coalesce timer value in milli-sec +route_map_process_delay = 1*3DIGIT; { 0..600}; Time in secs to wait before processing route-map changes +deterministic_med = "true" / "false" ; Pick the best-MED path among paths advertised from the neighboring AS +med_confed = "true" / "false" ; Compare MED among confederation paths when set to true +med_missing_as_worst = "true" / "false" ; Treat missing MED as the least preferred one when set to true +compare_confed_as_path = "true" / "false" ; Compare path lengths including confederation sets & sequences in selecting a route +as_path_mp_as_set = "true" / "false" ; Generate an AS_SET +default_ipv4_unicast = "true" / "false" ; Activate ipv4-unicast for a peer by default +default_local_preference = "true" / "false" ; Configure default local preference value +default_show_hostname = "true" / "false" ; Show hostname in BGP dumps +default_shutdown = "true" / "false" ; Apply administrative shutdown to newly configured peers +default_subgroup_pkt_queue_max = 1*3DIGIT; {20..100}; Configure subgroup packet queue max +max_med_time = 1*5DIGIT; {5..86400}; Time (seconds) period for max-med +max_med_val = 1*10DIGIT; Max MED value to be used +max_med_admin = "true" / "false" ; Advertise routes with max-med, administratively applied, for an indefinite period +max_med_admin_val = 1*10DIGIT; Max MED value to be used +max_delay = 1*4DIGIT; {0..3600}; Maximum delay for best path calculation +establish_wait = 1*4DIGIT; {1..3600}; Maximum delay for updates +confed_id = 1*10DIGIT; AS confederation identifier +confed_peers = 1*10DIGIT; Peer ASs in BGP confederation +keepalive = 1*5DIGIT; {0..65535}; Keepalive interval +holdtime = 1*5DIGIT; {0..65535}; Holdtime + +``` +#### 3.2.1.2 BGP_GLOBALS_AF +```JSON +;Defines BGP Address family table +; +;Status: stable + +key = BGP_GLOBALS_AF|vrf_name|afi_safi ; +vrf_name = 1\*15VCHAR ; VRF name should be "default" or prefixed with "Vrf" for user VRFs +afi_safi = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +max_ebgp_paths = 1*3DIGIT ; {1..256}; Number of eBGP paths +max_ibgp_paths = 1*3DIGIT ; {1..256}; Number of iBGP paths +import_vrf = 1\*15VCHAR ; VRF name to import the routes from +import_vrf_route_map = 1*64VCHAR ; route map name +route_download_filter = 1*64VCHAR ; route map name to filter routes from VRF +ebgp_route_distance = 1*3DIGIT; { 1.255}; Distance for routes external to the AS +ibgp_route_distance = 1*3DIGIT; { 1.255}; Distance for routes internal to the AS +local_route_distance = 1*3DIGIT; { 1.255}; Administrative distance for local routes +ibgp_equal_cluster_length = "true" / "false" ; Match the cluster length for iBGP paths +route_flap_dampen = "true" / "false" ; Route-flap dampening status +route_flap_dampen_half_life = 1*2DIGIT; {1..45}; Half-life time for the penalty +route_flap_dampen_reuse_threshold = 1*5DIGIT; {1..20000}; Value to start reusing a route +route_flap_dampen_suppress_threshold = 1*5DIGIT; {1..20000}; Value to start suppressing a route +route_flap_dampen_max_suppress = 1*3DIGIT; {1..255}; Maximum duration to suppress a stable route + +``` +#### 3.2.1.3 BGP_GLOBALS_AF_AGGREGATE_ADDR +```JSON +;Defines BGP Address family aggregate address table +; +;Status: stable + +key = BGP_GLOBALS_AF_AGGREGATE_ADDR|vrf_name|afi_safi|ip_prefix; +vrf_name = 1\*15VCHAR ; VRF name should be "default" or prefixed with "Vrf" for user VRFs +afi_safi = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +ip_prefix = IPv4Prefix / IPv6prefix ; Aggregate prefix +as_set = "true" / "false" ; Generate AS set path information +summary_only = "true" / "false" ; Filter more specific routes from updates +policy = 1*64VCHAR ; Apply route map to aggregate network +``` +#### 3.2.1.4 BGP_GLOBALS_AF_NETWORK +```JSON +;Defines BGP Address family network address table +; +;Status: stable + +key = BGP_GLOBALS_AF_NETWORK|vrf_name|afi_safi|ip_prefix; +vrf_name = 1\*15VCHAR ; VRF name should be "default" or prefixed with "Vrf" for user VRFs +afi_safi = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +ip_prefix = IPv4Prefix / IPv6prefix ; Network address to announce via BGP +policy = 1*64VCHAR ; Route-map to modify the attributes +backdoor = "true" / "false" ; Backdoor route +``` + +#### 3.2.1.5 BGP_LISTEN_PREFIX +```JSON +;Defines BGP Listen Prefix table +; +;Status: stable + +key = BGP_GLOBALS_LISTEN_PREFIX|vrf_name|ip_prefix ; +vrf_name = 1\*15VCHAR ; VRF name +ip_prefix = IPv4Prefix / IPv6prefix ; +peer_group = 1*64VCHAR ; Peer group this listen prefix is associated with +``` + +#### 3.2.1.6 BGP_NEIGHBOR + +```JSON +;Defines BGP neighbor table +; +;Status: stable + +key = BGP_NEIGHBOR|vrf_name|neighbor ; +vrf_name = 1\*15VCHAR ; VRF name +neighbor = IPv4Prefix / IPv6prefix / 1*64VCHAR ; IPv4/IPv6 address/Interface name +peer_group_name = 1*64VCHAR ; peer group name +local_asn = 1*10DIGIT ; Local ASN for the BGP neighbor +name = 1*64VCHAR ; BGP neighbor description +asn = 1*10DIGIT; Remote ASN value +peer_type = "internal" / "external" Internal/External BGP peer +ebgp_multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +ebgp_multihop_ttl = 1*3DIGIT ; {1..255} EBGP multihop count +auth_password = STRING ; Set a password +keepalive = 1*4DIGIT ; {1..3600} keepalive interval +holdtime = 1*4DIGIT ; {1..3600} hold time +conn_retry = 1*5DIGIT ; {1..65535} Connection retry timer +min_adv_interval = 1*3DIGIT ; {1..600} Minimum interval between sending BGP routing updates +local_addr = IPprefix ; local IP address +passive_mode = "true" / "false" ; Don't send open messages +capability_ext_nexthop = "true" / "false" ; Advertise extended next-hop capability +disable_ebgp_connected_route_check = "true" / "false" ; one-hop away EBGP peer using loopback address +enforce_first_as = "true" / "false" ; Enforce the first AS for EBGP route +solo_peer = "true" / "false" ; Solo peer - part of its own update group +ttl_security_hops = 1*3DIGIT ; {1.254} BGP ttl-security parameters +bfd = "true" / "false" ; Enable BFD support +bfd_check_ctrl_plane_failure = "true" / "false" ; Link dataplane status with BGP controlplane +capability_dynamic = "true" / "false" ; Advertise dynamic capability +dont_negotiate_capability = "true" / "false" ; Do not perform capability negotiation +enforce_multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +override_capability = "true" / "false" ; Override capability negotiation result +peer_port = 1*5DIGIT ; {0..65535} Neighbor's BGP port +shutdown_message = "true" / "false" ; Add a shutdown message +strict_capability_match = "true" / "false" ; Strict capability negotiation match +admin_status = "true" / "false" ; Neighbor admin status +local_as_no_prepend = "true" / "false" ; Do not prepend local-as to updates from ebgp peers +local_as_replace_as = "true" / "false" ; Do not prepend local-as to updates from ibgp peers + +``` +#### 3.2.1.7 BGP_NEIGHBOR_AF +```JSON +;Defines BGP Neighbor table at an address family level +; +;Status: stable + +key = BGP_NEIGHBOR_AF|vrf_name|neighbor|afi_safi ; +vrf_name = 1\*15VCHAR ; VRF name +neighbor = IPv4Prefix / IPv6prefix ; +afi_safi = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +admin_status = "true" / "false" ; Neighbor admin status +send_default_route = "true" / "false" ; +default_rmap = 1*64VCHAR ; Filter sending default routes bsaed on this route map. +max_prefix_limit = 1*10DIGIT; Maximum number of prefixes to accept from this peer +max_prefix_warning_only = "true" / "false" ; Only give warning message when limit is exceeded +max_prefix_warning_threshold = 1*3DIGIT; Threshold value (%) at which to generate a warning msg +max_prefix_restart_interval = 1*5DIGIT; Restart bgp connection after limit is exceeded +route_map_in = 1*64VCHAR ; Apply route map on incoming routes from neighbor +route_map_out = 1*64VCHAR ; Apply route map on outgoing routes to neighbor +soft_reconfiguration_in = "true" / "false" ; Per neighbor soft reconfiguration +unsuppress_map_name = 1*64VCHAR ; Route-map to selectively un-suppress suppressed routes +rrclient = "true" / "false" ; Configure a neighbor as Route Reflector client +weight = 1*5DIGIT ; {0..65535} Set default weight for routes from this neighbor +as_override = "true" / "false" ; Override ASNs in outbound updates if aspath equals remote-as +send_community = "standard" / "extended" / "both" / "none" ; Send Community attribute to this neighbor +tx_add_paths = "tx_all_paths" / "tx_best_path_per_as" ; Advertise all paths to a neighbor / Send only best path per AS to a neighbor +unchanged_as_path = "true" / "false" ; AS-PATH attribute is propagated unchanged to a neighbor +unchanged_med = "true" / "false" ; MED attribute is propagated unchanged to a neighbor +unchanged_nexthop = "true" / "false" ; Nexthop attribute is propagated unchanged to a neighbor +filter_list_in = 1*64VCHAR ; AS Path filter for incoming routes +filter_list_out = 1*64VCHAR ; AS path Filter for outgoing routes +nhself = "true" / "false" ; Disable the next hop calculation for this neighbor +nexthop_self_force = "true" / "false" ; Set the next hop to self for reflected routes +prefix_list_in = 1*64VCHAR ; Prefix list to apply on incoming routes +prefix_list_out = 1*64VCHAR ; Prefix list to apply on outcoming routes +remove_private_as_enabled = "true" / "false" ; Remove private ASNs in outbound updates +replace_private_as = "true" / "false" ; Replace private ASNs with our ASN in outbound update +remove_private_as_all = "true" / "false" ; Apply to all AS numbers +allow_as_in = "true" / "false" ; Accept as-path with my AS present in it +allow_as_count = 1*3DIGIT ; Number of occurences of AS number +allow_as_origin = "true" / "false" ; Only accept my AS in the as-path if the route was originated in my AS +cap_orf = "send" / "receive" / "both" ; Outbound route filtering capability +route_server_client = "true" / "false" ; Configure a neighbor as Route Server client +``` + +#### 3.2.1.8 BGP_PEER_GROUP +The existing BGP_PEER_RANGE (peer group) table does not have vrf-name as the key (but added as the field in the table as per VRF HLD) but FRR and open config models have the peer-group under VRF context, so, the new table BGP_PEER_GROUP has been introduced for configurations from management framework (This is not a backward compatible change, expecting the user to migrate to this table in the near future). + +```JSON +;Defines BGP peer group table +; +;Status: stable + +key = BGP_PEER_GROUP|vrf_name|peer_group_name ; +vrf_name = 1\*15VCHAR ; VRF name +peer_group_name = 1*64VCHAR ; alias name for the peer group , must be unique +local_asn = 1*10DIGIT ; Local ASN for the BGP neighbor +name = 1*64VCHAR ; BGP neighbor description +asn = 1*10DIGIT; Remote ASN value +peer_type = "internal" / "external" Internal/External BGP peer +ebgp_multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +ebgp_multihop_ttl = 1*3DIGIT ; {1..255} EBGP multihop count +auth_password = STRING ; Set a password +keepalive = 1*4DIGIT ; {1..3600} keepalive interval +holdtime = 1*4DIGIT ; {1..3600} hold time +conn_retry = 1*5DIGIT ; {1..65535} Connection retry timer +min_adv_interval = 1*3DIGIT ; {1..600} Minimum interval between sending BGP routing updates +local_address = IPprefix ; local IP address +passive_mode = "true" / "false" ; Don't send open messages +capability_ext_nexthop = "true" / "false" ; Advertise extended next-hop capability +disable_ebgp_connected_route_check = "true" / "false" ; one-hop away EBGP peer using loopback address +enforce_first_as = "true" / "false" ; Enforce the first AS for EBGP route +solo_peer = "true" / "false" ; Solo peer - part of its own update group +ttl_security_hops = 1*3DIGIT ; {1.254} BGP ttl-security parameters +bfd = "true" / "false" ; Enable BFD support +bfd_check_ctrl_plane_failure = "true" / "false" ; Link dataplane status with BGP controlplane +capability-dynamic = "true" / "false" ; Advertise dynamic capability +dont-negotiate-capability = "true" / "false" ; Do not perform capability negotiation +enforce-multihop = "true" / "false" ; Allow EBGP neighbors not on directly connected networks +override-capability = "true" / "false" ; Override capability negotiation result +peer-port = 1*5DIGIT ; {0..65535} Neighbor's BGP port +shutdown-message = "true" / "false" ; Add a shutdown message +strict-capability-match = "true" / "false" ; Strict capability negotiation match +admin_status = "true" / "false" ; Neighbor admin status +local_as_no_prepend = "true" / "false" ; Do not prepend local-as to updates from ebgp peers +local_as_replace_as = "true" / "false" ; Do not prepend local-as to updates from ibgp peers +``` +#### 3.2.1.9 BGP_PEER_GROUP_AF + +```JSON +;Defines BGP per address family peer group table +; +;Status: stable + +key = BGP_PEER_GROUP_AF|vrf_name|peer_group_name|afi_safi" ; +vrf = 1\*15VCHAR ; VRF name +peer_group_name = 1*64VCHAR ; alias name for the peer group template, must be unique +afi_safi = "ipv4_unicast" / "ipv6_unicast" / "l2vpn_evpn" ; address family +admin_status = "true" / "false" ; Neighbor admin status +send_default_route = "true" / "false" ; +default_rmap = 1*64VCHAR ; Filter sending default routes bsaed on this route map. +max_prefix_limit = 1*10DIGIT; Maximum number of prefixes to accept from this peer +max_prefix_warning_only = "true" / "false" ; Only give warning message when limit is exceeded +max_prefix_warning_threshold = 1*3DIGIT; Threshold value (%) at which to generate a warning msg +max_prefix_restart_interval = 1*5DIGIT; Restart bgp connection after limit is exceeded +route_map_in = 1*64VCHAR ; Apply route map on incoming routes from neighbor +route_map_out = 1*64VCHAR ; Apply route map on outgoing routes to neighbor +soft_reconfiguration_in = "true" / "false" ; Per neighbor soft reconfiguration +unsuppress_map_name = 1*64VCHAR ; Route-map to selectively un-suppress suppressed routes +rrclient = "true" / "false" ; Configure a neighbor as Route Reflector client +weight = 1*5DIGIT ; {0..65535} Set default weight for routes from this neighbor +as_override = "true" / "false" ; Override ASNs in outbound updates if aspath equals remote-as +send_community = "standard" / "extended" / "both" / "none" ; Send Community attribute to this neighbor +tx_add_paths = "tx_all_paths" / "tx_best_path_per_as" ; Advertise all paths to a neighbor / Send only best path per AS to a neighbor +unchanged_as_path = "true" / "false" ; AS-PATH attribute is propagated unchanged to a neighbor +unchanged_med = "true" / "false" ; MED attribute is propagated unchanged to a neighbor +unchanged_nexthop = "true" / "false" ; Nexthop attribute is propagated unchanged to a neighbor +filter_list_in = 1*64VCHAR ; AS Path filter for incoming routes +filter_list_out = 1*64VCHAR ; AS path Filter for outgoing routes +nhself = "true" / "false" ; Disable the next hop calculation for this neighbor +nexthop_self_force = "true" / "false" ; Set the next hop to self for reflected routes +prefix_list_in = 1*64VCHAR ; Prefix list to apply on incoming routes +prefix_list_out = 1*64VCHAR ; Prefix list to apply on outcoming routes +remove_private_as_enabled = "true" / "false" ; Remove private ASNs in outbound updates +replace_private_as = "true" / "false" ; Replace private ASNs with our ASN in outbound update +remove_private_as_all = "true" / "false" ; Apply to all AS numbers +allow_as_in = "true" / "false" ; Accept as-path with my AS present in it +allow_as_count = 1*3DIGIT ; Number of occurences of AS number +allow_as_origin = "true" / "false" ; Only accept my AS in the as-path if the route was originated in my AS +cap_orf = "send" / "receive" / "both" ; Outbound route filtering capability +route_server_client = "true" / "false" ; Configure a neighbor as Route Server client +``` + +### 3.2.1.10 ROUTE_MAP_SET +```JSON +;Defines route map set table +; +;Status: stable + +key = ROUTE_MAP_SET|name ; route map set name must be unique +name = 1*255VCHAR ; route map set name + +``` + +#### 3.2.1.10.1 ROUTE_MAP +```JSON +;Defines route map table +; +;Status: stable + +key = ROUTE_MAP|route_map_name|stmt_name ; +route_map_name = 1*64VCHAR ; route map name +stmt_name = 1*64VCHAR ; statment name +route_operation = "ACCEPT" / "REJECT"; Permit/Deny operation +match_interface = 1*64VCHAR ; Match interface name +match_prefix_set = 1*64VCHAR ; Match prefix sets +match_protocol = "bgp" / "ospf" / "ospf3" / "static" / "connected";Match protocol via which the route was learnt +match_next_hop_set = 1*64VCHAR ; ;Match next-hop present in prefix list +match_src_vrf = 1*15VCHAR ; Match source VRF name +match_neighbor = IPv4Prefix / IPv6prefix ; Match neighbor IP +match_tag = 1*10DIGIT ; Match tag value +match_med = 1*10DIGIT ; Match metric of route +match_origin = 1*64VCHAR ; Match BGP origin code +match_local_pref = 1*64VCHAR ; Match local-preference of route +match_community = 1*64VCHAR ; Match BGP community list +match_ext_community = 1*64VCHAR ; Match BGP/VPN extended community list +match_as_path = 1*64VCHAR ; Match BGP AS path list +call_route_map = 1*64VCHAR ; Jump to another Route-Map after match+set + +set_origin = 1*64VCHAR ; Set BGP origin code +set_local_pref = 1*64VCHAR ; Set BGP local preference path attribute +set_next_hop = 1*64VCHAR ; Set IP address of next hop +set_med = 1*64VCHAR ; Set Metric value for destination routing protocol +set_ipv6_next_hop_global = IPv6prefix ; Set IPv6 address of next hop +set_ipv6_next_hop_prefer_global = "true" / "false" ; Prefer global over link-local if both exist +set_repeat_asn = 1*3DIGIT ; NO of times the set_asn number to be repeated +set_asn = 1*10DIGIT ; Set ASN number +set_community_inline = 1*64VCHAR ; Set BGP community attribute inline +set_community_ref = 1*64VCHAR ; Refer BGP community attribute from community table +set_ext_community_inline = 1*64VCHAR ; Set BGP extended community attribute inline +set_ext_community_ref = 1*64VCHAR ; Refer BGP extended community attribute from extended community table +set_metric_action = "METRIC_SET_VALUE"/"METRIC_ADD_VALUE"/"METRIC_SUBSTRACT_VALUE"/"METRIC_SET_RTT"/"METRIC_ADD_RTT"/"METRIC_SUBSTRACT_RTT" ; Set action type for handling metric value +set_metric = 1*10DIGIT ; Metric value for given set metric action +set_asn_list = number list ; Set BGP prepended AS-path attribute + ; Acceptable List of ASN number "ASN, ASN" +``` +#### 3.2.1.11 ROUTE_REDISTRIBUTE +```JSON +;Defines route redistribution table +; +;Status: stable + +key = ROUTE_REDISTRIBUTE|vrf_name|src_protocol|dst_protocol|addr_family ; +vrf = 1*15VCHAR ; VRF name +src_protocol = "connected" / "static" / "ospf" / "ospf3"; Source protocol +dst_protocol = "bgp" ; Destination protocol +addr_family = "ipv4" / "ipv6" ; Address family +route_map = 1*64VCHAR ; route map filter to apply for redistribution +metric = 1*10DIGIT ; Metric value +``` + +### 3.2.1.12 PREFIX_SET +```JSON +;Defines prefix set table +; +;Status: stable + +key = PREFIX_SET|name ; prefix_set_name must be unique +name = 1*255VCHAR ; community set name +mode = "IPv4"/"IPv6" ; mode of prefix set. + +``` +#### 3.2.1.12.1 PREFIX +```JSON +;Defines prefix table +; +;Status: stable +key = PREFIX|set_name|ip_prefix|masklength_range; an instance of this key will be repeated for each prefix + ; an instance of this key/value pair will be repeated for each prefix +set_name = 1*255VCHAR ; community set name +ip_prefix = IPv4prefix / IPv6prefix ; prefix, example 1.1.1.1/32 +masklength_range = 1*255VCHAR ; exact or (masklength_range..low-masklength_range_high). example 8..16 or exact +action = "PERMIT" / "DENY" ; Actions +``` +### 3.2.1.13 COMMUNITY_SET +```JSON +;Defines community table +; +;Status: stable +key = COMMUNITY_SET|name ; name must be unique +set_type = "STANDARD"/"EXPANDED" +match_action = "ANY/ALL" +community_member = string list ; community member list + ; Acceptable List of communities as ("AA:NN","local-AS", "no-advertise", "no-export" | regex) + +``` +### 3.2.1.14 EXTENDED_COMMUNITY_SET +```JSON +;Defines extended community table +; +;Status: stable +key = EXTENDED_COMMUNITY_SET|name ; name must be unique +set_type = "STANDARD"/"EXPANDED" +match_action = "ANY/ALL" +community_member = string list ; community member list + ; Acceptable List of communities as ("route-target/route-origin:AA:NN" or "IP_Address" or regex) +``` +### 3.2.1.15 AS_PATH_SET +```JSON +;Defines extended community table +; +;Status: stable +key = AS_PATH_SET|name ; name must be unique +as_path_set_member = string list ; AS path list + ; Acceptable List of as paths "string, string" +``` + +### 3.2.1.16 DEVICE_METADATA +```JSON +; Ability to start frrcfgd (new FRR config deamon fully based on config-DB events) or bgpcfgd (template based legacy daemon) based on 'frr_mgmt_framework_config' field with "true"/"false" +; +;Status: stable +key = localhost ; name must be unique +frr_mgmt_framework_config = "true" / "false" ; +``` + +### 3.2.2 APP DB +N/A + +### 3.2.3 STATE DB +No changes to State DB, State and statistics information will be retrieved directly from FRR-BGP. + +### 3.2.4 ASIC DB +N/A + +### 3.2.5 COUNTER DB +N/A + +## 3.3 Switch State Service Design + +### 3.3.1 Orchestration Agent +No changes to Orch agent. + +### 3.3.2 Other Process + +#### 3.3.2.1 FRR Template Changes + +FRR template must be enhanced to contain FRR-BGP related configuration that are supported via FRR-BGP extended unified (Config DB is propagated to FRR config at startup) with non-integrated mode (FRR configuration is saved in individual files: “bgpd.conf”, “zebra.conf” and ospfd.conf....etc) config management framework. + +On startup, sonic-cfggen will use "/usr/share/sonic/templates/bgpd.conf.j2" to generate "/etc/frr/bgpd.conf". + +## 3.4 SyncD +No changes to SyncD + +## 3.5 SAI +No changes to SAI APIs. + +## 3.6 User Interface +### 3.6.1 Data Models +List of Open-config YANG models required for FRR-BGP Unified Configuration and Management are, + + 1) openconfig-network-instance.yang + + 2) openconfig-routing-policy.yang + +BGP and "routing policy" related augmented and not-supported fields are available in openconfig-bgp-ext.yang and openconfig-routing-policy-ext.yang files respectively. +Supported YANG containers: +``` +module: openconfig-network-instance + +--rw network-instances + +--rw network-instance* [name] + +--rw table-connections + | ... + +--rw protocols + +--rw protocol* [identifier name] + +--rw bgp + +--rw global + | | ... + | +--rw afi-safis + | +--rw afi-safi* [afi-safi-name] + | ... + +--rw neighbors + | +--rw neighbor* [neighbor-address] + | | ... + | +--rw afi-safis + | +--rw afi-safi* [afi-safi-name] + | ... + +--rw peer-groups + | +---rw peer-group* [peer-group-name] + | | ... + | +--rw afi-safis + | +--rw afi-safi* [afi-safi-name] + | ... + +--ro rib + +--ro afi-safis + ... + +module: openconfig-routing-policy + +--rw routing-policy + +--rw defined-sets + | ... + +--rw policy-definitions + +--rw policy-definition* [name] + ... + +``` +### 3.6.2 CLI + 1. For all configuration commands, the CLI request is converted to a corresponding REST client SDK request based on the Open Config data model that was generated by the Swagger generator, and is given to the REST server. + + 2. From there on it will follow the same path as a REST config request for create, update and delete operations. + + 3. The Swagger generated REST server handles all the REST requests from the client SDK and invokes a common handler for all the create, update, replace, delete and get operations along with path and payload. This common handler converts all the requests into Transformer arguments and invokes the corresponding Transformer APIs. + + 4. For show commands, the CLI request is converted to a corresponding REST client SDK get request based on Open Config data model's config or state object on a case by case basis. + + 5. For show commands that requires retrieval of the data that doesn't contain any state information (information only based on the configuration), the backend callback will fetch the data from CONFIG_DB. + + 6. For show commands that requires retrieval of state or statistics information the backend, management framework executes the FRR CLI using "docker exec bgp vtysh -c " to fetch the data in JSON format from FRR-BGP. + + 7. At transformer the JSON output (retrieved from FRR BGP) is converted back to corresponding open config objects and returned to the caller. + + 8. For CLI show, the output returned in object format is then translated back to CLI Jinga template for output display in CLI. + + +#### 3.6.2.1 Configuration Commands + +##### 3.6.2.1.1 BGP Router mode commands + +|Command Description |CLI Command | +|:-----------------|:---------------| +|Enable BGP routing instance |sonic(config)# router bgp \ [vrf \] | +|Override configured BGP router-id |sonic(config-router-bgp)# router-id \ | +|Configure default best path selection |sonic(config-router-bgp)# bestpath {as-path { confed \| ignore \| multipath-relax \[as-set] \| med { confed \| missing-as-worst } } | +|Configure graceful restart capability params |sonic(config-router-bgp)# graceful-restart preserve-fw-state
sonic(config-router-bgp)# graceful-restart restart-time <1-3600>
sonic(config-router-bgp)# graceful-restart stalepath-time <1-3600>| +|Configure BGP IPv4/IPv6 neighbor |sonic(config-router-bgp)# neighbor { \ \| \ } | +|Configure BGP peer group |sonic(config-router-bgp)# peer-group \| +|Enter address family command mode|sonic(config-router-bgp)# address-family { ipv4 unicast \| ipv6 unicast \| l2vpn evpn} | +| Subgroup coalesce timer | sonic(config-router-bgp)# coalesce-time \ | +| How many packets to read from peer socket per I/O cycle | sonic(config-router-bgp)# read-quanta \ | +| How many packets to write to peer socket per run | sonic(config-router-bgp)# write-quanta \ | +| Configure client to client route reflection | sonic(config-router-bgp)# client-to-client reflection | +| Configure Route-Reflector Cluster-id | sonic(config-router-bgp)# cluster-id { <32-bit-val> \| } | +| Log neighbor up/down and reset reason(default) | sonic(config-router-bgp)# log-neighbor-changes | +| Pick the best-MED path among paths advertised from the neighboring AS | sonic(config-router-bgp)# deterministic-med | +| Enable route-flap dampening | sonic(config-router-bgp)# dampening \ \ \ | +| Disable checking if nexthop is connected on ebgp sessions | sonic(config-router-bgp)# disable-ebgp-connected-route-check | +| Graceful shutdown parameters | sonic(config-router-bgp)# graceful-shutdown | +| BGP Dynamic Neighbors listen commands | sonic(config-router-bgp)# bgp listen \{ limit \ \| range \ } | +| Advertise routes with max-med | sonic(config-router-bgp)# max-med on-startup [\