-
Notifications
You must be signed in to change notification settings - Fork 266
Context configuration
Context configuration describes parameters that can be passed to syncd and orchagent (OA) (internally using sairedis library) in multi-syncd, multi-asic environment. Single context is describing a single syncd instance, but context name is more appropriate, since it's describes environment in which syncd is running and not syncd instance itself.
Sample context_config.json can be found here: https://github.com/Azure/sonic-sairedis/blob/master/lib/context_config.json
In basic environment we have only 1 OA and only 1 syncd, where syncd is supporting only 1 switch (ASIC). This is default behavior, and no special configuration is needed for OA or syncd to communicate and be aware in what environment they are running. But since SONiC is growing fast and support to have multiple ASIC support in single syncd is needed, and for future planning, support for multiple syncd as well (line-card with own CPU in chassis scenario).
Current support for multiple ASIC in single syncd and multiple syncd in entire system is already supported in sairedis library, but multi scenario is not fully tested yet, as there was no hardware to support this at a time. There are 3 scenarios that are supported now:
- single OA, single syncd with single ASIC support (default behavior)
- single OA, single syncd with multiple ASIC support (requires context configuration)
- single OA, multiple syncd, each with single or multiple ASIC support (requires context configuration)
Currently OA is not aware of multiple syncd or multiple switches, and it requires redesign to support those scenarios. Current workaround for that is to have multiple instances of "1 OA and 1 syncd" environments (for example for testing).
Multiple OA are not supported, although if you make everything totally separate and not conflicting, it is possible to make this work. Ping me if you will need to know this.
If syncd needs to support multiple switches/ASICs, or OA that needs to talk to multiple syncd, then context configuration is required, and here is where context_config.json is providing that configuration. Same configuration file needs to be used in current environment (OA + syncd) otherwise unpredicted communication errors can happen.
When syncd (actually vendor SAI) supports multiple switch instances, each switch is created using SAI API "create_switch", and to identify each switch uniquely, some unique identifier is needed to point to exact hardware we want to connect. This is achieved by SAI switch attribute SAI_SWITCH_ATTR_SWITCH_HARDWARE_INFO, which is suggested to be a readable char C string, something that SAI vendor can interpret internally and choose which hardware connect while calling create_switch. Only 1 switch can be created using SAI_SWITCH_ATTR_SWITCH_HARDWARE_INFO value, and this value is treated as a KEY, passing the same value 2nd time when there is existing switch in SAI, should result with SAI error.
Syncd and orchagent don't know up front how many switches current SAI supports, and what are hardware info values for them (default scenario, this attribute is empty), that's why context configuration is required and those values are needed for metadata validation in both syncd and orchagent.
How many switches are supported should be provided in "switches" key in each context in configuration file.
Sample context_config.json can be found here: https://github.com/Azure/sonic-sairedis/blob/master/lib/context_config.json
Main key in json is "CONTEXTS", which is array of json objects where each object represents separate context (syncd) instance. Each context object have the same keys but different values.
Context keys - All fields are mandatory, there are no default values here.
- guid - global unique identification number in entire context configuration file, value must be in range [0..255], it's encoded in each VID (virtual OID), and it puts limit on how many syncd instances we can be running at the same moment, which is 256
- name - user friendly string name for each syncd instance running in the system, it should be unique for convenience, currently will be used only in logs to notify for some configuration and errors, to easy locate where we have problem
- dbAsic - location of ASIC database for this particular syncd instance, this string will be passed to DBConnector constructor, and this database will be used to store SAI objects, this database can hold SAI objects from multiple switches from the same syncd instance. DO NOT USE SAME DATABASE FOR MANY SYNCD INSTANCES
- dbCounters - location of COUNTERS database for this particular syncd instance, this string will be passed to DBConnector constructor, and this database will be used to store COUNTERS, this database can hold COUNTERS from multiple switches (NOT TESTED) from the same syncd instance. DO NOT USE SAME DATABASE FOR MANY SYNCD INSTANCES
-
dbFlex - location of FLEX_COUNTERS database for this particular syncd instance, this string will be passed to DBConnector constructor, and this database will be used to store FLEX_COUNTERS, this database can hold FLEX_COUNTERS from multiple switches (NOT TESTED) from the same syncd instance. DO NOT USE SAME DATABASE FOR MANY SYNCD INSTANCES
- NOTE: actually dbFlex database is used as communication channel between syncd and OA to register for counters and counter groups, this actually needs to be fixed, and dbAsic should be used for that purpose, because current configuration is causing race conditions TODO, FIXME
-
dbState - location of STATE database for this particular syncd instance, this string will be passed to DBConnector constructor, and this database will be used to store "warm-shutdown" state flags when syncd will be requested to shutdown. DO NOT USE SAME DATABASE FOR MANY SYNCD INSTANCES
- NOTE: currently when syncd is starting, it check swss::WarmStart::checkWarmStart() method, which is implemented inside swss-common, and that swss::WarmStart class have hardcoded STATE_DB string in initialize method, and this means same database will be used to warm boot scenario, even in multiple syncd case, or if database name will be changed in database_config.json file, this means multiple syncd is not supported for warm boot because of this, and that scenario will have unpredicted behavior, this needs to be addressed and fixed on swss-common repo TODO, FIXME
- zmq_enable - bool flag, which indicates whether this syncd instance should use ZMQ channel for communication with OA, rather than default REDIS server channel (dbAsic), this flag will be deprecated in near future and replaced with "communication_mode" which will be (redis_async, redis_sync, zmq_sync)
- zmq_endpoint - ZMQ endpoint for main main communication channel, valid only when zmq_enable=true, this channel will be used for general synchronous communication like create/remove/set/get API etc, learn more on ZMQ and channels here: https://zeromq.org/get-started/, DO NOT USE SAME CHANNEL FOR MANY SYNCD INSTANCES
- zmq_ntf_endpoint - ZMQ endpoint for notifications communication channel, valid only when zmq_enable=true, this channel will be used for asynchronous delivery of notifications from syncd to OA, cannot be equal to "zmq_endpoint" DO NOT USE SAME CHANNEL FOR MANY SYNCD INSTANCES
- switches - array of json objects, which each one represents single switch instance in current syncd instance, this must match what vendor SAI supports. described below
Swiches keys - All fields are mandatory, there are no defaults here.
- index - local unique index number assigned to switch, values are in range [0..255], which are encoded in each VID (virtual OID) and this currently limits maximum number of switch/ASICs running in single syncd instance to 256, this value must be unique in "switches" array, but it may repeat across multiple CONTEXT instances, used internally by syncd and OA to uniquely identify switch
- hwinfo - locally unique string value that vendor accepts as SAI_SWITCH_ATTR_SWITCH_HARDWARE_INFO, this will be used in validation process during create_switch API call, and no other values will be accepted except the ones defined in "switches" array, this string must be the same value that vendor SAI will accept during create_switch call
All database strings are used internally via swss-common, and are selecting REDIS database based on database_config.json.
Orchagent consume context_config.json automatically, currently it's hardcoded to this location:
sonic-swss/orchagent/saihelper.cpp:#define CONTEXT_CFG_FILE "/usr/share/sonic/hwsku/context_config.json"
And this value will be passed using SAI_REDIS_KEY_CONTEXT_CONFIG profile key, which should be available via service_method_table passed in sai_api_initialize API.
Syncd will consume context_config.json by explicitly passing it via command line using "-x --contextConfig" parameter and also explicit context must be selected by syncd from context file, this is done by explicitly passing guid via command line using "-g --globalContext" parameter. DO NOT USE SAME GUID IN MULTIPLE SYNCD INSTANCES
Since orchagent is able to talk to multiple syncd instances, there is a need to tell OA during create_switch API, on which syncd we want to call create_switch. This is done explicitly by passing extra attribute to create_switch:
sai_attribute_t attr;
attr.id = SAI_REDIS_SWITCH_ATTR_CONTEXT;
attr.value.u32 = guid; // matching one of guids from context_config.json
This attribute must be last attribute in attribute list passed to create_switch.
After successful creation of the switch, guid and switch index (index is deduced based on SAI_SWITCH_ATTR_SWITCH_HARDWARE_INFO) are encoded into every VID, and internally sairedis is able to make sense of those values and talk to right syncd instance when that VID is used in any later API call.
In multiple syncd scenario there is possibility to mix VIDs from multiple syncd to single SAI API call, but this scenario is validated by metadata checks, and it will return SAI error and a proper syslog message.
Syncd and OA can communicate in 2 ways: asynchronous and synchronous.
- asynchronous mode - default mode, this means that for every create/remove/set operation there is no feedback (return status) from syncd, and OA assumes that that operation succeeded. This have some benefits and downsides. Benefits are, that those operations can be put into pipeline inside sairedis, and pushed to syncd in single operation, which save a lot of time by not having return. Pipeline is automatically flushed when there is synchronous operations executed like GET API or if there are no operations, then after 1 second timeout. Downside is, that if any of create/remove/set operation fails, syncd will crash, since it produces inconsistent state between OA and syncd, and there is no recovery from that.
- synchronous mode - in this mode OA receives status for every API that is called, and in case of failure it can decide whether error is critical or not, and take proper action based on error code. Downside is that when API is executed, OA is waiting for response, and since it's single threaded, it's doing nothing in that time. This is especially visible in fast-boot scenario, on the same branch when we measure async vs sync mode, the difference can be dramatic like 8-10 seconds slower for sync mode.
Currently there are 3 communication modes between syncd and OA:
- redis_async - default asynchronous, REDIS server is used as communication proxy, and ASIC database is used for transporting messages
- redis_sync - same as redis_async, but this one is synchronous
- zmq_sync - synchronous mode, but uses ZMQ library to communicate, channel can be ether tcp address or ipc (pipe file on disk) - this mode is currently partially supported, since notifications on OA are directly listened on REDIS database, issue is described here: https://github.com/Azure/sonic-swss/issues/1145
Syncd and OA must run in the same mode to properly working, mode can be set at the start of the process via command line, and cannot be changed dynamically later on.
Syncd communication mode can be enabled using command line "-z --redisCommunicationMode" parameter or deprecated "-s --syncMode". Orchagent communication mode can be enabled using command line "-z redis communication mode" parameter, or deprecated "-s enable synchronous mode".
When you have a chance, don't use "-s" parameters as they will be removed in near future.
DO NOT USE - will not stop you using it, but then you will need to deal with errors yourself