refactor!: multi-level snapshot comparison and config parser refactor

docs/panos-upgrade-assurance/api/snapshot_compare.md

@@ -92,8 +92,7 @@ __Parameters__
     For the elements specified as
 
     * `str` - the element value is the name of the report (state area),
-    * `dict` - the element contains the report name (state area) and the key value and report configuration as the
-        element value.
+    * `dict` - the element contains the report name (state area) as the key and report configuration as the element value.
 
 __Raises__
 
@@ -194,10 +193,10 @@ def calculate_diff_on_dicts(
 
 The static method to calculate a difference between two dictionaries.
 
-By default dictionaries are compared by going down all nested levels, to the point where key-value pairs are just strings
-or numbers. It is possible to configure which keys from these pairs should be compared (by default we compare all
-available keys). This is done using the `properties` parameter. It's a list of the bottom most level keys. For example,
-when comparing route tables snapshots are formatted like:
+By default dictionaries are compared by going down all nested levels. It is possible to configure which keys on each
+level should be compared (by default we compare all available keys). This is done using the `properties` parameter.
+It's a list of keys that can be compared or skipped on each level. For example, when comparing route tables snapshots are
+formatted like:
 
 ```python showLineNumbers
 {
@@ -217,8 +216,9 @@ when comparing route tables snapshots are formatted like:
 }
 ```
 
-The bottom most level keys are:
+The keys to process here can be:
 
+- `default_0.0.0.0/0_ethernet1/3_10.26.129.129`,
 - `virtual-router`,
 - `destination`,
 - `nexthop`,
@@ -241,12 +241,23 @@ the `added` key in the results.
 represented under the `changed` key in the results.
 
 This is a **recursive** method. When calculating changed values, if a value for the key is `dict`, we run the method
-again on that dictionary - we go down one level in the nested structure. We do that to a point where the value is of the
-`str` type. Therefore, when the final comparison results are presented, the `changed` key usually contains a nested
-results structure. This means it contains a dictionary with the `missing`, `added`, and `changed` keys.
+again on that dictionary - we go down one level in the nested structure. We do that to a point where the value is one of
+`str`, `int` type or None. Therefore, when the final comparison results are presented, the `changed` key usually contains
+a nested results structure. This means it contains a dictionary with the `missing`, `added`, and `changed` keys.
 Each comparison perspective contains the `passed` property that immediately informs if this comparison gave any results
 (`False`) or not (`True`).
 
+`properties` can be defined for any level of nested dictionaries which implies:
+
+- Allow comparison of specific parent dictionaries.
+- Skip specific parent dictionaries.
+- Allow comparison/exclusion of specific sub-dictionaries or keys only.
+- If given keys have parent-child relationship then all keys for a matching parent are compared.
+Meaning it doesn’t do an "AND" operation on the given properities for nested dictionaries.
+
+Also note that missing/added keys in parent dictionaries are not reported for comparison when specific keys
+are requested to be compared with the `properties` parameter.
+
 **Example**
 
 Let's assume we want to compare two dictionaries of the following structure:

docs/panos-upgrade-assurance/api/utils.md

@@ -122,17 +122,27 @@ against the list of valid configuration items.
 There are no hardcoded items against which the configuration is checked. This class is used in many places in this package
 and it uses a specific [`dialect`](/panos/docs/panos-upgrade-assurance/dialect).
 
+`ConfigElement` (`str`, `dict`): Type alias for a configuration element in `requested_config` which is either a string or a
+    dictionary with a single key. This alias is being used over the `ConfigParser` class to increase clarification.
+
+:::note
+Configuration elements beginning with an exclamation mark (!) is referred to as `not-element`s in this dialect and it should
+be considered such in any place documented in the `ConfigParser` class. Please refer to the `dialect` documentation for
+details.
+:::
+
 __Attributes__
 
 
-- `_requested_config_names (set)`: Contains only element names of the requested configuration. When no requested configuration is
-    passed (implicit `'all'`), this is equal to `self.valid_elements`.
+- `_requested_config_element_names (set)`: Contains only element names of the requested configuration. When no requested
+    configuration is passed, this is equal to `self.valid_elements` which is like an implicit `'all'`.
+- `_requested_all_not_elements (bool)`: Identifies if requested configurations consists of all `not-element`s.
 
 ### `ConfigParser.__init__`
 
 ```python
 def __init__(valid_elements: Iterable,
-             requested_config: Optional[List[Union[str, dict]]] = None)
+             requested_config: Optional[List[ConfigElement]] = None)
 ```
 
 ConfigParser constructor.
@@ -142,13 +152,16 @@ Introduces some initial verification logic:
 * `valid_elements` is converted to `set` - this way we get rid of all duplicates,
 * if `requested_config` is `None` we immediately treat it as if `all`  was passed implicitly
     (see [`dialect`](/panos/docs/panos-upgrade-assurance/dialect)) - it's expanded to `valid_elements`
-* `_requested_config_names` is introduced as `requested_config` stripped of any element configurations. Additionally, we
-    do verification if elements of this variable match `valid_elements`. An exception is thrown if not.
+* `_requested_config_element_names` is introduced as `requested_config` stripped of any element configurations.
+    Meaning top level keys of nested dictionaries in the `requested_config` are used as element names.
+    Additionally, we do verification if all elements of this variable match `valid_elements`,
+    if they do not, an exception is thrown by default.
+* `_requested_all_not_elements` is set to `True` if all elements of `requested_config` are `not-element`s.
 
 __Parameters__
 
 
-- __valid_elements__ (`iterable`): Valid elements against which we check the requested config.
+- __valid_elements__ (`Iterable`): Valid elements against which we check the requested config.
 - __requested_config__ (`list, optional`): (defaults to `None`) A list of requested configuration items with an optional
     configuration.
 
@@ -157,31 +170,81 @@ __Raises__
 
 - `UnknownParameterException`: An exception is raised when a requested configuration element is not one of the valid elements.
 
-### `ConfigParser._is_element_included`
+### `ConfigParser.is_all_not_elements`
 
 ```python
-def _is_element_included(element: str) -> bool
+@staticmethod
+def is_all_not_elements(config: Iterable[ConfigElement]) -> bool
 ```
 
-Method verifying if a config element is a correct (supported) value.
+Method to check if all config elements are `not-element`s (all exclusive).
+
+__Parameters__
+
+
+- __config__ (`Iterable`): List of config elements.
+
+__Returns__
+
+
+`bool`: `True` if all config elements are `not-element`s (exclusive) or config is empty, otherwise returns `False`.
 
-This method can also handle `not-elements` (see [`dialect`](/panos/docs/panos-upgrade-assurance/dialect)).
+### `ConfigParser.is_element_included`
+
+```python
+@staticmethod
+def is_element_included(element_name: str,
+                        config: Union[Iterable[ConfigElement], None],
+                        all_not_elements_check: bool = True) -> bool
+```
+
+Method verifying if a given element name should be included according to the config.
 
 __Parameters__
 
 
-- __element__ (`str`): The config element to verify. This can be a `not-element`. This parameter is verified against
-    `self.valid_elements` `set`. Key word `'all'` is also accepted.
+- __element_name__ (`str`): Element name to check if it's included in the provided `config`.
+- __config__ (`Iterable`): Config to check against.
+- __all_not_elements_check__ (`bool, optional`): (defaults to `True`) Accept element as included if all the `config` elements are
+    `not-element`s; otherwise it checks if the element is explicitly included without making an
+    [`is_all_not_elements()`](#configparseris_all_not_elements) method call.
 
 __Returns__
 
-`bool`: `True` if the value is correct, `False` otherwise.
+
+`bool`: `True` if element name is included or if all config elements are `not-element`s depending on the
+    `all_not_elements_check` parameter.
+
+### `ConfigParser.is_element_explicit_excluded`
+
+```python
+@staticmethod
+def is_element_explicit_excluded(
+        element_name: str, config: Union[Iterable[ConfigElement],
+                                         None]) -> bool
+```
+
+Method verifying if a given element should be excluded according to the config.
+
+Explicit excluded means the element is present as a `not-element` in the requested config, for example "ntp_sync" is
+excluded explicitly in the following config `["!ntp_sync", "candidate_config"]`.
+
+__Parameters__
+
+
+- __element_name__ (`str`): Element name to check if it's present as a `not-element` in the provided `config`.
+- __config__ (`Iterable`): Config to check against.
+
+__Returns__
+
+
+`bool`: `True` if element is present as a `not-element`, otherwise `False`.
 
 ### `ConfigParser._extract_element_name`
 
 ```python
 @staticmethod
-def _extract_element_name(config: Union[str, dict]) -> str
+def _extract_element_name(element: ConfigElement) -> str
 ```
 
 Method that extracts the name from a config element.
@@ -192,35 +255,109 @@ If a config element is a string, the actual config element is returned. For elem
 __Parameters__
 
 
-- __config__ (`str, dict`): A config element to provide a name for.
+- __element__ (`ConfigElement`): A config element to provide a name for.
 
 __Raises__
 
 
-- `WrongDataTypeException`: Thrown when config does not meet requirements.
+- `WrongDataTypeException`: Thrown when element does not meet requirements.
 
 __Returns__
 
 
 `str`: The config element name.
 
-### `ConfigParser._expand_all`
+### `ConfigParser._iter_config_element_names`
 
 ```python
-def _expand_all() -> None
+@staticmethod
+def _iter_config_element_names(
+        config: Iterable[ConfigElement]) -> Iterator[str]
 ```
 
-Expand key word `'all'` to  `self.valid_elements`.
+Generator for extracted config element names.
+
+This method provides a convenient way to iterate over configuration items with their config element names extracted by
+[`_extract_element_name()`](#configparser_extract_element_name) method.
+
+__Parameters__
+
 
-During expansion, elements from `self.valid_elements` which are already available in `self.requested_config` are skipped.
-This way we do not introduce duplicates for elements that were provided explicitly.
+- __config__ (`Iterable`): Iterable config items as str or dict.
 
-This method directly operates on `self.requested_config`.
+__Returns__
+
+
+`Iterator`: For config element names extracted by [`ConfigParser._extract_element_name()`](#configparser_extract_element_name)
+
+### `ConfigParser._strip_element_name`
+
+```python
+def _strip_element_name(element_name: str) -> str
+```
+
+Get element name with exclamation mark removed if so.
+
+Returns element name removing exclamation mark for a `not-element` config.
+
+__Parameters__
+
+
+- __element_name__ (`str`): Element name.
+
+__Returns__
+
+
+`str`: Element name with exclamation mark stripped of from the beginning.
+
+### `ConfigParser._is_valid_element_name`
+
+```python
+def _is_valid_element_name(element_name: str) -> bool
+```
+
+Method verifying if a config element name is a correct (supported) value.
+
+This method can also handle `not-element`s (see [`dialect`](/panos/docs/panos-upgrade-assurance/dialect)).
+
+__Parameters__
+
+
+- __element_name__ (`str`): The config element name to verify. This can be a `not-element` as well. This parameter is verified
+     against `self.valid_elements` set. Key word `'all'` is also accepted.
+
+__Returns__
+
+
+`bool`: `True` if the value is correct, `False` otherwise.
+
+### `ConfigParser.get_config_element_by_name`
+
+```python
+def get_config_element_by_name(
+        element_name: str) -> Union[ConfigElement, None]
+```
+
+Get configuration element from requested configuration for the provided config element name.
+
+This method returns config element as str or dict from `self.requested_config` for the provided config element name.
+It does not support returning `not-element` of a given config element.
+
+__Parameters__
+
+
+- __element_name__ (`str`): Element name.
+
+__Returns__
+
+
+`ConfigElement`: str if element is provided as string or dict if element is provided as dict with optional configuration in
+    the requested configuration.
 
 ### `ConfigParser.prepare_config`
 
 ```python
-def prepare_config() -> List[Union[str, dict]]
+def prepare_config() -> List[ConfigElement]
 ```
 
 Parse the input config and return a machine-usable configuration.
@@ -232,7 +369,8 @@ This method handles most of the [`dialect`](/panos/docs/panos-upgrade-assurance/
 
 __Returns__
 
-`list`: The parsed configuration.
+
+`List[ConfigElement]`: The parsed configuration.
 
 ### `interpret_yes_no`
 

docs/panos-upgrade-assurance/configuration_details.mdx

@@ -1154,7 +1154,7 @@ Runs comparison of ARP tables snapshots.
 
 parameter | description
 --- | ---
-`properties` | (optional) a set of properties to skip when comparing two ARP table entries, all properties are checked when this parameter is skipped
+`properties` | (optional) a set of properties to skip or include when comparing two ARP table entries, all properties are checked when this parameter is skipped
 `count_change_threshold` | (optional) maximum difference percentage of changed entries in ARP table in both snapshots, skipped when this property is not specified
 
 **Sample configuration**
@@ -1217,12 +1217,12 @@ Compares configuration and the state of IPSec tunnels.
 
 parameter | description
 --- | ---
-`properties` | (optional) a set of properties to skip when comparing two IPSec tunnels, all properties are checked when this parameter is skipped
+`properties` | (optional) a set of properties to skip or include when comparing two IPSec tunnels, all properties are checked when this parameter is skipped
 `count_change_threshold` | (optional) maximum difference percentage of changed entries in IPSec tunnels in both snapshots, skipped when this property is not specified
 
 **Sample configuration**
 
-The following configuration compares the state of IPSec tunnels as
+The following configuration compares only the state of IPSec tunnels as
 captured in snapshots.
 
 This report produces the standardized dictionary.
@@ -1270,7 +1270,7 @@ Compares installed licenses. This report does not only check if we have the same
 
 parameter | description
 --- | ---
-`properties` | (optional) a set of properties to skip when comparing two licenses, all properties are checked when this parameter is skipped
+`properties` | (optional) a set of properties to skip or include when comparing two licenses, all properties are checked when this parameter is skipped
 `count_change_threshold` | (optional) maximum difference percentage of changed licenses in both snapshots, skipped when this property is not specified
 
 **Sample configuration**
@@ -1377,7 +1377,7 @@ Provides a report on differences between Route Table entries. It includes:
 
 parameter | description
 --- | ---
-`properties` | (optional) a set of properties to skip when comparing two routes, all properties are checked when this parameter is skipped
+`properties` | (optional) a set of properties to skip or include when comparing two routes, all properties are checked when this parameter is skipped
 `count_change_threshold` | (optional) maximum difference percentage of changed entries routes in both snapshots, skipped when this property is not specified
 
 **Sample configuration**
@@ -1433,7 +1433,7 @@ Compares configuration and the status of BGP peers.
 
 parameter | description
 --- | ---
-`properties` | (optional) a set of properties to skip when comparing two BGP peers, all properties are checked when this parameter is skipped
+`properties` | (optional) a set of properties to skip or include when comparing two BGP peers, all properties are checked when this parameter is skipped
 `count_change_threshold` | (optional) maximum difference percentage of changed entries in BGP peers in both snapshots, skipped when this property is not specified
 
 **Sample configuration**
@@ -1488,7 +1488,7 @@ Provides a report on differences between FIB Table entries. It includes:
 
 parameter | description
 --- | ---
-`properties` | (optional) a set of properties to skip when comparing two fib entries, all properties are checked when this parameter is skipped
+`properties` | (optional) a set of properties to skip or include when comparing two fib entries, all properties are checked when this parameter is skipped
 `count_change_threshold` | (optional) maximum difference percentage of changed entries for fib routes in both snapshots, skipped when this property is not specified
 
 **Sample configuration**