Feat(eos_validate_state): ANTA New action plugin to generate the reports

ansible_collections/arista/avd/docs/plugins/Modules_and_action_plugins/eos_validate_state_reports.md

@@ -0,0 +1,69 @@
+---
+# This title is used for search results
+title: arista.avd.eos_validate_state_reports
+---
+<!--
+  ~ Copyright (c) 2023 Arista Networks, Inc.
+  ~ Use of this source code is governed by the Apache License 2.0
+  ~ that can be found in the LICENSE file.
+  -->
+
+# eos_validate_state_reports
+
+!!! note
+    Always use the FQCN (Fully Qualified Collection Name) `arista.avd.eos_validate_state_reports` when using this plugin.
+
+!!! warning "This module is in **preview** mode"
+    This module is not guaranteed to have a backwards compatible interface.
+
+Generates validation reports for the eos\_validate\_state role
+
+## Synopsis
+
+The <code>arista.avd.eos\_validate\_state\_reports</code> module is an Ansible Action Plugin designed to generate validation reports from the test results produced by the ANTA test framework.
+
+This plugin requires a JSON file for each host in the Ansible play, containing all test results. The JSON file is created automatically by the <code>eos\_validate\_state\_runner</code> plugin and is saved in the test results directory with the following naming convention <code>\<inventory\_hostname\>\-results.json</code>.
+
+The plugin offers the following functionalities\:
+  \- It aggregates all test results from every host in the Ansible play and generates a CSV report.
+  \- It produces a detailed Markdown report with various sections presenting key statistics derived from the results.
+  \- The design allows for easy extension to support additional report formats in the future.
+
+## Parameters
+
+| Argument | Type | Required | Default | Value Restrictions | Description |
+| -------- | ---- | -------- | ------- | ------------------ | ----------- |
+| validation_report_csv | bool | optional | True |  | Indicates whether a CSV report should be generated. |
+| validation_report_md | bool | optional | True |  | Indicates whether a Markdown report should be generated. |
+| only_failed_tests | bool | optional | False |  | Determines if the generated reports should include only the failed tests. |
+| csv_report_path | str | optional | None |  | The absolute path where the CSV report will be saved.<br>Required if <code>validation\_report\_csv</code> is set to <code>True</code>. |
+| md_report_path | str | optional | None |  | The absolute path where the Markdown report will be saved.<br>Required if <code>validation\_report\_md</code> is set to <code>True</code>. |
+| test_results_dir | any | optional | None |  | The directory where the test results JSON file for each host will be saved. |
+| cprofile_file | any | optional | None |  | The filename for storing cProfile data, useful for debugging performance issues.<br>Be aware that enabling cProfile can affect performance, so use it only for troubleshooting. |
+
+## Notes
+
+- Enabling the cProfile feature for performance profiling may impact the plugin\'s performance, especially in production environments.
+- Hosts with <code>is\_deployed</code> is False are automatically skipped, and no test results are processed for these hosts.
+
+## See Also
+
+- ANTA website: [https://anta.ninja](https://anta.ninja)<br>Documentation for the ANTA test framework
+
+## Examples
+
+```yaml
+- name: Generate validation reports from ANTA test results
+  arista.avd.eos_validate_state_reports:
+    csv_report_path: "/my_avd_project/reports/my-fabric-state.csv"
+    md_report_path: "/my_avd_project/reports/my-fabric-state.md"
+    validation_report_csv: true
+    validation_report_md: true
+    only_failed_tests: false
+  delegate_to: localhost
+  run_once: true
+```
+
+## Authors
+
+- Arista Ansible Team (@aristanetworks)

ansible_collections/arista/avd/docs/plugins/Modules_and_action_plugins/eos_validate_state_runner.md

@@ -16,48 +16,55 @@ title: arista.avd.eos_validate_state_runner
 !!! warning "This module is in **preview** mode"
     This module is not guaranteed to have a backwards compatible interface.
 
-Leverage ANTA for eos\_validate\_state role
+Uses ANTA for eos\_validate\_state role
 
 ## Synopsis
 
 The <code>arista.avd.eos\_validate\_state\_runner</code> module is an Ansible Action Plugin leveraging the ANTA test framework to validate that the generated structured configurations by AVD are applied to the devices and that the deployed network is working correctly.
 
-This plugin expects that the structued\_configs of each device is present in hostvars, otherwise no test will be generated.
+This plugin requires a valid structured configuration for each device in the hostvars\; otherwise, some tests will not be generated.
 
-The plugin provides the following capabilities\:
-    \- Generate a per\-device test catalog based on the structured\_configs
-    \- Run the generated tests against each device and generate a report in Markdown and CSV format.
-    \- When using check\_mode, only generate the test catalog and generate a report to preview what would tests be run against each device
-    \- Dumping the per\-device catalog to a file.
-    \- Backward compatibility with existing ansible tags behavior for eos\_validate\_state to filter categories of tests.
+The plugin offers the following capabilities\:
+    \- Generating a per\-device test catalog based on the AVD structured\_config.
+    \- Running generated tests against each device, saving the results in a single JSON file per\-device.
+    \- In check\_mode, only the test catalog is generated, and a report is created to preview the tests that would be run against each device.
+    \- Saving per\-device test catalogs and results in specified directories for use by the <code>eos\_validate\_state\_reports</code> plugin.
+    \- Maintaining backward compatibility with existing ansible tags for eos\_validate\_state to filter test categories.
 
 ## Parameters
 
 | Argument | Type | Required | Default | Value Restrictions | Description |
 | -------- | ---- | -------- | ------- | ------------------ | ----------- |
-| logging_level | str | False | WARNING | Valid values:<br>- <code>CRITICAL</code><br>- <code>ERROR</code><br>- <code>WARNING</code><br>- <code>INFO</code><br>- <code>DEBUG</code> | Controls the log level for the ANTA library. If unset, the Action plugin will set it to \"WARNING\" |
-| save_catalog | bool | optional | False |  | A boolean to indicate whether or not the catalog should be saved for each device. |
-| device_catalog_output_dir | str | optional | None |  | When <code>save\_catalog</code> is True, this is the directory where the device catalogs will be saved.<br>Required if <em>save\_catalog\=True</em> |
-| skipped_tests | list | optional | None |  | A list of dictionaries containing the categories and tests to skip<br>The keys for the dictionnary are <code>categories</code> and <code>tests</code>. |
-|     category | str | True | None |  | The name of one of the AvdTest categories. e.g., <code>AvdTestHardware</code> |
-|     tests | list | optional | None |  | A list of tests in the category. e.g, <code>VerifyRoutingProtocolModel</code> for <code>AvdTestBGP</code> |
+| logging_level | str | optional | WARNING | Valid values:<br>- <code>CRITICAL</code><br>- <code>ERROR</code><br>- <code>WARNING</code><br>- <code>INFO</code><br>- <code>DEBUG</code> | Sets the log level for the ANTA library. Defaults to \"WARNING\" if not specified. |
+| save_catalog | bool | optional | False |  | Indicates whether to save the test catalog for each device. |
+| device_catalog_path | str | optional | None |  | The absolute path where the device test catalog will be saved.<br>Required if <code>save\_catalog</code> is set to <code>True</code>. |
+| test_results_dir | str | optional | None |  | The directory where the test results JSON file for each host will be saved. |
+| skipped_tests | list | optional | None |  | A list of dictionaries specifying categories and, optionally, tests to skip.<br>Each dictionary must have a key <code>category</code> and can optionally include a <code>tests</code> key. |
+|     category | str | optional | None |  | The name of an AvdTest category \(e.g., <code>AvdTestHardware</code>\). |
+|     tests | list | optional | None |  | An optional list of specific tests in the category to skip \(e.g., <code>VerifyRoutingProtocolModel</code> in <code>AvdTestBGP</code>\).<br>If not specified, all tests in the category are considered.<br>For a complete list of available tests, see \[link to the test list\]\(https\://avd.sh/en/stable/roles/eos\_validate\_state/ANTA\-Preview.html\#test\-categories\). |
+| cprofile_file | any | optional | None |  | The filename for storing cProfile data, useful for debugging performance issues.<br>Be aware that enabling cProfile can affect performance, so use it only for troubleshooting. |
 
 ## Notes
 
-- <code>check\_mode</code> is supported for this module and allows to generate a Test Report without running the tests.
+- Enabling the cProfile feature for performance profiling may impact the plugin\'s performance, especially in production environments.
+- The plugin manages the creation of JSON files, which are used for storing test results. For each device, one JSON file containing all results is saved in the test results directory.
+- The file naming convention is hard coded as \"\<inventory\_hostname\>\-results.json\" and cannot be changed. This ensures that the report plugin can properly retrieve the files.
+- This module supports <code>check\_mode</code>, allowing the generation of test reports without executing the tests.
+- Regardless of whether they are running in <code>check\_mode</code> or not, the reports are generated by the <code>eos\_validate\_state\_reports</code> plugin.
 
 ## See Also
 
-- ANTA website: [https://anta.ninja](https://anta.ninja)<br>ANTA documentation
+- ANTA website: [https://anta.ninja](https://anta.ninja)<br>Documentation for the ANTA test framework
 
 ## Examples
 
 ```yaml
-- name: Run eos_validate_state_runner leveraging ANTA
+- name: Execute eos_validate_state_runner leveraging ANTA
   arista.avd.eos_validate_state_runner:
     logging_level: ERROR
-    save_catalog: True
-    eos_validate_state_dir: "/tmp"
+    save_catalog: true
+    device_catalog_path: "/my_avd_project/intended/test_catalogs/{{ inventory_hostname }}-catalog.yml"
+    test_results_dir: "/my_avd_project/reports/test_results"
     skipped_tests:
       - category: AvdTestHardware
       - category: AvdTestBGP

ansible_collections/arista/avd/molecule/eos_validate_state/intended/test_catalogs/dc1-leaf1a-catalog.yml

@@ -237,7 +237,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet3 - MLAG_PEER_dc1-leaf1b_Ethernet3
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet4
@@ -247,7 +247,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet4 - MLAG_PEER_dc1-leaf1b_Ethernet4
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet1
@@ -257,7 +257,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet1 - P2P_LINK_TO_DC1-SPINE1_Ethernet1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet2
@@ -267,7 +267,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet2 - P2P_LINK_TO_DC1-SPINE2_Ethernet1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet8
@@ -277,7 +277,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet8 - DC1-LEAF1C_Ethernet1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet5
@@ -287,7 +287,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet5 - dc1-leaf1-server1_PCI1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Port-Channel3
@@ -297,7 +297,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Port-Channel3 - MLAG_PEER_dc1-leaf1b_Po3
-      description: Port-Channel Interface & Line Protocol == \"up\"
+      description: Port-Channel Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Port-Channel8
@@ -307,7 +307,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Port-Channel8 - DC1-LEAF1C_Po1
-      description: Port-Channel Interface & Line Protocol == \"up\"
+      description: Port-Channel Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Port-Channel5
@@ -317,7 +317,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Port-Channel5 - dc1-leaf1-server1_PortChannel dc1-leaf1-server1
-      description: Port-Channel Interface & Line Protocol == \"up\"
+      description: Port-Channel Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan4093
@@ -327,7 +327,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan4093 - MLAG_PEER_L3_PEERING
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan4094
@@ -337,7 +337,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan4094 - MLAG_PEER
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan11
@@ -347,7 +347,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan11 - VRF10_VLAN11
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan12
@@ -357,7 +357,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan12 - VRF10_VLAN12
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan3009
@@ -367,7 +367,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: 'Vlan3009 - MLAG_PEER_L3_iBGP: vrf VRF10'
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan21
@@ -377,7 +377,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan21 - VRF11_VLAN21
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan22
@@ -387,7 +387,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan22 - VRF11_VLAN22
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan3010
@@ -397,7 +397,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: 'Vlan3010 - MLAG_PEER_L3_iBGP: vrf VRF11'
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback0
@@ -407,7 +407,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback0 - EVPN_Overlay_Peering
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback1
@@ -417,7 +417,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback1 - VTEP_VXLAN_Tunnel_Source
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback10
@@ -427,7 +427,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback10 - VRF10_VTEP_DIAGNOSTICS
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback11
@@ -437,7 +437,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback11 - VRF11_VTEP_DIAGNOSTICS
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vxlan1
@@ -447,14 +447,9 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vxlan1
-      description: Vxlan Interface Status & Line Protocol == \"up\"
+      description: Vxlan Interface Status & Line Protocol == 'up'
 anta.tests.mlag:
-- VerifyMlagStatus:
-    result_overwrite:
-      categories:
-      - MLAG
-      custom_field: MLAG
-      description: MLAG State active & Status connected
+- VerifyMlagStatus: {}
 anta.tests.routing.bgp:
 - VerifyBGPSpecificPeers:
     address_families: