Feat(eos_validate_state): ANTA New action plugin to generate the reports

ansible_collections/arista/avd/docs/plugins/Modules and action plugins/eos_validate_state_reports.md

@@ -0,0 +1,65 @@
+<!--
+  ~ Copyright (c) 2023 Arista Networks, Inc.
+  ~ Use of this source code is governed by the Apache License 2.0
+  ~ that can be found in the LICENSE file.
+  -->
+
+# eos_validate_state_reports
+
+!!! note
+    Always use the FQCN (Fully Qualified Collection Name) `arista.avd.eos_validate_state_reports` when using this plugin.
+
+!!! warning "This module is in **preview** mode"
+    This module is not guaranteed to have a backwards compatible interface.
+
+Generates validation reports for the eos\_validate\_state role
+
+## Synopsis
+
+The <code>arista.avd.eos\_validate\_state\_reports</code> module, an Ansible Action Plugin, is designed to generate validation reports from the test results produced by the ANTA test framework.
+
+This plugin requires a temporary JSON file for each host in the Ansible play, which contains all test results. The path to this temporary JSON file, used by the plugin, is supplied in the hostvars by the \`eos\_validate\_state\_runner\` plugin.
+
+The plugin offers the following functionalities\:
+  \- It aggregates all test results from every host in the Ansible play and generates a CSV report.
+  \- It produces a detailed Markdown report with various sections presenting key statistics derived from the results.
+  \- The design allows for easy extension to support additional report formats in the future.
+
+## Parameters
+
+| Argument | Type | Required | Default | Value Restrictions | Description |
+| -------- | ---- | -------- | ------- | ------------------ | ----------- |
+| validation_report_csv | bool | optional | True |  | Indicates whether a CSV report should be generated. |
+| validation_report_md | bool | optional | True |  | Indicates whether a Markdown report should be generated. |
+| only_failed_tests | bool | optional | False |  | Determines if the generated reports should include only the failed tests. |
+| csv_report_path | str | optional | None |  | The absolute path where the CSV report will be saved.<br>Required if <code>validation\_report\_csv</code> is set to <code>True</code>. |
+| md_report_path | str | optional | None |  | The absolute path where the Markdown report will be saved.<br>Required if <code>validation\_report\_md</code> is set to <code>True</code>. |
+| cprofile_file | any | optional | None |  | The filename for storing cProfile data, useful for debugging performance issues.<br>Be aware that enabling cProfile can affect performance, so use it only for troubleshooting. |
+
+## Notes
+
+- Enabling the cProfile feature for performance profiling may impact the plugin\'s performance, especially in production environments.
+- The plugin manages temporary files created for processing test results, ensuring clean\-up post\-execution.
+- Hosts marked as not deployed are automatically skipped, and no test results are processed for these hosts.
+
+## See Also
+
+- ANTA website: [https://anta.ninja](https://anta.ninja)<br>Documentation for the ANTA test framework
+
+## Examples
+
+```yaml
+- name: Generate validation reports from ANTA test results
+  arista.avd.eos_validate_state_reports:
+    csv_report_path: "/git_projects/my_avd_project/reports/my-fabric-state.csv"
+    md_report_path: "/git_projects/my_avd_project/reports/my-fabric-state.md"
+    validation_report_csv: true
+    validation_report_md: true
+    only_failed_tests: false
+  delegate_to: localhost
+  run_once: true
+```
+
+## Authors
+
+- Arista Ansible Team (@aristanetworks)

ansible_collections/arista/avd/docs/plugins/Modules and action plugins/eos_validate_state_runner.md

@@ -16,48 +16,56 @@ title: arista.avd.eos_validate_state_runner
 !!! warning "This module is in **preview** mode"
     This module is not guaranteed to have a backwards compatible interface.
 
-Leverage ANTA for eos\_validate\_state role
+Utilizes ANTA for eos\_validate\_state role validation
 
 ## Synopsis
 
 The <code>arista.avd.eos\_validate\_state\_runner</code> module is an Ansible Action Plugin leveraging the ANTA test framework to validate that the generated structured configurations by AVD are applied to the devices and that the deployed network is working correctly.
 
-This plugin expects that the structued\_configs of each device is present in hostvars, otherwise no test will be generated.
+This plugin requires that structured\_configs for each device be present in hostvars\; otherwise, no tests will be executed.
 
-The plugin provides the following capabilities\:
-    \- Generate a per\-device test catalog based on the structured\_configs
-    \- Run the generated tests against each device and generate a report in Markdown and CSV format.
-    \- When using check\_mode, only generate the test catalog and generate a report to preview what would tests be run against each device
-    \- Dumping the per\-device catalog to a file.
-    \- Backward compatibility with existing ansible tags behavior for eos\_validate\_state to filter categories of tests.
+The plugin offers the following capabilities\:
+    \- Generating a per\-device test catalog based on the AVD structured\_config.
+    \- Running generated tests against each device, saving the results in a temporary JSON file,
+      and recording the file path in hostvars for use by the report plugin.
+    \- In check\_mode, only the test catalog is generated, and a report is created to preview the tests that would be run against each device.
+    \- Saving per\-device test catalogs and results to specified directories.
+    \- Maintaining backward compatibility with existing ansible tags for eos\_validate\_state to filter test categories.
 
 ## Parameters
 
 | Argument | Type | Required | Default | Value Restrictions | Description |
 | -------- | ---- | -------- | ------- | ------------------ | ----------- |
-| logging_level | str | False | WARNING | Valid values:<br>- <code>CRITICAL</code><br>- <code>ERROR</code><br>- <code>WARNING</code><br>- <code>INFO</code><br>- <code>DEBUG</code> | Controls the log level for the ANTA library. If unset, the Action plugin will set it to \"WARNING\" |
-| save_catalog | bool | optional | False |  | A boolean to indicate whether or not the catalog should be saved for each device. |
-| device_catalog_output_dir | str | optional | None |  | When <code>save\_catalog</code> is True, this is the directory where the device catalogs will be saved.<br>Required if <em>save\_catalog\=True</em> |
-| skipped_tests | list | optional | None |  | A list of dictionaries containing the categories and tests to skip<br>The keys for the dictionnary are <code>categories</code> and <code>tests</code>. |
-|     category | str | True | None |  | The name of one of the AvdTest categories. e.g., <code>AvdTestHardware</code> |
-|     tests | list | optional | None |  | A list of tests in the category. e.g, <code>VerifyRoutingProtocolModel</code> for <code>AvdTestBGP</code> |
+| logging_level | str | optional | WARNING | Valid values:<br>- <code>CRITICAL</code><br>- <code>ERROR</code><br>- <code>WARNING</code><br>- <code>INFO</code><br>- <code>DEBUG</code> | Sets the log level for the ANTA library. Defaults to \"WARNING\" if not specified. |
+| save_catalog | bool | optional | False |  | Indicates whether to save the test catalog for each device. |
+| save_results | bool | optional | False |  | Indicates whether to save test results in a JSON file for each device. |
+| device_catalog_output_dir | str | optional | None |  | The directory where device test catalogs will be saved.<br>Required if <code>save\_catalog</code> is set to <code>True</code>. |
+| device_results_output_dir | str | optional | None |  | The directory where device test results will be saved.<br>Required if <code>save\_results</code> is set to <code>True</code>. |
+| skipped_tests | list | optional | None |  | A list of dictionaries specifying categories and tests to skip.<br>Each dictionary should have keys <code>category</code> and <code>tests</code>. |
+|     category | str | optional | None |  | The name of an AvdTest category \(e.g., <code>AvdTestHardware</code>\). |
+|     tests | list | optional | None |  | A list of specific tests in the category \(e.g., <code>VerifyRoutingProtocolModel</code> in <code>AvdTestBGP</code>\). |
+| cprofile_file | any | optional | None |  | The filename for storing cProfile data, useful for debugging performance issues.<br>Be aware that enabling cProfile can affect performance, so use it only for troubleshooting. |
 
 ## Notes
 
-- <code>check\_mode</code> is supported for this module and allows to generate a Test Report without running the tests.
+- Enabling the cProfile feature for performance profiling may impact the plugin\'s performance, especially in production environments.
+- The plugin manages the creation and cleanup of temporary JSON files used for storing test results.
+- This module supports <code>check\_mode</code>, allowing the generation of test reports without executing the tests.
 
 ## See Also
 
-- ANTA website: [https://anta.ninja](https://anta.ninja)<br>ANTA documentation
+- ANTA website: [https://anta.ninja](https://anta.ninja)<br>Documentation for the ANTA test framework
 
 ## Examples
 
 ```yaml
-- name: Run eos_validate_state_runner leveraging ANTA
+- name: Execute eos_validate_state_runner leveraging ANTA
   arista.avd.eos_validate_state_runner:
     logging_level: ERROR
-    save_catalog: True
-    eos_validate_state_dir: "/tmp"
+    save_catalog: true
+    save_results: false
+    device_catalog_output_dir: "/git_projects/my_avd_project/intended/test_catalogs"
+    device_results_output_dir: "/git_projects/my_avd_project/reports/test_results"
     skipped_tests:
       - category: AvdTestHardware
       - category: AvdTestBGP

ansible_collections/arista/avd/molecule/eos_validate_state/intended/test_catalogs/dc1-leaf1a-catalog.yml

@@ -215,7 +215,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet3 - MLAG_PEER_dc1-leaf1b_Ethernet3
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet4
@@ -225,7 +225,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet4 - MLAG_PEER_dc1-leaf1b_Ethernet4
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet1
@@ -235,7 +235,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet1 - P2P_LINK_TO_DC1-SPINE1_Ethernet1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet2
@@ -245,7 +245,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet2 - P2P_LINK_TO_DC1-SPINE2_Ethernet1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet8
@@ -255,7 +255,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet8 - DC1-LEAF1C_Ethernet1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Ethernet5
@@ -265,7 +265,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Ethernet5 - dc1-leaf1-server1_PCI1
-      description: Ethernet Interface & Line Protocol == \"up\"
+      description: Ethernet Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Port-Channel3
@@ -275,7 +275,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Port-Channel3 - MLAG_PEER_dc1-leaf1b_Po3
-      description: Port-Channel Interface & Line Protocol == \"up\"
+      description: Port-Channel Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Port-Channel8
@@ -285,7 +285,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Port-Channel8 - DC1-LEAF1C_Po1
-      description: Port-Channel Interface & Line Protocol == \"up\"
+      description: Port-Channel Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Port-Channel5
@@ -295,7 +295,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Port-Channel5 - dc1-leaf1-server1_PortChannel dc1-leaf1-server1
-      description: Port-Channel Interface & Line Protocol == \"up\"
+      description: Port-Channel Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan4093
@@ -305,7 +305,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan4093 - MLAG_PEER_L3_PEERING
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan4094
@@ -315,7 +315,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan4094 - MLAG_PEER
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan11
@@ -325,7 +325,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan11 - VRF10_VLAN11
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan12
@@ -335,7 +335,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan12 - VRF10_VLAN12
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan3009
@@ -345,7 +345,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: 'Vlan3009 - MLAG_PEER_L3_iBGP: vrf VRF10'
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan21
@@ -355,7 +355,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan21 - VRF11_VLAN21
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan22
@@ -365,7 +365,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vlan22 - VRF11_VLAN22
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vlan3010
@@ -375,7 +375,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: 'Vlan3010 - MLAG_PEER_L3_iBGP: vrf VRF11'
-      description: Vlan Interface & Line Protocol == \"up\"
+      description: Vlan Interface & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback0
@@ -385,7 +385,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback0 - EVPN_Overlay_Peering
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback1
@@ -395,7 +395,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback1 - VTEP_VXLAN_Tunnel_Source
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback10
@@ -405,7 +405,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback10 - VRF10_VTEP_DIAGNOSTICS
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Loopback11
@@ -415,7 +415,7 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Loopback11 - VRF11_VTEP_DIAGNOSTICS
-      description: Loopback Interface Status & Line Protocol == \"up\"
+      description: Loopback Interface Status & Line Protocol == 'up'
 - VerifyInterfacesStatus:
     interfaces:
     - interface: Vxlan1
@@ -425,14 +425,9 @@ anta.tests.interfaces:
       categories:
       - Interface State
       custom_field: Vxlan1
-      description: Vxlan Interface Status & Line Protocol == \"up\"
+      description: Vxlan Interface Status & Line Protocol == 'up'
 anta.tests.mlag:
-- VerifyMlagStatus:
-    result_overwrite:
-      categories:
-      - MLAG
-      custom_field: MLAG
-      description: MLAG State active & Status connected
+- VerifyMlagStatus: {}
 anta.tests.routing.bgp:
 - VerifyBGPSpecificPeers:
     address_families: