Feat(eos_validate_state): Add ANTA integration to eos_validate_state role

[gmuloc]

Change Summary

Integration of ANTA Python framework to eos_validate_state role.

Tasks List

• (hardware) Validate environment (power supplies status)
• (hardware) Validate environment (fan status).
• (hardware) Validate environment (temperature).
• (hardware) Validate transceivers manufacturer.
• (NTP) Validate NTP status.
• (interface_state) Validate Ethernet interfaces admin and operational status.
• (interface_state) Validate Port-Channel interfaces admin and operational status.
• (interface_state) Validate VLAN interfaces admin and operational status.
• (interface_state) Validate VXLAN interfaces admin and operational status.
• (interface_state) Validate Loopback interfaces admin and operational status.
• (lldp_topology_fqdn) Validate LLDP topology when there is a domain name configured.
• (lldp_topology_no_fqdn) Validate LLDP topology when there is no domain name configured.
• (MLAG) Validate MLAG status.
• (ip_reachability) Validate IP reachability (on directly connected interfaces).
• (loopback_reachability) Validate loopback reachability (between devices).
• (bgp_check) Validate ArBGP is configured and operating.
• (bgp_check) Validate IP BGP and BGP EVPN sessions state.
• (reload_cause) Validate last reload cause. (Optional)
  • Change of behavior as now there is a new reload cause that is valid: https://github.com/arista-netdevops-community/anta/blob/main/anta/tests/system.py#L72C1-L75C11
• (routing_table) Validate remote Lo0 addresses and remote Lo1 addresses are in the routing table (based on devices type).
• schema -> probably not needed as part of this PR except if we want to add the 3 accepted_ values - later discussion
• make eos_validate_state agnostic of eos_designs -> by adding knobs in structured_config where it is required eos_validate_state should not rely on eos_designs variable type #1047 - will be addressed after this PR
• discuss mode strict behavior (vs mode loose) - not implemented for anta mode
• Usage of display (ansible future proofing) ? Do we need all of them that exist today or are some present only because of the impossibility to remove "skipped" messages on Ansible -> make it a logger
• Release tag for anta - with CACHE
• Use check flag in ansible
• module documentation -> DONE need review - the module is considered as preview
• eos_validate_state preview validation - make the "feature" a preview for first release
  • add warning to state that loose mode is ignored
  • Add all input variables as per documented in the current overview document
• testing current eos_validate_state and anta based eos_validate_test and make sure there is no change - document the changes for the tests
• Use anta variable instead of tag for running eos_validate_state.
• Fixes Feat(eos_validate_state): Detailed report for tests to be executed. #3138

Postponed Tasks List

• fix loopback_reachability tags support when running anta -> PR for existing validate state + rebase.
• Remove even further the ties to ansible in the plugins_utils
• Hardware tests are now grouped together in the final report. Will need to add a functionality in ANTA to get more specific results per test.
• Adding unit tests for module and Handler
• Fix(eos_validate_state): When an interface is shutdown, VerifyReachability tests are still created for the peer IP address on that interface. #3182
• Fix(eos_validate_state): When an interface is shutdown, the BGP peer check VerifyBGPSpecificPeers is still created for the peer on that interface. #3183
• Fix(eos_validate_state): In my tests, when I set dc2-leaf1c and dc2-leaf2c (the L2 leave) and set shutdown_interfaces_towards_undeployed_peers: True #3188
• Fix(eos_validate_state): VerifyRoutingProtocolModel is being run on devices that do not have BGP configuration which is not the actual behavior of eos_validate_state. #3189
• Fix(eos_validate_state): Take the verbosity level of ansible to select the level of logging. #3190

Component(s) name

arista.avd.eos_validate_state

Proposed changes

• eos_validate_state_runner.py is an action plugin to run the ANTA framework. It calls the get_anta_results function to grab the test results.
• get_anta_results.py is responsible of creating the ANTA tests catalog and the other required objects to run ANTA.
• Connections to the devices are made with the regular Ansible httpapi connection plugin.
• Results from ANTA are returned as JSON and registered as the anta_results variable in Ansible.
• ANTA related tasks have been added to eos_validate_state tasks/main.yml and tagged anta to keep the legacy eos_validate_state working.
• A new task has been added to tasks/reports.yml to generate ANTA results, leveraging yaml_templates_to_facts plugin and template/generate_anta_report_results.j2 template specific to ANTA results but keeping the same format.
• The following Python classes have been created to replace the existing eos_validate_state tests:

__all__ = [
    "AvdTestP2PIPReachability",
    "AvdTestLoopback0Reachability",
    "AvdTestLLDPTopology",
    "AvdTestInbandReachability",
    "AvdTestHardware",
    "AvdTestMLAG",
    "AvdTestNTP",
    "AvdTestReloadCause",
    "AvdTestInterfacesState",
    "AvdTestRoutingTable",
    "AvdTestBGP",
]

How to test

• Install the main branch of anta:
  pip install git+https://github.com/arista-netdevops-community/anta.git
• eos_validate_state can be used as usual, with --tags anta if you want to run the role with ANTA:
  ansible-playbook playbooks/fabric-validate.yaml --tags anta
  Not providing the anta tag will run the regular eos_validate_state
• Legacy Ansible tags are also supported if you want to run/skip tests:
  ansible-playbook playbooks/fabric-validate.yaml --tags anta,routing_table

The variable skipped_tests can now be used for running/skipping tests:

skipped_tests:
    AvdTestHardware:

You can also decide to skip specific subtests (ANTA test) for more granularity:

skipped_tests:
    AvdTestHardware:
        - VerifyTransceiversTemperature

Checklist

User Checklist

• N/A

Repository Checklist

• My code has been rebased from devel before I start
• I have read the CONTRIBUTING document.
• My change requires a change to the documentation and documentation have been updated accordingly.
• I have updated molecule CI testing accordingly. (check the box if not applicable)

[gmuloc]

Remove anta_ from the module parameters

[chetryan]

In my tests, when I set dc2-leaf1c and dc2-leaf2c (the L2 leave) and set shutdown_interfaces_towards_undeployed_peers: True, I noticed:

• the interface check on interfaces towards the L2 Leafs were set to check for 'down' status as expected
• The LLDP neighborchip check on L3 Leafs still expected to find the L2 Leaf downstream I think this is a bug

[chetryan]

Some changes to the documentation.

[chetryan]

When an interface is shutdown, I can still see VerifyReachability tests created for the peer IP address on that interface.
This is an existing behaviour in current eos_validate_state that has caused confusion/grip about false positives before as well.

[chetryan]

Similarly - when an interface is shutdown, the BGP peer check VerifyBGPSpecificPeers is still created for the peer on that interface. Is this in scope to be solved in this PR ? Without this, we would still have false positive when some devices are not still deployed.

[chetryan]

When the devices are not present, I see that the eos_validate_state was successful , even though these devices are not reachable. I would expect that the task Run eos_validate_state_runner leveraging ANTA would fail for unreachable devices.

[carl-baillargeon]

VerifyRoutingProtocolModel is being run on devices that do not have BGP configuration which is not the actual behavior of eos_validate_state.

[carl-baillargeon]

When the devices are not present, I see that the eos_validate_state was successful , even though these devices are not reachable. I would expect that the task Run eos_validate_state_runner leveraging ANTA would fail for unreachable devices.

@chetryan Fixed in dd1c8c0

The task should fail and stop the play if the device is unreachable or there is a connection problem (e.g. wrong password or SSL problem).

Also, if for some reason we lose connectivity to the device during the tests, you should see a log with the failed commands and the tests associated with the failed commands will be marked as FAIL in the report.

It would be great if you can confirm these behaviors on your end.

Thanks

[carlbuchmann]

LGTM, great job @carl-baillargeon @gmuloc

[gmuloc]

Follow up to this PR is tracked in avd-internal repo: https://github.com/aristanetworks/avd-internal/issues/119