24. Automated Testing

Networking professionals often overlook the need for testing due to the tedious nature of this task. It takes significant time, because of the lack of tools to perform testing beyond doing it manually device by device. Also, testing configurations before pushing changes to a production environment requires some sort of lab or nonproduction environment. This setup can be very expensive simply because of the cost of the hardware Rackspace, power, and the like. Automation enables a much easier, more effective, and more efficient means to test a network infrastructure. These systems may be in the form of Python scripts or tools that are used to test connectivity among physical devices, or it can be in the form of new tools that enable more efficient access to equipment such as Cisco’s Virtual Internet Routing Lab and Cisco’s DevNet resources

Network test infrastructure:
    When you think of a network lab, more likely than not, you’re thinking of a data center that has network devices that are racked, stacked, and cabled together. In an ideal world, your lab network resembles the production network. Due to costs associated with physical space, power, cooling, and of network hardware, this setup is usually not a possibility. In fact, many organizations do not have any lab devices to test with. How unfortunate is that? Imagine if applications were deployed without proper test infrastructure. This situation would be a nightmare.

    Applications have become much more portable in the past two decades having migrated from bare-metal machines to virtual machines and now to containers. Virtual machines (and containers) changed the way application environments are built and tested. Full environments can be encapsulated, cloned, and replicated, which is exactly the type of infrastructure that is needed for network infrastructure testing

    Labs and testing network equipment cannot rely solely on hardware. While hardware will always have a place for testing. As an example: performance and throughput tests. Most testing is functional, especially as it pertains to network automation. When it comes to testing network automation, all that’s really needed is a software solution that exposes device APIs. What you need are low-cost alternatives for extensive testing that allows network engineers to test and certify new features. These alternatives must also be able to test the various types of APIs that exist on network devices. Also, without having to purchase hardware that requires space, power, and cooling

    Especially as you start testing network APIs on devices and SDN controllers, it does not make sense to deploy hardware first anymore. It’s too time consuming and costly especially when all you need is API access on the devices. When testing hardware performance, you may need high density 10G/100G ports; however, network automation testing has no such requirement

    There are three ways to test Cisco infrastructure without using hardware:
        Network Functions Virtualization — This system is the equivalent to the setup being used for this course
        Cisco Virtual Internet Routing Lab (VIRL) — A test platform offered by Cisco allows network engineers to drag and drop topologies simplifying the process even further for creating a lab environment
        DevNet Sandboxes — As you will see later in this lesson. The Cisco DevNet offers free and hosted devices and labs to test new APIs and products

Network function virtualization (NFV):
    Over the past few years, the industry has been undergoing significant change with how networks are built and managed. One of these such changes (and trends) is Network Functions Virtualization. NFV is about decoupling a network service from hardware and making it available as software either as an application, service, container, or virtual machine. It does not have to be a virtual machine, although that is the most common form factor of devices being deployed. Typically, you would use x86 servers such as the Cisco UCS platform. Many virtual machines from Cisco can be run on both on and off-premises in public cloud environments. For example, you can deploy Cisco CSR 1000V routers directly in AWS and have them connect back into the corporate network using to site to site VPN tunnels. In this case, you would effectively have a branch that is an Amazon virtual private cloud. Using software and NFV for testing puts the focus back on testing and minimizes the amount of time that is required for racking, stacking, and cabling

    Over the past few years, Cisco has been releasing virtual appliances for key network operating systems and controllers. For example, Cisco now offers virtual firewalls, virtual wireless LAN controllers, and SDN controllers. In fact, all devices being used in this course are virtual machines. Each network device and SDN controller that is used are all running as virtual machines. Not all virtual machines are meant for production, though. This feature is a key point to understand as you start using software-based networking solutions for testing. For example, NX-OSv and the ACI emulator are appliances that are meant specifically for testing, while ASAv and the CSR 1000V are production-grade virtual appliances supported by Cisco. Always check the data sheets as sometimes there are hardware-centric features that are not supported in virtual editions (even though they are running the same network operating system)

VIRL:
    Cisco’s Virtual Internet Routing Lab, better known as VIRL, is a powerful network simulation platform. It is meant to be a flexible, all-in-one virtual networking lab. This setup means that there is no need for bulky network equipment and hours of wiring. Also, it is easy to extend your virtual lab in VIRL to the physical world. A few key benefits of VIRL include the following:
       Cisco virtual machines running the same network operating systems as used in Cisco’s physical routers and switches. VIRL comes with a complete set of legal and licensed Cisco images with new OS releases provided regularly

       Powerful GUI for network design and simulation control

   VIRL supports various platforms, and from time to time, Cisco releases updates and new virtual platforms. VIRL also supports third-party VMs, which are typically Linux VMs, but other vendor devices can be run in VIRL as well. In recent days, there has been integration with GitHub providing you with very useful resources for getting started, managing projects, and using VIRL features. Over a dozen third-party VMs listed here: https://learningnetwork.cisco.com/docs/DOC-30476

   In VIRL, the virtual machines run the network operating system but are not representations of a particular hardware platform. This setup means that though features and configurations are the same between the virtual and physical platforms. There are limitations in VIRL regarding data plane activity. Also, virtual machines have no fans, no switch fabric, and no ASIC models. This characteristic is a great advantage as features and the actual network operating system are available in virtual form. However, there is also an inability to test certain attributes, such as throughput

   VIRL is built on top of Openstack and is used to orchestrate the virtual machines, building the topology and managing images. Openstack is normally known as a free and open-source platform for cloud computing, but in very recent days it has been leveraged for providing a virtual infrastructure for various hardware recourses including compute, networking and storage. Openstack uses a web-based dashboard and REST APIs which make Openstack programmable and customizable. This feature is the core of what enables the VIRL network simulation. Openstack provides the foundation for VIRL to be more than a basic network simulation program; instead, it provides the basis for full network virtualization and orchestration

   The VIRL topology service director is used to create the virtual machine, and then the virtual machines are linked (and therefore communicate) with one another using Openstack. In fact, Openstack allows communication not only among virtual machines, but among several of computers running VIRL in a cluster. This communication can be done in a cloud-based hosted platform or locally using Vmware ESXi or on bare-metal servers

   The components of VIRL are VM Maestro, which serves as VIRL’s front end, and the back-end where you actually run your virtual machines. VIRL’s back-end is simply Ubuntu Linux utilizing Openstack and KVM/Qemu. Though there is also a web user interface, VM Maestro controls the actual virtual machines. Cisco utilizes the very same source code they use to run their network devices in order to create virtual machine equivalents to run on standard x86 hardware. This process means that you can design and configure familiar network topologies using the same familiar network operating systems but on any reasonable computer including a laptop or desktop. This system is an incredibly powerful tool to create network simulations to mimic production environments for testing, proofs of concept. You can use your favorite terminal emulator to connect to the network devices in your simulation. With proper configuration, the virtual environment can connect to your physical environment as well

   VIRL allows you to perform almost any function that the physical platforms perform. This feature means that you can configure and test layer 2 and layer 3 connectivity. This option is a very powerful advantage to deploying large amounts of physical hardware. However, because the virtual platforms are almost identical to their physical counterparts, you can perform latency and packet loss tests. Try various access control scenarios, and experiment with new technologies such as containers

   AutoNetkit enables Cisco VIRL to provide you with a very intuitive graphical user interface to create new topologies on demand and point-and-click to drop new virtual devices into a topology. When you drop a new IOSv device into a topology, on the back-end VIRL is working with actual device images and Openstack connectivity. Therefore, AutoNetkit is much more than a simple web front end; it is a powerful automation engine to coordinate predefined variables and incorporate them into your topology. AutoNetkit allows you to very easily build baseline configurations saving you tremendous time. When testing the deployment of a new routing protocol, it is time consuming and wasteful to have to take an hour or more to create the topology and connections. Instead, AutoNetkit generates the topology for you allowing you to immediately get to work. Also, you can simply disable AutoNetkit in order to actually create topologies yourself. Therefore, VIRL gives you tremendous flexibility to test without wasting time or take the time that you want to experiment with devices

   Cisco VIRL topologies are stored as XML files. This process means that topology files are human-readable, editable, and shareable. Especially using VIRL’s GitHub integration, sharing files among peers is extremely easy. Files are saved with the .VIRL extension, and each element in the topology has an XML representation. This representation means that each node, connection, device property, and configuration is self-contained within the topology

   Also, VIRL is front-ended by REST APIs. There are two types of APIs that allow direct access to work with VIRL:

      First, there are the OpenStack APIs. OpenStack is the Cloud Management Platform that coordinates and orchestrates spinning up all required device images into the desired topology. As OpenStack is used, you can access these APIs just as you could in a general OpenStack deployment

      There are also APIs exposed for VIRL-specific operations. For example, common API calls can be issued to start and stop topologies. Python standard requests library can be used just as you have used in previous labs already. With a simple HTTP POST, you can pass the VIRL topology as XML and start the topology. This type of integration can be very attractive for organizations looking to deploy Continuous Integration pipelines

DevNet:
    The Cisco DevNet team continues to grow ever year empowering network engineers to learn more about network automation and device programmability. DevNet is the Developer Program from Cisco and provides tools that help you produce applications to sell to Cisco customers, and use Cisco APIs to enhance or manage your existing Cisco network. DevNet is much more than a simple website, but rather a fully integrated developer program consisting of a website, an interactive developer community, coordinated developer tools, integrated discussion forums, and sandboxes. You will even find a dedicated DevNet zone at Cisco Live. The DevNet developer program includes support for much more than networking – it includes content on automation is it pertains to Cisco networking, collaboration, compatibility Testing, IoT, Cloud, Security, and Datacenter. There is a plethora of content that is geared towards hands-on labs. Just a sampling of what’s available includes support for APIC-EM, RESTCONF,NETCONF, YANG, Various APIs, Cisco Open SDN Controller, Jabber, Webex, and InterCloud Fabric

    DevNet sandbox: The DevNet team offers sandbox environments that lower the barrier to entry for getting hands-on access to equipment for testing new platforms and APIs. Various sandbox types exist across all technologies. Certain sandboxes exist that are always-on while other sandboxes need to be reserved. accessed at https://developer.cisco.com

    In addition to sandbox environments, DevNet offers Learning Labs. These labs offer step by step directions on how to use newer technologies as well. You will walk through Sandboxes and Learning Labs in upcoming Discovery Lab to easily compare and contrast them. available at https://learninglabs.cisco.com

    DevNet also has a GitHub community. You will find code samples (including samples that are used in Learning Labs), scripts, libraries, YANG models, and various open source projects hosted on the DevNet GitHub site. You should get familiar with the current projects and be sure to follow this community to be aware of when new GitHub repositories are created. https://github.com/ciscoDevNet

Network testing:
    Using platforms such as VIRL, DevNet, and virtual machines lower the barrier to entry for network testing, but after testing, you need to take action in your production network. While automation can be scary, a great way to start with automation is with automated testing

    Network testing is diverse. How do you validate a change was successful on the network today? Do you gather the output of show commands manually and compare the output manually? If you’re adding a new branch site with three subnets, you know that this site should be able to reach all other 10 sites. All existing sites should be able to reach each the three new subnets. If done properly, you can write tests that should fail before the deployment. When you run the tests after the deployment, they should all pass

    Do you think these tests and checks could be done more efficiently?

        Configuration and compliance tests: How do you verify that particular features are configured properly? How do you verify that all devices have the proper AAA or passwords configured?

        Ephemeral data tests: Referring to the dynamic state of network devices. As an example: neighbors, adjacencies, routes, mac addresses, ARP cache, and the like. How do you verify that the real-time state is correct? All neighbors being seen? Do you see a drop in OSPF routes?

        Reachability tests: How do you verify all sites and subnets, in all VRFs, has access to the systems they should have access to?

        Performance tests: For those organizations that do have spare hardware in the lab or DR site. How do you verify that a change on a device does not significantly affect the performance? As an example: Turning on QOS, multicast, BGP, and IPSec all at the same time

    For these examples, you are better off automating. The test can then tell you where there are diffs (or deviations) from what you think should be there

    When it comes to testing, there are a few different approaches you can take. The first approach is to test the network manually using utilities like ping and traceroute device by device. Another option is to build custom a custom scripts and tools from scratch, or leveraging an existing platform such as Ansible

    The right side of the figure, illustrates how Ansible can be used to perform validation of OS version across one or more devices. With changing only an inventory file, that one task can verify that the correct version of NX-OS is installed on all your Nexus switches. To do this procedure the old way, would be a tedious and manual endeavor that could take hours or days depending on the size of the network
        - name: show version
            nxos_command:
              commands:
                - show version
              username: cisco
              password: cisco
              host: "{{ inventory_hostname }}"
            register: output
          - assert:
              that:
                - "'7.3(1)D1(1)' in output['response'][0]"

    The single most important point is to realize that the role automation can have on an organization is much larger than managing and deploying configuration more quickly. Not every team needs to be deploying VLANs faster per se, but every team can use automation (in some fashion) to make more predictable changes. This process minimizes change errors and increases the ability to perform tests. In fact, the amount of work necessary to test network configurations manually is significant and tedious enough that testing is disregarded altogether

Unit tests:
    Unit tests are an advanced concept in programming and primarily for full-time software engineers, but you need to review this concept at least from a high level. Unit tests are how software engineers can test specific functions and features of a given application, library, or script

    When working with Python, a common test framework is unittest. There are several components of a test framework, which include the following:

        A test fixture represents the preparation that is needed to perform one or more tests, and any associate cleanup actions. This preparation may involve, for example, creating the instance of a network device

        A test case is the smallest unit of testing. It checks for a specific response to a particular set of inputs. unittest provides a base class, TestCase, which may be used to create new test cases

        A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together

        A test runner is a component that orchestrates the execution of tests and provides the outcome to the user. The runner may use a graphical interface, a textual interface, or return a special value to indicate the results of executing the tests

    As you can see, unit tests are another way that you can run checks on your network before and after you make changes. Writing test cases ensure that you know what the expected output will be. The key with writing tests is to ensure that you run them before and after the change in order to determine the change was deployed correctly