Skip to content

Commit

Permalink
Initial commit for training
Browse files Browse the repository at this point in the history
  • Loading branch information
@jeaubin
jeaubin committed Dec 22, 2020
1 parent 835a437 commit a2fb31c
Show file tree
Hide file tree
Showing 135 changed files with 120,015 additions and 0 deletions.
22 changes: 22 additions & 0 deletions 1-TestPlan/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Testplan

We've just received our new test plan, it begins like this:

## Testcase - Shut no shut BGP

1. Have BGP configuration on your devices
2. Shut BGP
3. Verify it has been shut
4. Unshut it
5. Verify it is back to its original state

We want to automate this test plan following these guidelines:

* Make it independent on specific device, topology, or hostname
* Make it independent on the device configuration
* Make the log readable - easy to understand the failures
* divide the actions
* log useful information
* Be able to re-use this test in combination with other test

> Good design and coding practices pays off big time in the long run
185 changes: 185 additions & 0 deletions 2-Manual/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
# Manual

Before starting with our automation, we can first test it on the device
manually.

## The Typical... Way

Telnet to the device and issue these commands

```text
show bgp process vrf all
router bgp <bgp id>
shutdown
show bgp process vrf all
router bgp <bgp id>
no shutdown
show bgp process vrf all
```

The nature of these commands shows that you are trying to accomplish the
following:

- **Step 1:** study what was there before

- **Step 2:** perform the action

- **Step 3:** verify what has changed and is it expected

- **Step 4:** perform second action

- **Step 5:** Make sure it recovered as expected

These steps requires human interaction and expertise to determine what has
changed and not just look to what is expected.

> What if something else changes that was not anticipated by these steps?
As engineers, we write automated scripts for a few reasons:

* Make sure our features react as expected
* Make sure there are no unexpected changes
* We don't want to manually test the samething every day: it's too slow and
too repetitive
* We need to go fast!
* We want the machine to analyze the data and tell us what has changed
* Find bugs

We should not be looking only at the expected (typical scenario). Rather,
also consider the surrounding aspects as well: the unexpected.

As an exercise, let's see what has changed:

We have three snapshot here

- First one is the initial state
- Second one is the modified state with Bgp shutdown
- Third one is the reverted state with Bgp unshut

```
# Initial
mock_device_cli --os nxos --mock_data_dir $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/initial/playback/nxos --state execute
# Bgp Shutdown
mock_device_cli --os nxos --mock_data_dir $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/modified/playback/nxos --state execute
# Bgp Unshut
mock_device_cli --os nxos --mock_data_dir $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/reverted/playback/nxos --state execute
```

Even though is a 1 line change, it still remains hard to find all the issues.


# pyATS Cli | Manual Automation way

| Note: As we are using a mock device, no need to perform the action on the device, just skip Step 2 and 5.

- **Step 1:** take a snapshot
```bash
pyats parse "show bgp process vrf all" --testbed-file $VIRTUAL_ENV/testing-bootcamp/tb.yaml --output initial --device nx-osv-1 --replay $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/initial/record/
```

- **Step 2:** perform the action
```text
router bgp <bgp id>
shutdown
```

- **Step 3:** take a snapshot
```bash
pyats parse "show bgp process vrf all" --testbed-file $VIRTUAL_ENV/testing-bootcamp/tb.yaml --output modified --device nx-osv-1 --replay $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/modified/record/
```

- **Step 4:** compare the snapshot
```bash
pyats diff initial modified --output modified_diff
```

- **Step 5:** perform second action
```text
router bgp <bgp id>
no shutdown
```

- **Step 6:** take a snapshot
```bash
pyats parse "show bgp process vrf all" --testbed-file $VIRTUAL_ENV/testing-bootcamp/tb.yaml --output recovered --device nx-osv-1 --replay $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/reverted/record/
```

- **Step 7:** compare the snapshot
```bash
pyats diff initial recovered --output recovered_diff
```

| Note: The `--record` is only needed as we are using mocked devices


This way, you are being told what has changed at each step: instead of relying
on human expertise (the probablity of error) on finding the issues, you can
make educated decisions based on the automation output instead: focus your
attention where it matters.

You are being told of all the changes that has happened, so that we can
revert to the original output and ensure nothing else was modified!

Take a look in the `output/output.txt` directory for the flow and all the log!

We can see all the values which changed; this is more than expected! Would we
have though about all of those?

```text
> cat modified_diff/diff_nx-osv-1_show-bgp-process-vrf-all_parsed.txt
--- initial/nx-osv-1_show-bgp-process-vrf-all_parsed.txt
+++ modified/nx-osv-1_show-bgp-process-vrf-all_parsed.txt
+bgp_protocol_state: shutdown
-bgp_protocol_state: running
+bytes_used: 100
-bytes_used: 200
+num_attr_entries: 1
-num_attr_entries: 2
vrf:
default:
address_family:
ipv4 unicast:
peers:
1:
+ active_peers: 0
- active_peers: 1
+ paths: 1
- paths: 2
+ routes: 1
- routes: 2
+ num_established_peers: 0
- num_established_peers: 1
```

You can also parse multiple commands to see this action affect the operation
state of the device.


Can we go one level deeper?

```
1. pyats learn bgp --testbed-file $VIRTUAL_ENV/testing-bootcamp/tb.yaml --output bgp_initial --device nx-osv-1 --replay $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/initial/record/
2. router bgp <bgp id>
shutdown
3. pyats learn bgp --testbed-file $VIRTUAL_ENV/testing-bootcamp/tb.yaml --output bgp_modified --device nx-osv-1 --replay $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/modified/record/
4. pyats diff bgp_initial bgp_modified --output bgp_modified_diff
5. router bgp <bgp id>
no shutdown
6. pyats learn bgp --testbed-file $VIRTUAL_ENV/testing-bootcamp/tb.yaml --output bgp_recovered --device nx-osv-1 --replay $VIRTUAL_ENV/testing-bootcamp/mocked_devices/2-manual/reverted/record/
7. pyats diff bgp_initial bgp_recovered --output bgp_recovered_diff
```

Now instead of just focusing on only 1 command, it learns BGP and related commands, and compare these.

https://pubhub.devnetcloud.com/media/genie-feature-browser/docs/#/models
96 changes: 96 additions & 0 deletions 2-Manual/output/initial/connection_nx-osv-1.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
[2019-04-23 17:08:03,573] +++ nx-osv-1 logfile initial/connection_nx-osv-1.txt +++
[2019-04-23 17:08:03,573] +++ Unicon plugin nxos +++
[2019-04-23 17:08:03,578] +++ connection to spawn: telnet 172.25.192.90 17002, id: 4620279144 +++
[2019-04-23 17:08:03,579] connection to nx-osv-1
Trying 172.25.192.90...
Connected to asg-virl-ubuntu.cisco.com.
Escape character is '^]'.

nx-osv-1#
[2019-04-23 17:08:04,468] +++ initializing handle +++
[2019-04-23 17:08:04,469] +++ nx-osv-1: executing command 'term length 0' +++
term length 0
nx-osv-1#
[2019-04-23 17:08:04,646] +++ nx-osv-1: executing command 'term width 511' +++
term width 511
nx-osv-1#
[2019-04-23 17:08:04,936] +++ nx-osv-1: executing command 'terminal session-timeout 0' +++
terminal session-timeout 0
nx-osv-1#
[2019-04-23 17:08:05,122] +++ nx-osv-1: config +++
config term
Enter configuration commands, one per line. End with CNTL/Z.
nx-osv-1(config)# no logging console
nx-osv-1(config)# line console
nx-osv-1(config-console)# exec-timeout 0
nx-osv-1(config-console)# terminal width 511
nx-osv-1(config-console)# end
nx-osv-1#
[2019-04-23 17:08:06,714] +++ nx-osv-1: executing command 'show bgp process vrf all' +++
show bgp process vrf all

BGP Process Information
BGP Process ID : 8610
BGP Protocol Started, reason: : configuration
BGP Protocol Tag : 65000
BGP Protocol State : Running
BGP MMODE : Not Initialized
BGP Memory State : OK
BGP asformat : asplain

BGP attributes information
Number of attribute entries : 2
HWM of attribute entries : 2
Bytes used by entries : 200
Entries pending delete : 0
HWM of entries pending delete : 0
BGP paths per attribute HWM : 1
BGP AS path entries : 0
Bytes used by AS path entries : 0

Information regarding configured VRFs:

BGP Information for VRF default
VRF Id : 1
VRF state : UP
Router-ID : 10.2.2.2
Configured Router-ID : 10.2.2.2
Confed-ID : 0
Cluster-ID : 0.0.0.0
No. of configured peers : 1
No. of pending config peers : 0
No. of established peers : 1
VRF RD : Not configured

Information for address family IPv4 Unicast in VRF default
Table Id : 1
Table state : UP
Peers Active-peers Routes Paths Networks Aggregates
1 1 2 2 1 0

Redistribution
None

Wait for IGP convergence is not configured


Nexthop trigger-delay
critical 3000 ms
non-critical 10000 ms

Information for address family IPv6 Unicast in VRF default
Table Id : 80000001
Table state : UP
Peers Active-peers Routes Paths Networks Aggregates
0 0 0 0 0 0

Redistribution
None

Wait for IGP convergence is not configured


Nexthop trigger-delay
critical 3000 ms
non-critical 10000 ms
nx-osv-1#
Expand Down
68 changes: 68 additions & 0 deletions 2-Manual/output/initial/nx-osv-1_show-bgp-process-vrf-all_console.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
+++ nx-osv-1: executing command 'show bgp process vrf all' +++
show bgp process vrf all

BGP Process Information
BGP Process ID : 8610
BGP Protocol Started, reason: : configuration
BGP Protocol Tag : 65000
BGP Protocol State : Running
BGP MMODE : Not Initialized
BGP Memory State : OK
BGP asformat : asplain

BGP attributes information
Number of attribute entries : 2
HWM of attribute entries : 2
Bytes used by entries : 200
Entries pending delete : 0
HWM of entries pending delete : 0
BGP paths per attribute HWM : 1
BGP AS path entries : 0
Bytes used by AS path entries : 0

Information regarding configured VRFs:

BGP Information for VRF default
VRF Id : 1
VRF state : UP
Router-ID : 10.2.2.2
Configured Router-ID : 10.2.2.2
Confed-ID : 0
Cluster-ID : 0.0.0.0
No. of configured peers : 1
No. of pending config peers : 0
No. of established peers : 1
VRF RD : Not configured

Information for address family IPv4 Unicast in VRF default
Table Id : 1
Table state : UP
Peers Active-peers Routes Paths Networks Aggregates
1 1 2 2 1 0

Redistribution
None

Wait for IGP convergence is not configured


Nexthop trigger-delay
critical 3000 ms
non-critical 10000 ms

Information for address family IPv6 Unicast in VRF default
Table Id : 80000001
Table state : UP
Peers Active-peers Routes Paths Networks Aggregates
0 0 0 0 0 0

Redistribution
None

Wait for IGP convergence is not configured


Nexthop trigger-delay
critical 3000 ms
non-critical 10000 ms
nx-osv-1#
Expand Down
Loading

0 comments on commit a2fb31c

Please sign in to comment.