Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vnetd daemon is crashing due to panic: next-hop address is not found in multipath #109

Closed
jignesh045 opened this issue Feb 16, 2018 · 7 comments
Closed

vnetd daemon is crashing due to panic: next-hop address is not found in multipath #109

jignesh045 opened this issue Feb 16, 2018 · 7 comments
Assignees
@dlobete

Comments

@jignesh045
Copy link

Issue Description: vnetd daemon is crashing due to panic: next-hop address is not found in multipath

Environment: Testbed setup2 (invader42, Invader43, Invader44, Invader45)

root@invader44:/# goes hget platina packages | grep version:
version: 43cfbc9
version: 6133a579e7805972e9138334a4557e6d4d3fac39
version: 60f39141fbbf78ddb2260dba74c68f2789374f18
root@invader44:/#

Steps:

  1. Login to invader28
  2. Go to /root/regtest directory
  3. Run: ansible-playbook -i hosts_testbed2 playbooks/bgp/quagga_bgp_peering_ebgp_loopback.yml -K
  4. wait for 10 min for vnetd to crash
  5. check goes status

root@invader44:/# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - Not OK
status: vnetd daemon not responding
root@invader44:/#

  1. Find the attached logs

Feb 15 23:45:34 invader44 ntpd[430]: peers refreshed
Feb 15 23:59:26 invader44 goes.vnetd[7156]: panic: vnet: requested next-hop 10.0.19.31 not found in multipath
Feb 15 23:59:26 invader44 goes.vnetd[7156]: goroutine 51 [running]:
Feb 15 23:59:26 invader44 goes.vnetd[7156]: runtime/debug.Stack(0xbc25c0, 0xc420ffcf60, 0xc4214276a8)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler.func1(0xc42020d818, 0xc42020dd70)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:181 +0x159
Feb 15 23:59:26 invader44 goes.vnetd[7156]: panic(0xbc25c0, 0xc420ffcf40)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /usr/local/go/src/runtime/panic.go:491 +0x283
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/ip4.(*Fib).replaceNextHop(0xc4201faa00, 0xc42027a000, 0xc421e40910, 0xc4201faa00, 0xfd00000143, 0xc41f13000a, 0x18dd1a0, 0xc421482060, 0x0, 0x0)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:749 +0x46a
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/ip4.(*mapFibResult).replaceWithLessSpecific(0xc4214279f8, 0xc42027a000, 0xc4201faa00, 0xc4214279e8)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:387 +0x13a
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/ip4.(*Fib).addDelReachable(0xc4201faa00, 0xc42027a000, 0xc421e40900, 0x100000143)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:452 +0x10d
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/ip4.(*Fib).addDel(0xc4201faa00, 0xc42027a000, 0xc421e40900, 0x100000143, 0xb675318d4730a96a)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:276 +0x3ee
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/ip4.(*Main).addDelRoute(0xc42027a000, 0xc4205df1a0, 0x14300000000, 0x633301, 0xc42027a0b0, 0x4f, 0xc400000000)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip4/fib.go:635 +0xf4
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/ip4.(*Main).(github.com/platinasystems/go/vnet/ip4.addDelRoute)-fm(0xc4205df1a0, 0x14300000000, 0xc400000001, 0xc422256f08, 0x1, 0x3d)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ip4/package.go:28 +0x4d
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/ethernet.(*ipNeighborMain).AddDelIpNeighbor(0xc42005a868, 0xc42027a068, 0xc4205df180, 0x4f1f130001, 0x1, 0xc421427d08, 0x6b1484)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/ethernet/neighbor.go:91 +0x28b
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).ip4NeighborMsg(0xc42078f680, 0xc4202b22a0, 0xc4202b22a0, 0x40000000001)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:625 +0x26a
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/vnet/unix.(*netlinkEvent).EventAction(0xc42078f680)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/vnet/unix/netlink.go:506 +0x624
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/elib/loop.(*nodeEvent).do(0xc4202da340)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:152 +0x8f
Feb 15 23:59:26 invader44 goes.vnetd[7156]: github.com/platinasystems/go/elib/loop.(*Loop).eventHandler(0xc42020d818, 0x18d5b60, 0xc42020dd68)
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:195 +0xa3
Feb 15 23:59:26 invader44 goes.vnetd[7156]: created by github.com/platinasystems/go/elib/loop.(*Node).maybeStartEventHandler
Feb 15 23:59:26 invader44 goes.vnetd[7156]: /home/fyang/gopath/src/github.com/platinasystems/go/elib/loop/event.go:280 +0x2a6
Feb 15 23:59:27 invader44 lldpd[381]: error while receiving frame on eth-27-1: Network is down
Feb 15 23:59:27 invader44 lldpd[381]: error while receiving frame on eth-29-1: Network is down
Feb 15 23:59:28 invader44 lldpd[381]: error while receiving frame on eth-7-1: Network is down
Feb 15 23:59:28 invader44 lldpd[381]: error while receiving frame on eth-9-1: Network is down
Feb 15 23:59:28 invader44 lldpd[381]: error while receiving frame on eth-21-1: Network is down
Feb 15 23:59:28 invader44 lldpd[381]: error while receiving frame on eth-11-1: Network is down
Feb 15 23:59:28 invader44 lldpd[381]: error while receiving frame on eth-15-1: Network is down
Feb 15 23:59:28 invader44 lldpd[381]: error while receiving frame on eth-31-1: Network is down
Feb 15 23:59:28 invader44 lldpd[381]: error while receiving frame on eth-1-1: Network is down
Feb 15 23:59:29 invader44 lldpd[381]: error while receiving frame on eth-5-1: Network is down
Feb 15 23:59:29 invader44 lldpd[381]: error while receiving frame on eth-19-1: Network is down
Feb 15 23:59:29 invader44 lldpd[381]: error while receiving frame on eth-3-1: Network is down
Feb 15 23:59:29 invader44 lldpd[381]: error while receiving frame on eth-17-1: Network is down
Feb 15 23:59:29 invader44 lldpd[381]: error while receiving frame on eth-23-1: Network is down
Feb 15 23:59:29 invader44 lldpd[381]: unable to send packet on real device for eth-9-1: No such device or address
Feb 15 23:59:29 invader44 lldpd[381]: unable to send packet on real device for eth-1-1: No such device or address
Feb 15 23:59:29 invader44 lldpd[381]: unable to send packet on real device for eth-5-1: No such device or address
Feb 15 23:59:29 invader44 lldpd[381]: unable to send packet on real device for eth-3-1: No such device or address
Feb 15 23:59:29 invader44 lldpd[381]: unable to send packet on real device for eth-11-1: No such device or address
Feb 15 23:59:29 invader44 lldpd[381]: unable to send packet on real device for eth-7-1: No such device or address
Feb 15 23:59:29 invader44 lldpd[381]: unable to send packet on real device for eth-15-1: No such device or address
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #100 eth-19-1, fe80::5218:4cff:fe00:66c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #99 eth-1-1, fe80::5218:4cff:fe00:624#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #98 eth-11-1, fe80::5218:4cff:fe00:64c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #97 eth-29-1, fe80::5218:4cff:fe00:694#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #96 eth-3-1, fe80::5218:4cff:fe00:62c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #95 eth-21-1, fe80::5218:4cff:fe00:674#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #94 eth-31-1, fe80::5218:4cff:fe00:69c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #93 eth-23-1, fe80::5218:4cff:fe00:67c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #92 eth-5-1, fe80::5218:4cff:fe00:634#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #91 eth-15-1, fe80::5218:4cff:fe00:65c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #90 eth-7-1, fe80::5218:4cff:fe00:63c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #89 eth-17-1, fe80::5218:4cff:fe00:664#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #88 eth-27-1, fe80::5218:4cff:fe00:68c#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #87 eth-9-1, fe80::5218:4cff:fe00:644#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #86 eth-23-1, 10.0.23.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #85 eth-17-1, 10.0.17.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #84 eth-30-1, 10.0.30.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #83 eth-3-1, 10.0.3.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #82 eth-4-1, 10.0.4.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #81 eth-19-1, 10.0.19.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #80 eth-18-1, 10.0.18.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #79 eth-10-1, 10.0.10.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #78 eth-5-1, 10.0.5.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #77 eth-1-1, 10.0.1.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #76 eth-32-1, 10.0.32.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #75 eth-24-1, 10.0.24.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #74 eth-31-1, 10.0.31.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #73 eth-28-1, 10.0.28.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #72 eth-15-1, 10.0.15.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #71 eth-14-1, 10.0.14.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #70 eth-11-1, 10.0.11.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #69 eth-25-1, 10.0.25.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #68 eth-21-1, 10.0.21.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #67 eth-22-1, 10.0.22.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #66 eth-20-1, 10.0.20.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #65 eth-13-1, 10.0.13.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #64 eth-8-1, 10.0.8.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #63 eth-2-1, 10.0.2.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #62 eth-26-1, 10.0.26.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #61 eth-16-1, 10.0.16.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #60 eth-12-1, 10.0.12.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #59 eth-9-1, 10.0.9.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #58 eth-7-1, 10.0.7.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #57 eth-6-1, 10.0.6.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #56 eth-29-1, 10.0.29.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: Deleting interface #55 eth-27-1, 10.0.27.30#123, interface stats: received=0, sent=0, dropped=0, active_time=900 secs
Feb 16 00:00:34 invader44 ntpd[430]: peers refreshed
Feb 16 00:04:48 invader44 systemd[1]: Starting user-1017.slice.
Feb 16 00:04:48 invader44 systemd[1]: Created slice user-1017.slice.
Feb 16 00:04:48 invader44 systemd[1]: Starting User Manager for UID 1017...
Feb 16 00:04:48 invader44 systemd[1]: Starting Session c3 of user sandeep.
Feb 16 00:04:48 invader44 systemd[1]: Started Session c3 of user sandeep.
Feb 16 00:04:48 invader44 systemd[10414]: Starting Paths.
Feb 16 00:04:48 invader44 systemd[10414]: Reached target Paths.
Feb 16 00:04:48 invader44 systemd[10414]: Starting Timers.
Feb 16 00:04:48 invader44 systemd[10414]: Reached target Timers.
Feb 16 00:04:48 invader44 systemd[10414]: Starting Sockets.
Feb 16 00:04:48 invader44 systemd[10414]: Reached target Sockets.
Feb 16 00:04:48 invader44 systemd[10414]: Starting Basic System.
Feb 16 00:04:48 invader44 systemd[10414]: Reached target Basic System.
Feb 16 00:04:48 invader44 systemd[10414]: Starting Default.
Feb 16 00:04:48 invader44 systemd[10414]: Reached target Default.
Feb 16 00:04:48 invader44 systemd[10414]: Startup finished in 13ms.
Feb 16 00:04:48 invader44 systemd[1]: Started User Manager for UID 1017.
Feb 16 00:17:01 invader44 CRON[10460]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Observation: Adding a static route is causing vnetd to go into panic status and got crashed after some time (5-10 minute)
@arthurxiong3
Copy link
Contributor

arthurxiong3 commented Feb 19, 2018 •
edited
Loading

I got vnetd not responding, however no kernel panic.

After the playbook stopped running at -
"
TASK [Add bgp config in /etc/quagga/bgpd.conf] *********************************
fatal: [invader44]: FAILED! => {"changed": false, "failed": true, "msg": "unsupported parameter for module: path"}
fatal: [invader43]: FAILED! => {"changed": false, "failed": true, "msg": "unsupported parameter for module: path"}
fatal: [invader45]: FAILED! => {"changed": false, "failed": true, "msg": "unsupported parameter for module: path"}
fatal: [invader42]: FAILED! => {"changed": false, "failed": true, "msg": "unsupported parameter for module: path"}
to retry, use: --limit @/home/arthur/go/src/github.com/platinasystems/go/ansible/regtest/playbooks/bgp/quagga_bgp_peering_ebgp_loopback.retry

PLAY RECAP *********************************************************************
invader42 : ok=4 changed=0 unreachable=0 failed=1
invader43 : ok=4 changed=1 unreachable=0 failed=1
invader44 : ok=4 changed=0 unreachable=0 failed=1
invader45 : ok=4 changed=0 unreachable=0 failed=1
server_emulator : ok=3 changed=1 unreachable=0 failed=0
"

on invader42 -
"
root@invader42:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader42:~# ip route | grep via
default via 172.17.2.1 dev eth0
192.168.44.1 via 10.0.11.30 dev eth-11-1
192.168.44.1 via 10.0.3.30 dev eth-3-1
192.168.45.1 via 10.0.27.29 dev eth-27-1
192.168.45.1 via 10.0.19.29 dev eth-19-1
"

on invader43 -
"
root@invader43:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader43:~# ip route | grep via
default via 172.17.2.1 dev eth0
192.168.44.1 via 10.0.21.30 dev eth-21-1
192.168.44.1 via 10.0.19.30 dev eth-19-1
192.168.45.1 via 10.0.5.29 dev eth-5-1
192.168.45.1 via 10.0.3.29 dev eth-3-1
"

on invader44 -
"
root@invader44:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader44:~# ip route | grep via
default via 172.17.2.1 dev eth0
192.168.42.1 via 10.0.11.32 dev eth-11-1
192.168.42.1 via 10.0.3.32 dev eth-3-1
192.168.43.1 via 10.0.21.31 dev eth-21-1
192.168.43.1 via 10.0.19.31 dev eth-19-1
"

on invader45 -
"
root@invader45:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader45:~# ip route | grep via
default via 172.17.2.1 dev eth0
192.168.42.1 via 10.0.27.32 dev eth-27-1
192.168.42.1 via 10.0.19.32 dev eth-19-1
192.168.43.1 via 10.0.5.31 dev eth-5-1
192.168.43.1 via 10.0.3.31 dev eth-3-1
"

Then on invader42 -
"
root@invader42:~# vtysh

Hello, this is Quagga (version 0.99.23.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

invader42# conf t
invader42(config)# router bgp 65232
-multihop 2
neighbor 192.168.44.1 update-source dummy0
neighbor 192.168.45.1 remote-as 65229
neighbor 192.168.45.1 ebgp-multihop 2
neighbor 192.168.45.1 update-source dummy0invader42(config-router)# bgp router-id 172.17.2.42
invader42(config-router)# neighbor 192.168.44.1 remote-as 65230
invader42(config-router)# neighbor 192.168.44.1 ebgp-multihop 2
invader42(config-router)# neighbor 192.168.44.1 update-source dummy0
invader42(config-router)# neighbor 192.168.45.1 remote-as 65229
invader42(config-router)# neighbor 192.168.45.1 ebgp-multihop 2
invader42(config-router)# neighbor 192.168.45.1 update-source dummy0
invader42(config-router)# end
invader42# exit
root@invader42:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader42:~#
"

on invader43 -
"
root@invader43:~# vtysh

Hello, this is Quagga (version 0.99.23.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

invader43# conf t
invader43(config)# router bgp 65231
ebgp-multihop 2
neighbor 192.168.44.1 update-source dummy0
neighbor 192.168.45.1 remote-as 65229
neighbor 192.168.45.1 ebgp-multihop 2
neighbor 192.168.45.1 update-source dummy0invader43(config-router)# bgp router-id 172.17.2.43
invader43(config-router)# neighbor 192.168.44.1 remote-as 65230
invader43(config-router)# neighbor 192.168.44.1 ebgp-multihop 2
invader43(config-router)# neighbor 192.168.44.1 update-source dummy0
invader43(config-router)# neighbor 192.168.45.1 remote-as 65229
invader43(config-router)# neighbor 192.168.45.1 ebgp-multihop 2
invader43(config-router)# neighbor 192.168.45.1 update-source dummy0
invader43(config-router)# end
invader43# exit
root@invader43:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
"

on invader44 -
"
root@invader44:~# vtysh

Hello, this is Quagga (version 0.99.23.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

invader44# conf t
invader44(config)# router bgp 65230
.168.42.1 update-source dummy0
neighbor 192.168.43.1 remote-as 65231
neighbor 192.168.43.1 ebgp-multihop 2
neighbor 192.168.43.1 update-source dummy0invader44(config-router)# bgp router-id 172.17.2.44
invader44(config-router)# neighbor 192.168.42.1 remote-as 65232
invader44(config-router)# neighbor 192.168.42.1 ebgp-multihop 2
invader44(config-router)# neighbor 192.168.42.1 update-source dummy0
invader44(config-router)# neighbor 192.168.43.1 remote-as 65231
invader44(config-router)# neighbor 192.168.43.1 ebgp-multihop 2
invader44(config-router)# neighbor 192.168.43.1 update-source dummy0
invader44(config-router)# end
invader44# exit
root@invader44:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - Not OK
status: vnetd daemon not responding
root@invader44:~#
"

then on invader42 -
"
root@invader42:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - Not OK
status: vnetd daemon not responding
root@invader42:#
"
Then after sometime,
"
root@invader42:# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader42:~# ip route | grep via
default via 172.17.2.1 dev eth0
nexthop via 192.168.21.45 dev eth-21-1.1 weight 1
nexthop via 172.16.21.29 dev eth-21-1 weight 1
nexthop via 172.16.3.30 dev eth-3-1 weight 1
nexthop via 192.168.3.44 dev eth-3-1.1 weight 1
nexthop via 192.168.21.45 dev eth-21-1.1 weight 1
nexthop via 172.16.21.29 dev eth-21-1 weight 1
nexthop via 172.16.3.30 dev eth-3-1 weight 1
nexthop via 192.168.3.44 dev eth-3-1.1 weight 1
"

"
root@invader43:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader43:~# ip route | grep via
default via 172.17.2.1 dev eth0
nexthop via 192.168.19.44 dev eth-19-1.1 weight 1
nexthop via 172.16.19.30 dev eth-19-1 weight 1
nexthop via 172.16.1.29 dev eth-1-1 weight 1
nexthop via 192.168.1.45 dev eth-1-1.1 weight 1
nexthop via 192.168.19.44 dev eth-19-1.1 weight 1
nexthop via 172.16.19.30 dev eth-19-1 weight 1
nexthop via 172.16.1.29 dev eth-1-1 weight 1
nexthop via 192.168.1.45 dev eth-1-1.1 weight 1
"

"
root@invader44:# ip route | grep via
default via 172.17.2.1 dev eth0
nexthop via 192.168.19.43 dev eth-19-1.1 weight 1
nexthop via 172.16.19.31 dev eth-19-1 weight 1
nexthop via 172.16.3.32 dev eth-3-1 weight 1
nexthop via 192.168.3.42 dev eth-3-1.1 weight 1
nexthop via 192.168.19.43 dev eth-19-1.1 weight 1
nexthop via 172.16.19.31 dev eth-19-1 weight 1
nexthop via 172.16.3.32 dev eth-3-1 weight 1
nexthop via 192.168.3.42 dev eth-3-1.1 weight 1
root@invader44:# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
"

"
root@invader45:~# goes status
GOES status

Mode - TUNTAP
PCI - OK
Check daemons - OK
Check Redis - OK
Check vnet - OK
root@invader45:~# ip route | grep via
default via 172.17.2.1 dev eth0
nexthop via 192.168.21.42 dev eth-21-1.1 weight 1
nexthop via 172.16.21.32 dev eth-21-1 weight 1
nexthop via 172.16.1.31 dev eth-1-1 weight 1
nexthop via 192.168.1.43 dev eth-1-1.1 weight 1
nexthop via 192.168.21.42 dev eth-21-1.1 weight 1
nexthop via 172.16.21.32 dev eth-21-1 weight 1
nexthop via 172.16.1.31 dev eth-1-1 weight 1
nexthop via 192.168.1.43 dev eth-1-1.1 weight 1
"

@arthurxiong3
Copy link
Contributor

The format seems messed up for the previous comment, please check the attachment for the same content.
issue109_1.txt

@jignesh045
Copy link
Author

Execute below steps & observe the goes service:

  1. Login to invader28
  2. Go to /root/regtest directory
  3. Run: ansible-playbook -i hosts_testbed2 playbooks/bgp/quagga_bgp_peering_ebgp_loopback.yml -K
  4. It shows the test case as passed.
  5. wait for 10 min for vnetd to crash
  6. check goes status
  7. Observed the log messages in /var/log/syslog files.
  8. you can also find the logs in redis client as well.

Observation: whenever we add static route (via "route add" linux command), it got crashed. This playbook contains the step of adding static route.

@saptaktakalkar
Copy link
Contributor

@arthurxiong3 Here are the logs of all invaders along with playbook run output.

inv42_log.txt
inv43_log.txt
inv44_log.txt
inv45_log.txt
playbook_run_output.txt

@saptaktakalkar
Copy link
Contributor

FYI Above logs have not captured addition of static routes using route add command. Hence I am posting all routes we are adding for this test case here:

On Invader42:

route add -net 192.168.44.1 netmask 255.255.255.255 gw 10.0.3.30
route add -net 192.168.45.1 netmask 255.255.255.255 gw 10.0.19.29
route add -net 192.168.44.1 netmask 255.255.255.255 gw 10.0.11.30
route add -net 192.168.45.1 netmask 255.255.255.255 gw 10.0.27.29

On Invader43:

route add -net 192.168.44.1 netmask 255.255.255.255 gw 10.0.19.30
route add -net 192.168.45.1 netmask 255.255.255.255 gw 10.0.3.29
route add -net 192.168.44.1 netmask 255.255.255.255 gw 10.0.21.30
route add -net 192.168.45.1 netmask 255.255.255.255 gw 10.0.5.29

On Invader44:

route add -net 192.168.42.1 netmask 255.255.255.255 gw 10.0.3.32
route add -net 192.168.43.1 netmask 255.255.255.255 gw 10.0.19.31
route add -net 192.168.42.1 netmask 255.255.255.255 gw 10.0.11.32
route add -net 192.168.43.1 netmask 255.255.255.255 gw 10.0.21.31

On Invader45:

route add -net 192.168.42.1 netmask 255.255.255.255 gw 10.0.19.32
route add -net 192.168.43.1 netmask 255.255.255.255 gw 10.0.3.31
route add -net 192.168.42.1 netmask 255.255.255.255 gw 10.0.27.32
route add -net 192.168.43.1 netmask 255.255.255.255 gw 10.0.5.31

@amit-pandia
Copy link
Contributor

amit-pandia commented Mar 7, 2018 •
edited
Loading

Multiple outputs of issue 109 test script.
issue 109 behaviour.txt

@jignesh045
Copy link
Author

jignesh045 commented Jul 23, 2018 •
edited
Loading

Issue verification Status: Verified
Execution Status: Passed

Issue verify in below build (in all setup2 invaders)

root@invader45:~# goes version
github.com/platinasystems/go: v0.41-244-g49d9e6da
github.com/platinasystems/fe1: v0.2-520-g85494c1
github.com/platinasystems/firmware-fe1a: v0.2
root@invader45:~#

root@invader45:~# dpkg --list | grep kernel
ii  kmod                           18-3                             amd64        tools for managing Linux kernel modules
ii  libdrm2:amd64                  2.4.58-2                         amd64        Userspace interface to kernel DRM services -- runtime
ii  linux-image-4.13.0-platina-mk1 4.13-117-g00814e4e1a1e           amd64        Linux kernel, version 4.13.0-platina-mk1
ii  rsyslog                        8.4.2-1+deb8u2                   amd64        reliable system and kernel logging daemon
root@invader45:~#

Steps:

  1. Login to invader28
  2. Run below playbook:
    abhishek@invader28:~/latest/go/ansible/regtest$ ansible-playbook -i hosts_testbed2 playbooks/bgp/quagga_bgp_peering_ebgp_loopback.yml -K -e reset_config=False
  3. It shows the test case as passed.
  4. waited for 10 min to check vnetd status -> It was showing OK.
  5. check goes status -> It was showing OK.
  6. Observed the log messages in /var/log/syslog files -> No panic observed

@fszyang fszyang closed this as completed Jul 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dlobete dlobete

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dlobete @fszyang @saptaktakalkar @jignesh045 @amit-pandia @arthurxiong3