Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] VNET Jails with static MAC network not reachable #821

Open
subnetspider opened this issue Jan 19, 2025 · 9 comments
Open

[BUG] VNET Jails with static MAC network not reachable #821

subnetspider opened this issue Jan 19, 2025 · 9 comments
Labels
bug Something isn't working

Comments

@subnetspider
Copy link
Contributor

[MANDATORY] Describe the bug [MANDATORY]
Networking on VNET jails with a static MAC address set on the epair_b interface does not work.

[MANDATORY] Bastille and FreeBSD version (paste bastille -v && freebsd-version -kru output)

0.12.20250111
14.2-RELEASE
14.2-RELEASE
14.2-RELEASE

[MANDATORY] How did you install bastille? (port/pkg/git)
pkg

[optional] Steps to reproduce?

  1. Install FreeBSD 14.2 on a physical host or VM
  2. Install the latest bastille pkg (version 0.12.20250111)
  3. Configure host/VM networking, bastille.conf, bootstrap 14.2-RELEASE.
  4. Create a VNET jail on a physical interface or an existing bridge
  • Example 1: bastille create -B test 14.2-RELEASE SLAAC bridge60
  • Example 2: bastille create -V test 14.2-RELEASE SLAAC vlan60

[optional] Expected behavior
The VNET jail with a static MAC should be able to send an receive network traffic like VNET jails without a static MAC.

[optional] Actual behavior
The VNET jail with a static MAC can:

  • can receive IPv6 neighbor solicitations from other VNET jails on the same bridge bridge60.
  • can receive IPv6 neighbor solicitations from other external hosts on VLAN60.
  • can send IPv6 neighbor advertisements to other VNET jails on the same bridge bridge60.
  • can't send anything outsite of the bridge bridge60.
  • If the MAC address of epair15b isn't overridden, the VNET jail is running as expected.

The traffic is leaving through igb0 and/or igb1 on the FreeBSD Host, but never arrives on devices on the same VLAN.

# tcpdump running on the FreeBSD Jail host, the VNET jail with a static MAC is trying to ping the link local IPV6 address of the firewall.
$ sudo tcpdump ether host 8b:a9:8e:36:c8:3b -i igb0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:06:41.731581 IP6 fe80::c8cb:5b4c:ee79:962 > fe80::ca4f:86ff:fefc:3: ICMP6, neighbor solicitation, who has fe80::ca4f:86ff:fefc:3, length 32
11:06:41.874583 IP6 fe80::c8cb:5b4c:ee79:962 > fe80::ca4f:86ff:fefc:3: ICMP6, echo request, id 26201, seq 5, length 16
11:06:42.731799 IP6 fe80::c8cb:5b4c:ee79:962 > fe80::ca4f:86ff:fefc:3: ICMP6, neighbor solicitation, who has fe80::ca4f:86ff:fefc:3, length 32
11:06:42.879024 IP6 fe80::c8cb:5b4c:ee79:962 > fe80::ca4f:86ff:fefc:3: ICMP6, echo request, id 26201, seq 6, length 16
11:06:43.732049 IP6 fe80::c8cb:5b4c:ee79:962 > fe80::ca4f:86ff:fefc:3: ICMP6, neighbor solicitation, who has fe80::ca4f:86ff:fefc:3, length 32
11:06:43.890559 IP6 fe80::c8cb:5b4c:ee79:962 > fe80::ca4f:86ff:fefc:3: ICMP6, echo request, id 26201, seq 7, length 16

On the firewall, no traffic ever arrives and the MAC address is never learned:

SFVH_SO01_SFOS 21.0.0 GA-Build169 HA-Primary# tcpdump -i lagg0.60 ether host 8b:a9:8e:36:c8:3b
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0.60, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
25 packets received by filter
0 packets dropped by kernel

[optional] Screenshots

[optional] Additional context

FreeBSD Jail Host

The pf Firewall is disabled.

/etc/rc.conf:

# Network
cloned_interfaces="lagg0 bridge12 bridge20 bridge60 bridge70 bridge80"
vlans_lagg0="10 12 20 60 70 80"
ifconfig_igb0="up -rxcsum"
ifconfig_igb1="up -rxcsum"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1"
ifconfig_lagg0_10="SYNCDHCP"
ifconfig_lagg0_10_ipv6="inet6 2003:____:_____:____::114/64 accept_rtadv"
rtsold_enable="YES"
rtsold_flags="lagg0.10"

ifconfig_lagg0_12="up"
ifconfig_lagg0_20="up"
ifconfig_lagg0_60="up"
ifconfig_lagg0_70="up"
ifconfig_lagg0_80="up"
ifconfig_bridge12="addm lagg0.12"
ifconfig_bridge20="addm lagg0.20"
ifconfig_bridge60="addm lagg0.60"
ifconfig_bridge70="addm lagg0.70"
ifconfig_bridge80="addm lagg0.80"

ifconfig: (limited)

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4c507ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 0c:c4:7a:71:4b:20
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4c507ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 0c:c4:7a:71:4b:20
	hwaddr 0c:c4:7a:71:4b:21
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<snip>
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4c507ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 0c:c4:7a:71:4b:20
	hwaddr 00:00:00:00:00:00
	laggproto lacp lagghash l2,l3,l4
	laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	groups: lagg
	media: Ethernet autoselect
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<snip>
bridge60: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=0
	ether 58:9c:fc:10:8e:65
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: e15a_tlsproxy01 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 59 priority 128 path cost 2000
	member: e14a_acme01 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 27 priority 128 path cost 2000
	member: e0a_unbound01 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 65 priority 128 path cost 2000
	member: e13a_torrent01 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 61 priority 128 path cost 2000
	member: e0a_nsd01 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 45 priority 128 path cost 2000
	member: e0a_adguard01 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 19 priority 128 path cost 2000
	member: lagg0.60 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 15 priority 128 path cost 10000
	groups: bridge
	nd6 options=9<PERFORMNUD,IFDISABLED>
<snip>
lagg0.60: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=0
	ether 0c:c4:7a:71:4b:20
	groups: vlan
	vlan: 60 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
	media: Ethernet autoselect
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
<snip>
e15a_tlsproxy01: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 8b:a9:8e:36:c8:3a
	hwaddr 02:8a:2e:49:29:0a
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

VNET Jail `tlsproxy01

jail.conf:

tlsproxy01 {
  enforce_statfs = 2;
  devfs_ruleset = 13;
  exec.clean;
  exec.consolelog = /var/log/bastille/tlsproxy01_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = tlsproxy01;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/tlsproxy01/fstab;
  path = /usr/local/bastille/jails/tlsproxy01/root;
  securelevel = 2;
  osrelease = 14.2-RELEASE;

  vnet;
  vnet.interface = e15b_tlsproxy01;
  exec.prestart += "ifconfig epair15 create";
  exec.prestart += "ifconfig bridge60 addm epair15a";
  exec.prestart += "ifconfig epair15a up name e15a_tlsproxy01";
  exec.prestart += "ifconfig epair15b up name e15b_tlsproxy01";
  exec.prestart += "ifconfig e15a_tlsproxy01 ether 8b:a9:8e:36:c8:3a";
  exec.prestart += "ifconfig e15b_tlsproxy01 ether 8b:a9:8e:36:c8:3b";
  exec.poststop += "ifconfig bridge60 deletem e15a_tlsproxy01";
  exec.poststop += "ifconfig e15a_tlsproxy01 destroy";

ifconfig:

vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 8b:a9:8e:36:c8:3b
	hwaddr 02:e5:cd:e4:f7:0b
	inet6 fe80::c8cb:5b4c:ee79:962%vnet0 prefixlen 64 scopeid 0x3c
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

A IPv4 address is never set through DHCP
A IPv6 address is only created after the jail received a router advertisement (periodic / unsolicitated)

/etc/rc.conf

syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
ifconfig_e15b_tlsproxy01_name="vnet0"
ifconfig_vnet0="DHCP"
ifconfig_vnet0_ipv6="inet6 accept_rtadv"
openssh_enable="YES"

I'm not sure what to make of this, I tried the VNET jail with a static MAC on two physical FreeBSD hosts, one with VLAN and one without, as well as on a FreeBSD VM with VLANs.
@subnetspider subnetspider added the bug Something isn't working label Jan 19, 2025
@tschettervictor
Copy link
Collaborator

tschettervictor commented Jan 19, 2025 •
edited
Loading

Although I'm not sure why this is happening, the default behaviour has been returned to bastille.

It is now necessary to specify -M if the user does want a static MAC to be assigned to the jail.

This will be in the next release.

Current head has this already.

@bmac2
Copy link
Collaborator

bmac2 commented Jan 19, 2025

@subnetspider can you test the github version to see if it is fixed for you.

@tschettervictor
Copy link
Collaborator

Can you test it with IPv4? Wondering if the IPv6 has anything to do with it.

@subnetspider
Copy link
Contributor Author

subnetspider commented Jan 19, 2025 •
edited
Loading

@subnetspider can you test the github version to see if it is fixed for you.

Is there a special branch or have you updated the main branch?

Can you test it with IPv4? Wondering if the IPv6 has anything to do with it.

I did with DHCP, it kept broadcasting DHCP requests but they never arrived at the firewall.

@bmac2
Copy link
Collaborator

bmac2 commented Jan 19, 2025

main branch test please

@subnetspider
Copy link
Contributor Author

Update:

On a second host, I have created another VNET jail with the static MAC enabled, and it runs without issue:

FreeBSD host:

em0bridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=0
	ether 58:9c:fc:00:11:12
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: e0a_bastille1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 8 priority 128 path cost 2000
	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 3 priority 128 path cost 20000
	groups: bridge
	nd6 options=9<PERFORMNUD,IFDISABLED>

FreeBSD VNET jail

vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 58:9c:fc:eb:66:2b
	hwaddr 02:0c:fd:c3:2a:0b
	inet 10.1.10.102 netmask 0xffffff00 broadcast 10.1.10.255
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

On the host with the issue, the network setup is a little bit more complicated:

igb0 + igb1 --> lagg0 --> lagg0.60 --> bridge60 --> epair --> jail

On this host, it's a lot simpler:

em0 --> em0bridge --> epair --> jail

@bmac2 and I suspect that either the lagg or the vlan interface are what's causing this.
I will run a few more tests (with lagg, with vlan, with lagg + vlan) as soon as I get some more free time.

@tschettervictor
Copy link
Collaborator

em0bridge looks like an automatically created bridge. Did you create that with "ifconfig bridge create"?

What happens when you try with a bridge created using the above command?

@bmac2
Copy link
Collaborator

bmac2 commented Jan 20, 2025

For now this issue is kind of on hold until we can get some better data on what is going on. If there is a lagg issue we can isolate we will look at fixing the code for that.

For now this one should not hold up the release or any additional work.

@yaazkal @tschettervictor @subnetspider

@tschettervictor
Copy link
Collaborator

This issue is because we are missing the "up" parameter inside the "generate_vnet_block"

It should be fixed in the next release when #792 is included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bmac2 @tschettervictor @subnetspider