PfSense 23.01/2.7 (FreeBSD 14.0) #79

bigjohns97 · 2023-01-14T17:04:07Z

With the new release coming up and quite a few changes I figured it was time to start a discussion on bypass scripts for the new version of PfSense about to release.

A couple of notes that hopefully saves some people some time.

https://redmine.pfsense.org/issues/12821?next_issue_id=12820

Looks like we won't have to be using netgraph moving forward which would be nice

With the introduction of native PCP VLAN0 tagging in pfSense Plus 23.01 and the new bridge filtering to pass along EAP traffic from another interface natively integrated into pfSense Plus

Which is good because the src isn't available for 14.0 yet so we wouldn't be able to compile a new Intel driver against it.

But this does mean we are going to need to build a new script with new commands to tag VLAN and authenticate EAP traffic.

bigjohns97 · 2023-01-14T18:15:35Z

Some extra threads for reference

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224961
https://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+14.0-CURRENT&arch=default&format=html

I tried just about everything to get this thing to listen, you do have to remove the previously compiled drivers from the other thread or the system halts, also you can't run any pfatt scripts because the ngeth0 interface comes up with a random MAC, and either way it doesn't seem like it would work based upon the above threads.

bigjohns97 · 2023-01-14T19:28:06Z

So the good news is I was able to get it working with the old school script but it's still using netgraph. :(

Maybe someone with a little more skills and time can get it working without netgraph, I've used up all of my political capital with the family today :)

I will post the script below (see attached)

As you can see I only added the "/sbin/ifconfig $ONT_IF -vlanhwfilter" under the enabling promisc section for my supplicant option, if you use the bridge go ahead and do the same under the bridge section of the script. (should only be required for Intel NIC's)

This doesn't require any custom driver anymore so at least there is a small win there.
2301_bypass.txt

dmitripr · 2023-01-20T21:20:21Z

If I'm reading the comment correctly, I think someone solved this without using the script:
https://redmine.pfsense.org/issues/12070#note-7

If someone can contact them and ask about the procedure?

bigjohns97 · 2023-01-21T02:17:35Z

If I'm reading the comment correctly, I think someone solved this without using the script: https://redmine.pfsense.org/issues/12070#note-7

If someone can contact them and ask about the procedure?

I just tried what they suggested tonight and wasn't able to get it working, I posted in that thread in hopes that he will respond.

Shadow00Caster · 2023-02-16T02:09:07Z

Is there any known fix for this?

liuhaotian · 2023-02-16T06:28:09Z

While trying out 23.01 RC, I found another way to get ATT working by using same VPID for WAN and OPT and keep pfSense and ATT RG as dual gateway, confirmed on 22.05 without pfatt. hope if helps:
https://navigateheadwinds.blogspot.com/2023/02/pfsense-att-dual-gateway.html

bigjohns97 · 2023-02-16T14:09:07Z

Is there any known fix for this?

Known fix for what?

I posted the edit of the original script to use with Intel NIC.

Still working on trying to figure out how to not require the full script and take ngeth out of the equation.

TTK-199837 · 2023-02-19T01:27:40Z

Sorry, but could someone clarify for me? If I have pfatt working fine on 22.05 will upgrading to 23.01 break it?

bigjohns97 · 2023-02-19T01:32:02Z

It depends are you using the custom compiled driver?

TTK-199837 · 2023-02-19T01:34:35Z

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

bigjohns97 · 2023-02-19T14:29:24Z

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3.

23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine.

This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best.

GL

TTK-199837 · 2023-02-19T17:59:42Z

Thank you for that explanation. I do remember doing something with the intel NICs on my SG5100 in one of the recent updates.

…

On Sun, Feb 19, 2023, at 6:29 AM, bigjohns97 wrote: > How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help! > If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3. 23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine. This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best. GL — Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AND3BCWPUQK3OR6IQ3LFNR3WYIU47ANCNFSM6AAAAAAT3LWZAM>. You are receiving this because you commented.Message ID: ***@***.***>

unawarez · 2023-02-23T23:32:55Z

@bigjohns97 are you using wpa-supplicant/is that relevant in this issue? I've been trying to navigate this whole thing across different threads/bugs and I'm thinking maybe there's a mixture of feedback between wpa-supplicant and passthrough setups out there. FWIW I have a supplicant setup and also would like to get rid of netgraph. Even with netgraph I've always needed wpa-supplicant running on the "raw" interface so that its traffic doesn't get VLAN tags. I've been trying to come up with some clever setup with epair and a bridge or something like that but can never get the return traffic routed to the right interface.

bigjohns97 · 2023-02-24T14:18:45Z

@bigjohns97 are you using wpa-supplicant/is that relevant in this issue? I've been trying to navigate this whole thing across different threads/bugs and I'm thinking maybe there's a mixture of feedback between wpa-supplicant and passthrough setups out there. FWIW I have a supplicant setup and also would like to get rid of netgraph. Even with netgraph I've always needed wpa-supplicant running on the "raw" interface so that its traffic doesn't get VLAN tags. I've been trying to come up with some clever setup with epair and a bridge or something like that but can never get the return traffic routed to the right interface.

Yes I am using supplicant.

However, in my testing with this I never did the wpa_supplicant before applying the PCP values, I can try this later on today and see if it makes a difference.

unawarez · 2023-02-25T04:04:48Z

Well I'm posting this from a zero-netgraph supplicant setup after a reboot 😎

~~And as a side-effect my bufferbloat score has gone from B to A, without any traffic shaping (which I couldn't set up before anyway due to using netgraph)~~ I think my B was from pfSense 22.05, can't remember if I tried on 23.01 before this

My approach was using wpa_cli's -a flag which lets you run a hook script every time the interface dis/connects. I use that to run ifconfig $ifname pcp 0 every time wpa_supplicant connects. I have no idea how well it handles disconnects though. I'll attach my current minimal scripts. I'm running this as a shellcmd at startup.

Edit: the wpa_cli hook does not handle disconnects well. When my fiber dropped out I had to reboot the switch to get IPv6 back. The disconnect hook never fired. That really makes me want to use dhcpcd hooks but I don't have time right now to figure all that out.

pfatt_sup14.txt
pfatt_sup14_action.txt

bigjohns97 · 2023-02-25T05:47:22Z

Well I'm posting this from a zero-netgraph supplicant setup after a reboot 😎

~~And as a side-effect my bufferbloat score has gone from B to A, without any traffic shaping (which I couldn't set up before anyway due to using netgraph)~~ I think my B was from pfSense 22.05, can't remember if I tried on 23.01 before this

My approach was using wpa_cli's -a flag which lets you run a hook script every time the interface dis/connects. I use that to run ifconfig $ifname pcp 0 every time wpa_supplicant connects. I have no idea how well it handles disconnects though. I'll attach my current minimal scripts. I'm running this as a shellcmd at startup.

pfatt_sup14.txt pfatt_sup14_action.txt

I wasn't able to get mine to authenticate using these scripts even tried -vlanhwfilter as an extra step to see if it was that.

No-Go

unawarez · 2023-02-25T05:58:57Z

I wasn't able to get mine to authenticate using these scripts even tried -vlanhwfilter as an extra step to see if it was that.

Ugh sorry. FWIW I'm using a Netgate SG-3100 which is 32-bit ARM and Marvell network interfaces. I can provide command or log output of anything that would help

unawarez · 2023-02-25T07:27:15Z

On reboot number 2, wpa_supplicant logged a failed auth, then tried again 30 seconds later and succeeded

bigjohns97 · 2023-02-25T15:10:21Z

Are you running this as a earlyshellcmd?

That is how I was trying to use it and once booted up I could run ifconfig and noticed that I didn't have pcp assigned to the interface.

I am wondering if you could post how you came up with the order of things.

Judging by your script.

bring up interface and set promisc mode
remove any previous pcp attribute
run wpa auth along with connected watcher (4,5 below)
if not connected add pcp attribute
once connected remove pcp attribute

The whole time wpa and connected watcher stays running in the background.

Is this accurate?

bigjohns97 · 2023-02-25T15:11:09Z

BTW I noticed you reference en interface so your WAN is intel en?

bigjohns97 · 2023-02-25T17:09:13Z

Tried again this morning, toggling everything I could

with and without pcp
with and without vlanhwfilter
using my RG MAC as the wpa identity
using the purchased certs wpa identity

I could never get the wpa process to successfully authenticate, and I waited until it told me it failed.

I am wondering if this has anything to do with my certs being associated to another MAC than my RG, I still assign my RG MAC to my ngeth and my igb0 interface regardless but it just seems to work when using ngeth.

At a dead end, up for trying anything at this point if anyone has any ideas.

unawarez · 2023-02-25T23:01:08Z

BTW I noticed you reference en interface so your WAN is intel en?

No sorry that was pretty bad on my part, en123 is a placeholder name, and it's a Linuxism at that, where every wired iface gets named en<something>. My real WAN is mvneta2

Are you running this as a earlyshellcmd?

Only tried as a shellcmd so far. I want to try as an earlyshellcmd soon.

I am wondering if you could post how you came up with the order of things.

Judging by your script.
1. bring up interface and set promisc mode

2. remove any previous pcp attribute

3. run wpa auth along with connected watcher (4,5 below)

4. if not connected add pcp attribute

5. once connected remove pcp attribute
The whole time wpa and connected watcher stays running in the background.

Is this accurate?

Yep that's all accurate. wpa_cli is a second long-running process alongside wpa_supplicant. I was looking for any of the standard daemons that might support external hooks. And actually I think long-term doing this as dhcpcd hooks instead of supplicant hooks might be a little more reliable, but that's just a hunch, and idek if you can add config options like that to the dhcpcd instances that pfSense automatically runs for interfaces.

To get here I'd been experimenting with setting my WAN interface in pfSense to the real WAN iface (mvneta2 in my case) instead of ngeth0. Through a combo of watching the system logs in the web UI and doing some packet captures I saw for whatever reasons in my setup, my ONT will only respond to 802.1X frames that do NOT have any 802.1Q tags, and will only respond to regular traffic that DOES have 802.1Q tags. I toggled the tags by running ifconfig mvneta2 pcp 0 or -pcp and occasionally dis/enabling the interface. I was struggling for a long time to get a non-netgraph equivalent of two network interfaces that had the same external MAC, so I could set pcp 0 on one and leave wpa_supplicant on the other. Using connection hooks to toggle it instead on one interface was a shower thought and only worked because these daemons happen to support hooks.

I don't know if promisc mode is still needed. That's left from original pfatt.

For debugging this script live I was ssh'd in and alternating between running this script and running a pfatt_cleanup.txt script I threw together. All the while watching the system logs in web UI for output from supplicant/etc.

I am wondering if this has anything to do with my certs being associated to another MAC than my RG, I still assign my RG MAC to my ngeth and my igb0 interface regardless but it just seems to work when using ngeth.

My RG MAC is forced on mvneta2 in the web UI. Can't help with the different cert identity unfortunately.

If you haven't already I'd recommend getting a packet capture in the web UI of a successful supplicant handshake with your known good setup, download it and check it out in Wireshark or something. This probably means booting normally, physically disconnecting the cable, starting capture, and then reconnecting the cable. That should at least give a baseline of if it's expecting .Q tags or not.

bigjohns97 · 2023-02-26T21:13:40Z

Well unfortunately the packet captures didn't produce any valuable information.

The EAPOL start frame looks exactly the same on your script and the original pfatt script.

The only difference is the response which through the pfatt script is a success and your script is replies on VLAN 0 requesting an identity.

I'm going to have to wait for someone with more skills than myself to come up with a solution.

unawarez · 2023-02-26T23:18:30Z

Your traffic sounds similar to other reports I’ve seen in different threads. Maybe there’s more variance in ONT hardware (or firmware?) than folks have been thinking :( I tried getting my ONT’s model number but it’s conspicuously unlabeled. I’m not sure how I’d approach this then for one and only one untagged packet. I don’t know anything about freebsd pf but if it’s able to match on the start packet and only the start packet, I bet there’d be a way from there to redirect or modify that start packet, and just have your WAN interface using `pcp 0` all the time.

…

On Sun, Feb 26, 2023, at 15:13, bigjohns97 wrote: Well unfortunately the packet captures didn't produce any valuable information. The EAPOL start frame looks exactly the same on your script and the original pfatt script. The only difference is the response which through the pfatt script is a success and your script is replies on VLAN 0 requesting an identity. — Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3HIGZBNH7WLW5BCEVFSWCDWZPBRBANCNFSM6AAAAAAT3LWZAM>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

bigjohns97 · 2023-02-26T23:23:25Z

When I was running ESXi I would just create the NIC with my RG MAC at the virtual layer and tag VLAN 0 at the virtual layer as well and everything would work fine.

When you do an ifconfig, do you see an ether address as well as a hwaddr? Are they the same or different? Are they your RG MAC?

unawarez · 2023-02-26T23:30:08Z

The MACs are different. ether is my RG MAC, but hwaddr is 00:08:a2:something (same prefix for all my real interfaces)

bigjohns97 · 2023-02-26T23:35:21Z

The ngeth interface had a different hwaddr as well. I guess I need to dig into that and really see what makes that different than what I am doing.

unawarez · 2023-02-26T23:42:15Z

When I was running ESXi I would just create the NIC with my RG MAC at the virtual layer and tag VLAN 0 at the virtual layer as well and everything would work fine.

I find this especially weird...is there some standards-edge-case about eapol starts never supposed to be tagged? Otherwise if ESXi was tagging the start packet (which I would've assumed, just out of a hope for sanity) then why was the ONT accepting those starts on VLAN 0 but not accepting on pfSense if you just put supplicant on an interface with pcp 0 all the time?

The ngeth interface had a different hwaddr as well. I guess I need to dig into that and really see what makes that different than what I am doing.

Let me paste my supplicant conf too. I do have a cert that matches my provisioned RG MAC, so that is different, but I can't imagine that affecting this .Q stuff.

eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/pfatt/wpa/ca.pem"
        client_cert="/conf/pfatt/wpa/client.pem"
        eap=TLS
        eapol_flags=0
        identity="[redacted RG MAC]"
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/pfatt/wpa/private.pem"
}

I guess you could also try adding a line into the script that forces the RG MAC on the interface before it comes up. I'm relying on pfSense config to do that, and I could imagine that working for shellcmd but not earlyshellcmd, if the script happens before pfSense applies all the user config options.

bigjohns97 · 2023-02-27T00:18:04Z

I tried the line to the script setting the ether addr but one thing I just noticed is your eapol_version is 2 and mine is 1.

Checking the original pfatt script it looks like it has it set to 2 as well, will check to see if this makes a difference a little later on tonight.

bigjohns97 · 2023-02-27T02:53:10Z

Still a negative, even trued using my identity MAC as my spoofed MAC on my WAN interface this time thinking maybe that would work.

no go

Also tried setting a pcp of 7 since that is what the ONT was replying with when asking for my identity, still didn't work.

Giving up, surely someone else will come along and test out your script and see if it's me or maybe you just have something special with those realtek interfaces.

bigjohns97 · 2023-02-27T19:05:33Z

@unawarez would you mind posting your interface options?

Here are mine
RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP
for a while we had to disable vlanhwfilter for the pfatt script to work whenever they moved to a consolidated igb and em intel driver. (I still do this BTW and have been trying this in my testing, just wondering if there are more I should be disabling)

I am wondering if one of these are disabled on your driver by default and may explain our different experiences.

unawarez · 2023-02-27T23:19:36Z

Sure:

mvneta2: flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	description: WAN
	options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>

so you have these I don't: TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP

and I have LINKSTATE

edit: I just discovered ifconfig -m $IFNAME which also displays the available capabilities. The only extra one I have that is disabled is LRO. I notice there's no VLAN_HWFILTER even as an option for mine...

capabilities=804bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,LINKSTATE>

mdelise · 2023-03-17T16:39:27Z

Hello, looking for some help. I'm currently using pfsense 2.4.5 with pfatt in supplicant with working certs.

I purchased a DFP-34X-2C2 ONT stick, cloned the white ATT GPON ONT and installed in a pcie card in the pfsense server.

I'm trying to figure out what necessary changes to the pfatt script are required?
Netgraph is still needed to handle the VLAN0 situation, but I also need the WAN VLAN Tagged on 962 because of where internet is available from the DFP ONT.

# omcicli mib get 84
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
VlanTagFilterData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=================================
EntityID: 0x1102
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x02
NumOfEntries: 1
=================================
=================================
EntityID: 0x110b
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x10
NumOfEntries: 1
=================================

aholmes55 · 2023-03-27T13:22:46Z

I have both supplicant and bridge/tethered bypass working with netgraph on pfSense 23.01 on Netgate hardware (Intel igb nics) with ipv6. I have a BGW210 gateway. The included Intel drivers work fine for this release but you do have to add the vlanhwfilter flags to ifconfig in both bridge and supplicant modes per Issue 65. I am using certificates from an old device with a different MAC than my assigned gateway. The only issue I've seen is pfSense struggles to update packages (super slow or fails) when ipv6 is enabled. Without ipv6 it is quick.

pfatt.txt

bigjohns97 · 2023-03-27T13:39:31Z

Hello, looking for some help. I'm currently using pfsense 2.4.5 with pfatt in supplicant with working certs.

I purchased a DFP-34X-2C2 ONT stick, cloned the white ATT GPON ONT and installed in a pcie card in the pfsense server.

I'm trying to figure out what necessary changes to the pfatt script are required? Netgraph is still needed to handle the VLAN0 situation, but I also need the WAN VLAN Tagged on 962 because of where internet is available from the DFP ONT.
# omcicli mib get 84
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
VlanTagFilterData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=================================
EntityID: 0x1102
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x02
NumOfEntries: 1
=================================
=================================
EntityID: 0x110b
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x10
NumOfEntries: 1
=================================

Please open up another thread to discuss this

TTK-199837 · 2023-03-30T18:35:16Z

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3.

23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine.

This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best.

GL

Hi again! Requesting some more hand holding if you do not mind ;)

You mention "just disable hwvlanfilter", however I am not familiar with this setting, or having ever enabled it. After some exploring, I believe these steps should remove / disable the old custom intel drivers, and have a working router, please let me know if I am correct:

Reverse steps found here: Intel igb/em Interfaces Broken on 2.6/22.01+ #67 (comment)
Remove old intel driver if_igb.ko from /boot/kernel
Remove if_igb_load="YES" and if_igb_name="/boot/modules/if_igb.ko" from /boot/loader.conf
(?) should I keep my current pfatt.sh unchanged? It seems a LOT different than the couple posted here, I believe because I am using bridge mode? Do I, or should I modify it in anyway? (https://github.com/TTK-199837/pfatt/blob/master/bin/pfatt.sh)
Update via GUI, and we are up and running?

Thanks for your time!

gpz1100 · 2023-04-02T10:05:58Z

@bigjohns97 Did you ever get the igb nonsense figured out?

I have a i211 passed through using proxmox. Get absolutely no eapol traffic on tcpdump in 23.01. I can have proxmox handle the vlan0 tagging which results in virtio drivers for wan. This works, but at line speeds, cpu usage is way high (top in proxmox).

In sophos utm, with the i211 passsed, i'm seeing roughly 1/4 of the cpu usage doing line speed upload/download tests. Would like to move to pf, but this is not acceptable.

bigjohns97 · 2023-04-02T14:30:00Z

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3.
23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine.
This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best.
GL

Hi again! Requesting some more hand holding if you do not mind ;)

You mention "just disable hwvlanfilter", however I am not familiar with this setting, or having ever enabled it. After some exploring, I believe these steps should remove / disable the old custom intel drivers, and have a working router, please let me know if I am correct:

Reverse steps found here: Intel igb/em Interfaces Broken on 2.6/22.01+ #67 (comment)

Remove old intel driver if_igb.ko from /boot/kernel

Remove if_igb_load="YES" and if_igb_name="/boot/modules/if_igb.ko" from /boot/loader.conf

(?) should I keep my current pfatt.sh unchanged? It seems a LOT different than the couple posted here, I believe because I am using bridge mode? Do I, or should I modify it in anyway? (https://github.com/TTK-199837/pfatt/blob/master/bin/pfatt.sh)

Update via GUI, and we are up and running?

Thanks for your time!

You should check out the script I posted at the very top of this thread (3rd post) https://github.com/MonkWho/pfatt/files/10418690/2301_bypass.txt

I made this edit for supplicant mode (bolded part is my edit)
/usr/bin/logger -st "pfatt" "enabling promisc for $ONT_IF..."
/sbin/ifconfig $ONT_IF up
/sbin/ifconfig $ONT_IF promisc
/sbin/ifconfig $ONT_IF -vlanhwfilter

To do the same for bridge mode just add this part to bridge section
/usr/bin/logger -st "pfatt" "enabling promiscuous mode..."
/sbin/ifconfig $EAP_BRIDGE_IF promisc
/sbin/ifconfig $ONT_IF promisc
/sbin/ifconfig $ONT_IF -vlanhwfilter

BTW you are correct on how to remove all of the previous custom driver stuff from the previous version.

Also if you still want to use your script add something like this to the end

echo -n "$(getTimestamp) enabling promiscuous mode on $ONT_IF... "
/sbin/ifconfig $ONT_IF promisc
/sbin/ifconfig $ONT_IF -vlanhwfilter
echo "OK!"

Or you can do a whole new section if you want

echo -n "$(getTimestamp) disabling vlanhwfilter on $ONT_IF... "
/sbin/ifconfig $ONT_IF -vlanhwfilter
echo "OK!"

Hope this helps

bigjohns97 · 2023-04-02T14:34:08Z

@bigjohns97 Did you ever get the igb nonsense figured out?

I have a i211 passed through using proxmox. Get absolutely no eapol traffic on tcpdump in 23.01. I can have proxmox handle the vlan0 tagging which results in virtio drivers for wan. This works, but at line speeds, cpu usage is way high (top in proxmox).

In sophos utm, with the i211 passsed, i'm seeing roughly 1/4 of the cpu usage doing line speed upload/download tests. Would like to move to pf, but this is not acceptable.

If you want try disabling hardware checksum offload in pfsense on the guest and let me know if that solves your CPU usage issues.

(all three of these should be checked)

As to me getting where I would like to be I still haven't been able to get a good understanding why we can't remove netgraph and get it to work, I did get this message from ccope

Which would be great if he shared but I didn't want to push anymore than I already have.

gpz1100 · 2023-04-02T16:04:27Z

@bigjohns97 Unchecking all 3 of those leads to a slight marginal improvement in cpu usage for virtio. How ever the webgui becomes painfully sluggish.

By default the bottom 2 options are check marked.

Can you clarify what is the state of eapol traffic with igb (i211) driver/nic? Should this be working somehow?

TTK-199837 · 2023-04-02T17:25:25Z

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3.
23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine.
This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best.
GL

Hi again! Requesting some more hand holding if you do not mind ;)
You mention "just disable hwvlanfilter", however I am not familiar with this setting, or having ever enabled it. After some exploring, I believe these steps should remove / disable the old custom intel drivers, and have a working router, please let me know if I am correct:

Reverse steps found here: Intel igb/em Interfaces Broken on 2.6/22.01+ #67 (comment)

Remove old intel driver if_igb.ko from /boot/kernel

Remove if_igb_load="YES" and if_igb_name="/boot/modules/if_igb.ko" from /boot/loader.conf

(?) should I keep my current pfatt.sh unchanged? It seems a LOT different than the couple posted here, I believe because I am using bridge mode? Do I, or should I modify it in anyway? (https://github.com/TTK-199837/pfatt/blob/master/bin/pfatt.sh)

Update via GUI, and we are up and running?

Thanks for your time!

You should check out the script I posted at the very top of this thread (3rd post) https://github.com/MonkWho/pfatt/files/10418690/2301_bypass.txt

I made this edit for supplicant mode (bolded part is my edit) /usr/bin/logger -st "pfatt" "enabling promisc for $ONT_IF..." /sbin/ifconfig $ONT_IF up /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter

To do the same for bridge mode just add this part to bridge section /usr/bin/logger -st "pfatt" "enabling promiscuous mode..." /sbin/ifconfig $EAP_BRIDGE_IF promisc /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter

BTW you are correct on how to remove all of the previous custom driver stuff from the previous version.

Also if you still want to use your script add something like this to the end

echo -n "$(getTimestamp) enabling promiscuous mode on $ONT_IF... " /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter echo "OK!"

Or you can do a whole new section if you want

echo -n "$(getTimestamp) disabling vlanhwfilter on $ONT_IF... " /sbin/ifconfig $ONT_IF -vlanhwfilter echo "OK!"

Hope this helps

You are a lifesaver. Thank you very much! Update went smoothly and as planned!

gpz1100 · 2023-04-05T19:15:00Z

Wanted to update. Yes, the netgraph method still works in 23.01 with those ifconfig switches.

What's unclear; Is there still a need for netgraph?

With a i211, I could not get it to process any eapol traffic without netgraph.

aholmes55 · 2023-04-05T19:46:17Z

Want to point out that @bigjohns97's script "2301_bypass.txt" doesn't have the -vlanhwfilter on the bridge mode only on the supplicant mode. Both modes need the -vlanhwfilter on the ifconfig $ONT_IF promisc command.

dreamdenizen · 2023-04-23T08:56:57Z

@bigjohns97 thanks for posting how you got it working and a copy of your script. Got supplicant working for the first time after 1 year on bridged, and now I'm on 23.01 to boot. Thanks again!

ChronicledMonocle · 2023-05-06T17:52:35Z

FYI to those on this thread, all of the UI elements are added into pfSense Plus 23.05 to do this without a script and no netgraph required. You can enable promiscuous mode for the modem interface, set a PCP value for the WAN interface, and setup the 802.1X bridging using the new Ethernet Rules UI under Firewall --> Rules (Need to go to System --> Advanced --> Firewall & NAT --> Ethernet Filtering and enable the option first). I'm currently running an ATT connection bypassed with the bridge method (not wpa_supplicant) and it's been running for a few weeks trouble free.

If you are running an e1000-based NIC on either/both of your modem and WAN interfaces, you will need to disable vlanfwfilter still with an earlyshellcmd script (ifconfig igb# -vlanhwfilter) but any other NIC should just work.

Screenshot of the Ethernet Rules in Action:

dmitripr · 2023-05-17T17:28:24Z

@ChronicledMonocle Great news! Thanks for sharing. Is there a link to step-by-step instructions on setting this up via UI?

dmitripr · 2023-05-17T19:16:28Z

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.

Upgrade to 23.05-RC to get the option for Ethernet Filtering
Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)
Added the new USB interface via Interfaces -> Assignments
The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None
I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem
Added the two rules mentioned by @ChronicledMonocle in the screenshot under Firewall -> Rules -> Ethernet
Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X

This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I tested a restart of the pfSense box and the modem, and this setup survived the restart.

ChronicledMonocle · 2023-05-17T19:41:49Z

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.

* Upgrade to 23.05-RC to get the option for Ethernet Filtering

* Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)

* Added the new USB interface via Interfaces -> Assignments

* The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None

* I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem

* Added the two rules mentioned by @ChronicledMonocle  in the screenshot under Firewall -> Rules -> Ethernet

* Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X

This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I haven't tried to see if this will survive a reboot of the modem and the pfSense router yet.

We have a documentation recipe going up on docs.netgate.com that will have a configuration example of how to do this step by step. I wrote this with input from the pfSense Plus TAC and Development teams for editing and feedback. Attached is a PDF of the draft before it goes live there. Bear in mind you need to be on pfSense Plus 23.05 RC for these instructions to work:
Configuring WAN Connectivity for 802.1X Authentication Bridging and VLAN0 PCP Tagging.pdf

ChronicledMonocle · 2023-05-17T19:51:02Z

Also, the e1000 bug with vlanhwfilter should be fixed in the latest 23.05 RC release, so should be no need to disable vlanhwfilter anymore on those NICs.

dmitripr · 2023-05-17T23:19:54Z

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.
* Upgrade to 23.05-RC to get the option for Ethernet Filtering

* Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)

* Added the new USB interface via Interfaces -> Assignments

* The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None

* I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem

* Added the two rules mentioned by @ChronicledMonocle  in the screenshot under Firewall -> Rules -> Ethernet

* Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X
This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I haven't tried to see if this will survive a reboot of the modem and the pfSense router yet.
We have a documentation recipe going up on docs.netgate.com that will have a configuration example of how to do this step by step. I wrote this with input from the pfSense Plus TAC and Development teams for editing and feedback. Attached is a PDF of the draft before it goes live there. Bear in mind you need to be on pfSense Plus 23.05 RC for these instructions to work: Configuring WAN Connectivity for 802.1X Authentication Bridging and VLAN0 PCP Tagging.pdf

Good stuff. I would also recommend adding DHCPV6 Server and RA basic/recommended settings to the document.

MatthewGCampbell · 2023-05-29T03:24:51Z

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.
* Upgrade to 23.05-RC to get the option for Ethernet Filtering

* Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)

* Added the new USB interface via Interfaces -> Assignments

* The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None

* I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem

* Added the two rules mentioned by @ChronicledMonocle  in the screenshot under Firewall -> Rules -> Ethernet

* Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X
This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I haven't tried to see if this will survive a reboot of the modem and the pfSense router yet.
We have a documentation recipe going up on docs.netgate.com that will have a configuration example of how to do this step by step. I wrote this with input from the pfSense Plus TAC and Development teams for editing and feedback. Attached is a PDF of the draft before it goes live there. Bear in mind you need to be on pfSense Plus 23.05 RC for these instructions to work: Configuring WAN Connectivity for 802.1X Authentication Bridging and VLAN0 PCP Tagging.pdf

I really think this pdf document is great and the write up on docs.netgate.com is even better but I still can’t get this setup working with my SG-5100 I think it may have something to do with the nic’s and not liking the new Ethernet firewall rules, but I get a DHCP address but I’m unable to route traffic. Not sure if anyone else is having this issue or not.

bigjohns97 · 2023-06-09T15:53:23Z

Going to close out this issue as there is a newer version of 23.05 which when included with the new wpa_supplicant binary not only eliminates the need for Netgraph but also the need for stripper switches and even PCP (vlan0) values!

TTK-199837 · 2023-06-12T16:56:56Z

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.
* Upgrade to 23.05-RC to get the option for Ethernet Filtering

* Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)

* Added the new USB interface via Interfaces -> Assignments

* The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None

* I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem

* Added the two rules mentioned by @ChronicledMonocle  in the screenshot under Firewall -> Rules -> Ethernet

* Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X
This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I haven't tried to see if this will survive a reboot of the modem and the pfSense router yet.
We have a documentation recipe going up on docs.netgate.com that will have a configuration example of how to do this step by step. I wrote this with input from the pfSense Plus TAC and Development teams for editing and feedback. Attached is a PDF of the draft before it goes live there. Bear in mind you need to be on pfSense Plus 23.05 RC for these instructions to work: Configuring WAN Connectivity for 802.1X Authentication Bridging and VLAN0 PCP Tagging.pdf
I really think this pdf document is great and the write up on docs.netgate.com is even better but I still can’t get this setup working with my SG-5100 I think it may have something to do with the nic’s and not liking the new Ethernet firewall rules, but I get a DHCP address but I’m unable to route traffic. Not sure if anyone else is having this issue or not.

Hi!

Were you able to get things working for your SG-5100?

TTK-199837 · 2023-06-12T17:37:31Z

And a question for the experts (@bigjohns97) if I update pfsense via the GUI and keep all the current pfatt bypass files and earlyshellcmd scrips intact, will something break? Or will everything work like now, and give me the opportunity to switch over to the new method?

My current plan is to:

Update via GUI and Reboot
Remove the earlyshellcmd for /root/bin/pfatt.sh
Follow https://docs.netgate.com/pfsense/en/latest/recipes/authbridge.html
Reboot and pray

Is this the correct way of doing things?

Thank you!

bigjohns97 · 2023-06-12T17:45:30Z

And a question for the experts (@bigjohns97) if I update pfsense via the GUI and keep all the current pfatt bypass files and earlyshellcmd scrips intact, will something break? Or will everything work like now, and give me the opportunity to switch over to the new method?

Unfortunately I can't answer this as I wasn't running the ngeth based script once 23.05 dropped, I was using a stripper switch by then.

aholmes55 · 2023-06-12T21:20:27Z

@TTK-199837 I just did this. I ended up reflashing to 23.05 as I could never get the upgrade to work correctly. I was using pfatt.sh tethered and then with supplicant on 23.01. On 23.05, I am using the tethered authbridge builtin to pfsense 23.05. Their manual outlines the steps perfectly - make sure you don't miss any steps. One thing I've noticed is that there is a long ~60s delay waiting for the WAN interface during the boot sequence. Not sure what thats all about but it seems like I read other people have it too.

gpz1100 · 2023-06-12T22:05:30Z

One thing I've noticed is that there is a long ~60s delay waiting for the WAN interface during the boot sequence. Not sure what thats all about but it seems like I read other people have it too.

Ahh... That dreaded 60s pause at "configuring wan interface..."

That happens specifically because eapol auth hasn't taken place yet and dhcp can't pull an IP. We ran into the same issue with wpa_supplicant method. The work around was to start the supplicant daemon as earlyshellcmd. Then once past the configuring wan.... something happens to the wan interface leaving wpa_supplicant in an unresponsive (to eapol requests) state. So the solution was to have wpa_supplicant launched twice, once as earlyshellcmd, once as shellcmd. The latter killing the existing process first.

Since you can't quite do that with the bridged method, your best best may be to change the wan dhcp timeout (under advanced wan settings) to something like 10-15s instead of the default 60s.

juvinious · 2023-09-27T12:52:04Z

Wait so are you guys still tethered to the RG like the docs mention? One nic to ONT the other to the RG? https://docs.netgate.com/pfsense/en/latest/recipes/authbridge.html

Is it possible to use extracted certs with wpa_supplicant? I see no mention of it in the docs.
Was hoping to upgrade from 2.6 and continue to use my setup without having to pull the router out of the closet.

5ch17 mentioned this issue May 25, 2023

pfSense 23.05 - EarlyShellCmd Changes - negth0 prevents booting #83

Closed

bigjohns97 closed this as completed Jun 9, 2023

PfSense 23.01/2.7 (FreeBSD 14.0) #79

PfSense 23.01/2.7 (FreeBSD 14.0) #79

Comments

bigjohns97 commented Jan 14, 2023

bigjohns97 commented Jan 14, 2023

bigjohns97 commented Jan 14, 2023 • edited Loading

dmitripr commented Jan 20, 2023

bigjohns97 commented Jan 21, 2023

Shadow00Caster commented Feb 16, 2023

liuhaotian commented Feb 16, 2023

bigjohns97 commented Feb 16, 2023

TTK-199837 commented Feb 19, 2023

bigjohns97 commented Feb 19, 2023

TTK-199837 commented Feb 19, 2023

bigjohns97 commented Feb 19, 2023

TTK-199837 commented Feb 19, 2023 via email

unawarez commented Feb 23, 2023

bigjohns97 commented Feb 24, 2023

unawarez commented Feb 25, 2023 • edited Loading

bigjohns97 commented Feb 25, 2023

unawarez commented Feb 25, 2023

unawarez commented Feb 25, 2023 • edited Loading

bigjohns97 commented Feb 25, 2023

bigjohns97 commented Feb 25, 2023

bigjohns97 commented Feb 25, 2023

unawarez commented Feb 25, 2023

bigjohns97 commented Feb 26, 2023 • edited Loading

unawarez commented Feb 26, 2023 via email

bigjohns97 commented Feb 26, 2023

unawarez commented Feb 26, 2023

bigjohns97 commented Feb 26, 2023

unawarez commented Feb 26, 2023 • edited Loading

bigjohns97 commented Feb 27, 2023

bigjohns97 commented Feb 27, 2023

bigjohns97 commented Feb 27, 2023 • edited Loading

unawarez commented Feb 27, 2023 • edited Loading

mdelise commented Mar 17, 2023 • edited Loading

aholmes55 commented Mar 27, 2023 • edited Loading

bigjohns97 commented Mar 27, 2023

TTK-199837 commented Mar 30, 2023 • edited Loading

gpz1100 commented Apr 2, 2023

bigjohns97 commented Apr 2, 2023

bigjohns97 commented Apr 2, 2023

gpz1100 commented Apr 2, 2023

TTK-199837 commented Apr 2, 2023

gpz1100 commented Apr 5, 2023

aholmes55 commented Apr 5, 2023

dreamdenizen commented Apr 23, 2023

ChronicledMonocle commented May 6, 2023 • edited Loading

dmitripr commented May 17, 2023

dmitripr commented May 17, 2023 • edited Loading

ChronicledMonocle commented May 17, 2023 • edited Loading

ChronicledMonocle commented May 17, 2023

dmitripr commented May 17, 2023

MatthewGCampbell commented May 29, 2023

bigjohns97 commented Jun 9, 2023

TTK-199837 commented Jun 12, 2023

TTK-199837 commented Jun 12, 2023 • edited Loading

bigjohns97 commented Jun 12, 2023

aholmes55 commented Jun 12, 2023 • edited Loading

gpz1100 commented Jun 12, 2023 • edited Loading

juvinious commented Sep 27, 2023

bigjohns97 commented Jan 14, 2023 •

edited

Loading

unawarez commented Feb 25, 2023 •

edited

Loading

unawarez commented Feb 25, 2023 •

edited

Loading

bigjohns97 commented Feb 26, 2023 •

edited

Loading

unawarez commented Feb 26, 2023 •

edited

Loading

bigjohns97 commented Feb 27, 2023 •

edited

Loading

unawarez commented Feb 27, 2023 •

edited

Loading

mdelise commented Mar 17, 2023 •

edited

Loading

aholmes55 commented Mar 27, 2023 •

edited

Loading

TTK-199837 commented Mar 30, 2023 •

edited

Loading

ChronicledMonocle commented May 6, 2023 •

edited

Loading

dmitripr commented May 17, 2023 •

edited

Loading

ChronicledMonocle commented May 17, 2023 •

edited

Loading

TTK-199837 commented Jun 12, 2023 •

edited

Loading

aholmes55 commented Jun 12, 2023 •

edited

Loading

gpz1100 commented Jun 12, 2023 •

edited

Loading