nexmon blindness bug (brcmf_cfg80211_nexmon_set_channel)

[evilsocket]

every once in a while, nexmon dies with:

[ 4341.527847] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 4344.327806] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 4347.127853] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 4349.927917] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 4352.728074] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 4355.527970] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 4358.328022] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 4361.208095] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 4364.008157] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 4366.808218] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 4369.608431] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 4372.408345] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 4375.288408] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 4378.088474] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 4380.891399] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110

And only a reboot can fix the wifi, this is why the mon_max_blind_epochs parameter exists, to reboot the board when this happens.

Ideally we should document this known issue, the configuration and some day maybe fix it.

[evilsocket]

cc @hexwaxwing

[evilsocket]

docs added https://pwnagotchi.ai/usage/#known-issues

[fregkos]

You could try reloading the driver instead of rebooting the pi, using :

modprobe -r brcmfmac
modprobe brcmfmac

If this still fails, then a reboot will fix this.
There is a great discussion about that problem here : https://www.bountysource.com/issues/56252669-wlan-freezes-in-raspberry-pi-3b

[evilsocket]

tried that way, it doesn't always work, the only reliable way is rebooting

[jetbalsa]

So, The Nexmon firmware is a little picky on how its interfaced, How are your bringing up the mon interface in linux before bettercap gets to it?

[evilsocket]

iw phy phy0 interface add mon0 type monitor && ifconfig mon0 up

[evilsocket]

from https://github.com/evilsocket/pwnagotchi/blob/master/builder/pwnagotchi.yml#L333

[evilsocket]

(from re4son monstart script)

[jetbalsa]

I just noticed on the Nexmon Repo that the bcm43455c0 does not support wifi frame injection. That might be the reason for the drivers crashing as they are not handling the requests to do frame injection correctly from bettercap.

[jetbalsa]

Ok, So I looked, We are using the older version of the firmware provided by Nexmon, I think Re4son kernel does it as its the default for Nexmon to use that. Patches and Commits from the Nexmon Project show 7.45.189 as the latest version you can use (The base firmware comes from the OEM, not the RPI foundation) as of RC4, we are using version 7.45.154 of the bcm43455c0 firmware This issue should only affect 3B+ and 4s, the 3B and the 0W use the same wifi chip and are listed as supporting injection.

[jetbalsa]

More Details, I've looked into how the Re4son kernel builder pulls down its firmware, its even /worse/ Looks like it pulls from https://github.com/Re4son/re4son-nexmon as its source of nexmon patches... its years out of date. I'm doing some prototyping to update this now.

[jetbalsa]

Good News, The Nexmon Patches with the stock kernel works well, Here is my DMesg output after running for 10 minutes, I will be running it for the next 24 hours to see if its stable,

[    5.128852] brcmfmac: loading out-of-tree module taints kernel.
[    5.128864] brcmfmac: loading out-of-tree module taints kernel.
[    5.181306] brcmfmac: F1 signature read @0x18000000=0x15264345
[    5.190412] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    5.190907] usbcore: registered new interface driver brcmfmac
[    5.566003] brcmfmac: brcmf_sdio_bus_preinit: before brcmf_sdio_debugfs_create
[    5.569776] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    5.597271] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Oct 15 2019 20:30:25 version 7.45.189 (nexmon.org: -4) FWID 01-e1db26e2
[    5.675296] brcmfmac: brcmf_bus_started: before brcmf_debugfs_add_entry
[    8.495009] brcmfmac: brcmf_vif_add_validate: Attempt to add a MONITOR interface...
[    8.495024] brcmfmac: brcmf_mon_add_vif: brcmf_mon_add_vif called
[    8.495028] brcmfmac: brcmf_mon_add_vif: Adding vif "mon0"

Here is my uname

Linux pwnagotchi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

[jetbalsa]

And I failed, You get a little more debug output this time overall..

[ 5387.511962] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 5390.551934] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 5393.601927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5396.631949] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 5399.671984] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5402.711975] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5405.752020] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 5408.791991] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5411.831985] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4106, -110
[ 5414.872013] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53284, -110
[ 5417.911966] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53288, -110
[ 5420.951985] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53296, -110
[ 5423.511967] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5426.312006] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 5429.111938] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5431.911970] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5434.711974] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53308, -110
[ 5437.511963] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5440.311927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53356, -110
[ 5443.111922] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53360, -110
[ 5445.911981] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53368, -110
[ 5448.711973] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53372, -110
[ 5451.511925] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53380, -110
[ 5454.311928] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5457.111929] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 5459.911983] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5462.711974] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5465.511924] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53308, -110
[ 5468.311926] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5471.111925] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53356, -110
[ 5473.912031] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53360, -110
[ 5476.471932] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5479.271932] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5482.081987] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 5484.871994] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5487.671927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53292, -110
[ 5490.471930] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53300, -110
[ 5493.271977] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53304, -110
[ 5496.071923] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5498.872004] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53364, -110
[ 5501.671976] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53372, -110
[ 5504.471928] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53376, -110
[ 5507.031989] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 5507.032002] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)
[ 5509.591930] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 5509.591939] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110)
[ 5512.151989] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5514.952027] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110

Edit:
Added brcmfmac.debug=30 to /boot/cmdline.txt to try and get a better look at what the driver is doing, It enables full trace mode for the drivers, I have attacked a full debug log up to where it strops responding.
syslog.gz

[evilsocket]

yep when that happens even trying to change channel manually doesn't work, i think it's the heat

[jetbalsa]

Based off this photo, There is no TIM under the wifi can at all... Since its used for shielding RF, this is of no surprise. Someone should get a IR Camera on the board and take a look.

Also You notice that White IC in the top left, Thats the same chip they used to show people the SuperMicro Implant... lulz

[jetbalsa]

So, I'm looking at the datasheet, for the CYW43455 It states that the Max temp for operation is 120C while under normal loads on a 4 layer board Of course Max oper temp is +85C The thing only puts out 1.2W but I did notice that on the older RPI0W Chips it had a self limiter for overheat, This one is not stated in the data sheet as having one

[evilsocket]

so you're saying that it's the chinese sabotaging our wifi pwning, right?

[jetbalsa]

Maybe.... Until I can get a proper Temp readout of the die itself while under the heavy load, We wont know.

[DrSchottky]

By default airodump hops every 250ms, so the only difference that comes to my mind is that you do it through wext, while airodump uses (AFAIK) nl.
@andrewbeard could you try to disable hopping as evlisocket suggested?

[evilsocket]

@DrSchottky yes i think it's not the timing but the methodology ... i also don't think airodump uses calls to the iwconfig binary as i do in bettercap, pretty sure there's a cleaner way to do it :D

[DrSchottky]

@evilsocket exactly, it uses a totally different software stack (wext vs nl80211).
If turning off the hopping solves the problem the easiest fix is to replace iwconfig with iw in bettercap.

[DrSchottky]

@andrewbeard any news?

[neurovish]

I get the "brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4109, -110" errors after about an hour when I have bluetooth enabled in the config, but bluetooth does not connect for whatever reason. pi0w, temp shows as 41c. pwnagotchi process locks and goes defunct when this happens. Is this the same bug?

Edit: Just saw the link to #494

[TJFJefroy]

I got it "working" by writing a little script that reloads the driver everytime it crashes

[soaringswine]

I got it "working" by writing a little script that reloads the driver everytime it crashes

do you mind posting your script? "modprobe -r brcmfmac && modprobe brcmfmac && sleep 2 && /usr/bin/monstart && systemctl restart pwnagotchi" for the driver reload seems to kinda work some of the time for me but not always. wondering if you're doing something better.

[TJFJefroy]

echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/unbind
echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/unbind

echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/bind
echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/bind

this is working for me

[soaringswine]

echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/unbind
echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/unbind

echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/bind
echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/bind

this is working for me

are you running this while pwnagotchi service is still running? are you doing anything more than the driver un/binding like trying to add mon0 back or bring it up? and are you monitoring dmesg for brcmf_cfg80211_nexmon_set_channel errors to trigger it?

because it doesn't look like mon0 is brought back up after the binds and trying to "/usr/bin/monstart" or "iw phy phy0 interface add mon0 type monitor && ifconfig mon0 up" gives a "command failed: No such file or directory (-2)" error for me.

[soaringswine]

echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/unbind
echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/unbind
echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/bind
echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/bind
this is working for me

are you running this while pwnagotchi service is still running? are you doing anything more than the driver un/binding like trying to add mon0 back or bring it up? and are you monitoring dmesg for brcmf_cfg80211_nexmon_set_channel errors to trigger it?

because it doesn't look like mon0 is brought back up after the binds and trying to "/usr/bin/monstart" or "iw phy phy0 interface add mon0 type monitor && ifconfig mon0 up" gives a "command failed: No such file or directory (-2)" error for me.

ah, the error is because phy0 doesn't exist. when you rebind the driver, it increments the phy instance, so phy1, phy2 instead of phy0. this command seems to be more consistent (though I don't know if functionally it's any different): "iw dev wlan0 interface add mon0 type monitor && ifconfig mon0 up"

[soaringswine]

here's my nexmon blindness error watchdog script I whipped together that seems to work pretty well until the bug is fixed. needs to be run as root. seems to do its job from the testing I've done:

#!/usr/bin/env bash
set -euo pipefail
red=$(tput setaf 1)
green=$(tput setaf 2)
yellow=$(tput setaf 3)
reset=$(tput sgr0)

if ! [ $(id -u) = 0 ]; then
   echo ${red}"need to run this as root!${reset}"
   exit 1
fi

echo "${yellow}patiently waiting for nexmon blindness error (https://github.com/evilsocket/pwnagotchi/issues/267)..${reset}"
tail -n 0 -f /var/log/kern.log | while read -r kernel_log_line; do
	if [[ $kernel_log_line == *"brcmf_cfg80211_nexmon_set_channel: Set Channel failed"* || $kernel_log_line == *"brcmf_cfg80211_get_tx_power: error"* || $kernel_log_line == *"brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed"* ]]; then
		echo "${red}caught blindness error, reloading nexmon driver..${reset}"
		set -x
		echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/unbind
		sleep 1
		echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/unbind
		sleep 3
		echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/bind
		sleep 3
		echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/bind
		sleep 3
		modprobe brcmfmac #this modprobe may not be needed, but it seems to help sometimes.
		sleep 3
		#running a couple extra binds here in case the modprobe made a difference. it can't hurt!
		echo -n "mmc1:0001:1" > /sys/bus/sdio/drivers/brcmfmac/bind || true
		sleep 3
		echo -n "mmc1:0001:2" > /sys/bus/sdio/drivers/brcmfmac/bind || true
		sleep 3
		iw dev wlan0 interface add mon0 type monitor
		sleep 3
		ifconfig mon0 up
		set +x
		date
		echo "${green}nexmon driver reloaded and mon0 brought back up! looping until next error..${reset}"
	fi
done

[myusuf3]

I just read through this thread I was confident @DrSchottky and @andrewbeard where on to something.

[DrSchottky]

@myusuf3 I never heard anything back.
What should be tried is written a few posts above, so feel free to try and give a feedback.

[dadav]

Seems like there is a fix available (raspberrypi/linux#2453) ...trying this right now

[dadav]

not working, but there've been some kernel updates. I'll try them

[gwbres]

any updates guys? facing similar issues on RPi0w.

"brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110"

airodum has not picked up anything so far, and the bug happens right away for me.
Although iwlist scan does work properly.
The way I bring the interface up does not seem to make a difference.
I haven't tried disabling hopping or increasing delay between two channels yet

The people from pwnagotchi managed to get it working, I should be able to get better results, there must be something I'm missing here

I switched to the firmware ("latest"?) provided by cypress (sdio.bin..) from the .tar.gz someone uploaded here, did not improve things, did not make it worse either

I'm using the module from DrShottsky's fork (4.19), here's the log when module is registered or manually inserted

dmesg | grep brcm
[ 3.593952] brcmfmac: loading out-of-tree module taints kernel.
[ 3.631407] brcmfmac: F1 signature read @0x18000000=0x1541a9a6
[ 3.647217] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
[ 3.677400] usbcore: registered new interface driver brcmfmac
[ 4.013755] brcmfmac: brcmf_sdio_bus_preinit: before brcmf_sdio_debugfs_create
[ 4.029060] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
[ 4.060116] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM43430/1 wl0: Oct 22 2019 01:57:42 version 7.45.98.94 (r723000 CY) FWID 01-73a5ed62
[ 4.285213] brcmfmac: brcmf_bus_started: before brcmf_debugfs_add_entry

iwconfig wlan0

wlan0 IEEE 802.11 Mode:Monitor Frequency:2.412 GHz Tx-Power=31 dBm
Retry short limit:7 RTS thr:off Fragment thr:off
Power Management:on

iwconfig wlan0 power off
Error for wireless request "Set Power Management" (8B2C) :
SET failed on device wlan0 ; Invalid argument.

[meliodasren]

For the Pi 4, could the latest nexmon firmware fix this issue?
bcm43455c0 | 7_45_206
I am trying to test it myself but I am stuck trying to figure out why airodump-ng does not find anything.

[NetherStar64]

Any Update? I'm facing the same problem reproducing it 100% consitent. It seems that this bug is still not fixed and still persistent in the latest Kali Linux for ARM builds for the Raspberry Pi.

[evilsocket]

@NetherStar64 this is a nexmon bug, you're asking for updates in the wrong repo :)

[NetherStar64]

@NetherStar64 this is a nexmon bug, you're asking for updates in the wrong repo :)

Oh ok sorry.

[DrSchottky]

@NetherStar64 if you can reproduce the issue I can spare a few hours trying to debug it.
Hit me up on Tw/Tg (same handle) if interested in doing tests.

[NetherStar64]

@DrSchottky Yes I really would like to help you. Where can I contact you?

[DrSchottky]

@NetherStar64 Twitter, Telegram or Discord(DrSchottky#4172)

[sturles]

Could the AI be used to figure out how this bug is triggered and avoid it in the future?

It will require faster detection of the bug. E.g. by checking which channel the wifi is on every time after changing it, to detect the moment a change fails. Useful functions for this can be found in the source for aircrack-ng in src/osdep/linux.c.

The trigger could be anything from receiving a bad packet to sending packages to fast or slow or issues related to the package size or combinations of packages with different size.

[NetherStar64]

I think an AI would be pretty overkill for this Bug. In seemoo-lab/nexmon#335 (comment) there's already a proposed Patch that fixes the Crash 99% of the time, but it still crashes about ~30-60min while constantly injecting Packets. sturles <[email protected]> schrieb am Di., 2. Feb. 2021, 16:41:

…

Could the AI be used to figure out how this bug is triggered and avoid it in the future? It will require faster detection of the bug. E.g. by checking which channel the wifi is on every time after changing it, to detect the moment a change fails. Useful functions for this can be found in the source for aircrack-ng in src/osdep/linux.c. The trigger could be anything from receiving a bad packet to sending packages to fast or slow or issues related to the package size or combinations of packages with different size. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#267 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEXFJXF3XUXLMVVHHK6TNDS5AMJXANCNFSM4JADVA7A> .

[garryfieldd]

Hey guys, I'm new here. I checked an nexmon GitHub page, and in compatibility it says, that frame injection is supported for raspi4 on raspberry pi os (kerner 5.4 and above), not Raspbian. I'll try to install pwnagotchi on rpios in couple of days, and get back to you.

[mbattistini]

Any solutions or workarounds that can be implemented?

[aluminum-ice]

I do not think this is an issue anymore. I see these entries in my log (e.g., brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110) and yet my pwnagotchi is still capturing packets successfully if I just let it run. I also monitor the ai observations and as long as those are not blank for any given epoch (unless you're out in the middle of nowhere), it seems like you can ignore these errors. I turned off the watchdog plugin on mine because it was needlessly reboot the pwnagocthi.