UniFi AP Retries Stopped Working

[jordanwmcdonald]

CONFIGURATION

• Zabbix 7.0.0
• Ubuntu Server 24.04 LTS VM

ISSUE

• All functions of the UniFi AP template were working well until about 5 days ago (I suspect it was related to an update to the AP firmware, but rolling back the AP firmware did not resolve it.).
• Everything still works except for the Wi-Fi Retries Item.

ATTEMPTED RESOLUTIONS

• I did my best to dive into the Templates and compare the Keys with what I was seeing when I ran mca-dump directly SSH'd into an AP, but I could not figure it out.
• Tailing /var/log/zabbix/zabbix_server.log and /tmp/mcaDumpShort.err did not reveal any issues.

Please see attached screenshot showing the Last 7 days in Zabbix and let me know I can assist.

Thank you,
Jordan

[jordanwmcdonald]

@patricegautier

[patricegautier]

ack - I definitely see the problem here as well..

[patricegautier]

Try this latest committed AP template , let me know if that fixes it..

[jordanwmcdonald]

Thank you for your response and commit.

The issue remains - likely because of an oversight on my part. Here is what I did:

1. Verified that I overwrote the old templates during import
2. Removed an AP Host in Zabbix and added it back
3. Rebooted the Ubuntu 24 VM dedicated to Zabbix
4. Waited 6+ hours
5. Verified that the Widget was configured correctly

The 'Last value' shows as '0' each time I checked on it.

[patricegautier]

Ok let's check a couple of things:

In Zabbbix, looking at the list of items for one of the APs not reporting correctly, do you see any red error marks, or even orange (no longer discovered) marks?

anything in the zabbix server logs about that AP?

can you check the 'Unifi AP' template and make sure it's up to date by sending me the preprocessing screen for its active radio discovery rule?

[jordanwmcdonald]

Regarding your first request for marks, I currently do not see any red or orange marks. However, about 10 minutes ago I did, but got sidetracked before I could document them. Upon refreshing the screen several times over the last 10 minutes I have not seen the marks reappear. From memory, the three 'Channel Retries' Items for 6e/na/ng had red marks which, when hovered over, indicated a Timeout. For background, when I first encountered this issue last week I followed your guide's steps on increasing the timeout values in the two locations you indicated so, theoretically, timeouts should not be an issue (unless there's some underlying problem with my network in the lower OSI layers). There were many other items that, at that time, had red marks (none had orange marks). Again, from memory, when I hovered over anything other than 'Channel Retries' these Items had a much longer error message related to mca-dump.

Evaluating '/var/log/zabbix/zabbix_server.log' reveals this sequence repeating every 10-15 minutes:

 1039:20240714:163635.323 executing housekeeper
  1039:20240714:163649.108 housekeeper [deleted 109009 hist/trends, 375 items/triggers, 40 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 autoreg_host, 0 records in 13.781315 sec, idle for 1 hour(s)]
  1067:20240714:164323.798 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '10.24.10.42' '-P' '{$UNIFI_SSH_PORT}' '-u' 'jordan' '-i' '/home/jordan/.ssh/zabbix/zb_id_rsa' '-t' 'AP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '25' '-b'": Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:mca-dump-short.sh["-d","{HOST.CONN}", "-P", "{$UNIFI_SSH_PORT}", "-u", "{$UNIFI_USER}", "-i", "{$UNIFI_SSH_PRIV_KEY_PATH}", "-t", "AP", "-p", "{$UNIFI_SSHPASS_PASSWORD_PATH}", "-o", "{$UNIFI_CHECK_TIMEOUT}","-b"]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:client_count" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cpu" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:hostname" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:load_avg_15mn" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:load_avg_1mn" became not supported: Timeout while executing a shell script.
  1037:20240714:164323.798 discovery rule "EFCP-AP6E-042:active_radio_discovery" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:load_avg_5mn" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:mem_total" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:mem_used" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:sat" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:uptime" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:wireless_reception_bandwidth" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:wireless_transmission_bandwidth" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:channel[ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:channel[na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:channel[6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_rx[ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_rx[na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_rx[6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_total[ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_total[na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_total[6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_tx[ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_tx[na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:cu_tx[6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:retries[ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:retries[na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:retries[6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:client_count[11100030-6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:client_count[11100030-na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:client_count[11100050-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:client_count[11100030-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:rx_bandwidth[11100030-6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:rx_bandwidth[11100030-na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:rx_bandwidth[11100050-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:rx_bandwidth[11100030-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.798 item "EFCP-AP6E-042:tx_bandwidth[11100030-6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:tx_bandwidth[11100030-na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:tx_bandwidth[11100050-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:tx_bandwidth[11100030-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:client_count[11100024-6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:client_count[11100024-na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:client_count[11100024-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:rx_bandwidth[11100024-6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:rx_bandwidth[11100024-na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:rx_bandwidth[11100024-ng]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:tx_bandwidth[11100024-6e]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:tx_bandwidth[11100024-na]" became not supported: Timeout while executing a shell script.
  1048:20240714:164323.799 item "EFCP-AP6E-042:tx_bandwidth[11100024-ng]" became not supported: Timeout while executing a shell script.
  1037:20240714:164323.802 discovery rule "EFCP-AP6E-042:wifi_network_discovery" became not supported: Timeout while executing a shell script.
  1048:20240714:164351.485 item "EFCP-AP6E-042:cint[ng]" became not supported: Cannot evaluate function: item "/EFCP-AP6E-042/cu_total[ng]" is not supported at "last( //cu_total[ng] )- last( //cu_tx[ng])-last(//cu_rx[ng]))>0)*(last( //cu_total[ng] )- last( //cu_tx[ng])-last(//cu_rx[ng]))".
  1048:20240714:164352.485 item "EFCP-AP6E-042:cint[na]" became not supported: Cannot evaluate function: item "/EFCP-AP6E-042/cu_total[na]" is not supported at "last( //cu_total[na] )- last( //cu_tx[na])-last(//cu_rx[na]))>0)*(last( //cu_total[na] )- last( //cu_tx[na])-last(//cu_rx[na]))".
  1048:20240714:164353.486 item "EFCP-AP6E-042:cint[6e]" became not supported: Cannot evaluate function: item "/EFCP-AP6E-042/cu_total[6e]" is not supported at "last( //cu_total[6e] )- last( //cu_tx[6e])-last(//cu_rx[6e]))>0)*(last( //cu_total[6e] )- last( //cu_tx[6e])-last(//cu_rx[6e]))".
  1048:20240714:164421.510 item "EFCP-AP6E-042:memory" became not supported: Cannot evaluate function: item "/EFCP-AP6E-042/mem_used" is not supported at "last(//mem_used) / last(//mem_total)".
  1064:20240714:164423.587 Failed to execute command "/usr/lib/zabbix/externalscripts/mca-dump-short.sh '-d' '10.24.10.42' '-P' '{$UNIFI_SSH_PORT}' '-u' 'jordan' '-i' '/home/jordan/.ssh/zabbix/zb_id_rsa' '-t' 'AP' '-p' '{$UNIFI_SSHPASS_PASSWORD_PATH}' '-o' '25' '-b'": Timeout while executing a shell script.
  1048:20240714:164523.027 item "EFCP-AP6E-042:mca-dump-short.sh["-d","{HOST.CONN}", "-P", "{$UNIFI_SSH_PORT}", "-u", "{$UNIFI_USER}", "-i", "{$UNIFI_SSH_PRIV_KEY_PATH}", "-t", "AP", "-p", "{$UNIFI_SSHPASS_PASSWORD_PATH}", "-o", "{$UNIFI_CHECK_TIMEOUT}","-b"]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cpu" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:hostname" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:load_avg_15mn" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:load_avg_1mn" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:load_avg_5mn" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:mem_total" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:mem_used" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:sat" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:uptime" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:channel[ng]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:channel[na]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:channel[6e]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_rx[ng]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_rx[na]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_rx[6e]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_total[ng]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_total[na]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_total[6e]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_tx[ng]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_tx[na]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:cu_tx[6e]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count[11100030-6e]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count[11100030-na]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count[11100050-ng]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count[11100030-ng]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count[11100024-6e]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count[11100024-na]" became supported
  1048:20240714:164523.027 item "EFCP-AP6E-042:client_count[11100024-ng]" became supported
  1036:20240714:164523.040 discovery rule "EFCP-AP6E-042:active_radio_discovery" became supported
  1036:20240714:164523.068 discovery rule "EFCP-AP6E-042:wifi_network_discovery" became supported
  1049:20240714:164551.695 item "EFCP-AP6E-042:cint[ng]" became supported
  1048:20240714:164552.603 item "EFCP-AP6E-042:cint[na]" became supported
  1048:20240714:164553.604 item "EFCP-AP6E-042:cint[6e]" became supported
  1048:20240714:164621.631 item "EFCP-AP6E-042:memory" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:wireless_reception_bandwidth" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:wireless_transmission_bandwidth" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:retries[ng]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:retries[na]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:retries[6e]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:rx_bandwidth[11100030-6e]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:rx_bandwidth[11100030-na]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:rx_bandwidth[11100050-ng]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:rx_bandwidth[11100030-ng]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:tx_bandwidth[11100030-6e]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:tx_bandwidth[11100030-na]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:tx_bandwidth[11100050-ng]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:tx_bandwidth[11100030-ng]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:rx_bandwidth[11100024-6e]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:rx_bandwidth[11100024-na]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:rx_bandwidth[11100024-ng]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:tx_bandwidth[11100024-6e]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:tx_bandwidth[11100024-na]" became supported
  1048:20240714:164622.814 item "EFCP-AP6E-042:tx_bandwidth[11100024-ng]" became supported

'Unifi AP' template preprocessing screen for its active radio discovery rule (attached). The full text of the '2. JSONPath' is:

$.radio_table.[?(@.athstats.name != "" && (@.athstats.noise_floor < 0 || @.athstats.cu_total > 0))]