Audio analysis failing #208
Comments
|
Hi, birdnet pi is super stable on a rpi4. You should start to check wfh your power supply, sd card, and stability of microphone (is it a rtsp from a camera or a true mix) |
|
That's great info. I'm using RTSP and I thing the recording service doesn't recover gracefully if there's a hiccup in the stream. I may add a restart for the service to crontab. |
|
Hi, here is a script for a custom-service.d that monitors status of recording & analysis services and restarts them if stopped. It also sends apprise alerts in case of issues. It also solves bottlnecks of analysis by stopping the recorder for a while (useful on less powerful devices). It's adapted from mcguirepr89#393 (comment) |
|
Just chiming in here that I've noticed similar instability over the past 1-1.5 months after many consecutive months of stability running on a Raspberry Pi 5 8gb. I'm suspecting it started after apt updates, but it is hard to be sure (I apply updates and patches about bi-weekly). I am not using an RTSP mic, but rather a directly attached USB mic, I am using a good USB 3.0 boot drive rather than an SD card and I noticed that the analysis service will randomly crash similar to what @justlen1 reported. @alexbelgium is it possible to embed that monitoring script in this project's normal installation and have it running by default? Or is there a downside to doing so? |
|
Hi, there is no downside that I'm aware off (except that it forces the services to be active) and I'm running it already by default on the birdnet-pi installs using my homeassistant add-on. You just need to copy the file in /etc/custom-services.d, give it executable status and and it will execute on boot If you don't have /etc/custom-services.d you might use a cron job on @reboot instead Edit : if it is confirmed there is interest and the script works I can create a PR to add it to this repo |
|
Wow, what a response, thanks. My next question was going to be how to install it, but that got answered too. Many thanks |
|
Well crap. I installed as per the instructions above, and I am not getting any processing at all. (I am using RTSP). I did verify I have 30-monitoring.sh in /etc/custom-services.d: 17:15:57---[server][INFO] LOADING TF LITE MODEL... |
|
Aaaand suddenly it started working... 17:24:28---[birdnet_analysis][INFO] Analyzing BirdSongs/StreamData/2024-10-10-birdnet-RTSP_1-17:24:05.wav |
|
After a few days, I've noticed the audio analysis stops and I do not receive any notification. For the time being I have gone back to restarting the audio analysis periodically via crontab. Is there a way to verify 30-monitoring.sh is running? I ran the commands: sudo mkdir -p /etc/custom-services.d
sudo wget https://raw.githubusercontent.com/alexbelgium/hassio-addons/master/birdnet-pi/rootfs/custom-services.d/30-monitoring.sh -O /etc/custom-services.d/30-monitoring.sh
sudo chown $USER:$USER /etc/custom-services.d/30-monitoring.sh
sudo chmod +x /etc/custom-services.d/30-monitoring.sh |
|
Hi, well it IS supposed to print debug infos at each run - perhaps they get stuck in the log ? For me, on docker, they appear in the container log. I'll convert the script to systemctl so it is managed in the same manner as other services - this would be much more coherent. While I will be doing this, I had also the idea to change the logic :
The issue is that it modifies the main code, but the advantage is that it becomes a much more elegant solution more coherent with the global behaviors Edit : I didn't adress your main question, it all depends if your distribution supports /etc/custom-services.d execution at runtime. Actually you could try to execute it manually with |
|
@justlen1 looking at the logs and the timestamps therin, it seems that the analysis service is doing just fine. It seems to me that it is the recording service that somehow fails to write out any *.wav files sidenote: when the analysis service does not see any new wav file for a while it will restart. this is to ensure it is not just missing the kernel's i/o notificaions for some reason. that is what is hapening in your first post |
|
@Nachtzuster */30 * * * * systemctl restart birdnet_recording.service |
I can confirm that my installation has been rock solid for 7 days since adding this whereas it was failing after about 12-24 hours, thank you so much! I think this would definitely be useful to be integrated in the default installation. |
|
@justlen1 could you test the when it kicks in, you'll see this in the birdnet_recording.service log: |
|
Hi, thanks! With a bookworm distribution with an up-to-date ffmpeg I get |
|
@alexbelgium thanks for testing! |
|
I'll try another ffmpeg version... not sure too - it should be default since v4.0 of ffmpeg. If I find a solution, I'll post here. For info, I'm on rpi4 aarch64 using a dockerized bookworm fully up-to-date (according to apt). Edit :
Edit 2 : didn't work with ffmpeg 6.0.1 either, same behavior |
|
Sorry for the delay, was doing some traveling. Didn't work for me on Debian Bookworm and FFMPEG 5.1.6-0+deb12u1+rpt1. I attempted to restart the Recording service in the web UI and see this: So now I'm pretty much dead in the water. Can I get back to where I was by simply deleting //etc/custom-services.d/30-monitoring.sh? Or where I can find the actual recording service log? Edit: I was able to get things running gain by reverting the birdnet_recording.sh file. |
|
@justlen1 Thanks. I think I know how I goofed up. |
|
@justlen1 I pushed a fix to that testing branch, could you give it another try and see if it fixes the hanging of ffmpeg? |
I’m traveling cross country via train and won’t be home until next Tuesday. I’ll test it as soon as I get home. thanks for everything you’re doing. |
|
@justlen1 could you provide a status update? - saying 'have not had time' is ok :-) |
|
I can try it tomorrow and report back. I’ll do it as early as I can. |
|
I'm trying it out now. I disabled my crontab and I am updating with the instructions you provded on 10/27. |
|
I updated to the branch using the following: git remote remove origin
git remote add origin https://github.com/Nachtzuster/BirdNET-Pi.git
git fetch
./scripts/update_birdnet.sh -b rtsp_timeoutThe recording didn't come up, so I rebooted and restarted the recording service when I did I got: ● birdnet_recording.service - BirdNET Recording
Loaded: loaded (/etc/systemd/system/birdnet_recording.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-12-03 15:09:09 CST; 2s ago
Main PID: 7157 (bash)
Tasks: 2 (limit: 3909)
CPU: 2.950s
CGroup: /system.slice/birdnet_recording.service
├─7157 bash /usr/local/bin/birdnet_recording.sh
└─7214 ffmpeg -hide_banner -loglevel error -nostdin -vn -thread_queue_size 512 -stimeout 10000000 -i rtsp://justlen:hixX9A-w3_NnTEmE4@@172.25.84.10/Preview_01_main -map 0:a:0 -t 15 -acodec pcm_s16le -ac 2 -ar 48000 file:/home/birder/BirdSongs/StreamData/2024-12-03-birdnet-RTSP_1-15:09:12.wav -vn -thread_queue_size 512 -stimeout 10000000 -i rtsp://justlen:hixX9A-w3_NnTEmE4@@172.25.84.11/Preview_01_main -map 1:a:0 -t 15 -acodec pcm_s16le -ac 2 -ar 48000 file:/home/birder/BirdSongs/StreamData/2024-12-03-birdnet-RTSP_2-15:09:12.wav
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7189]: Unrecognized option 'stimeout'.
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7189]: Error splitting the argument list: Option not found
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7194]: Unrecognized option 'stimeout'.
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7194]: Error splitting the argument list: Option not found
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7199]: Unrecognized option 'stimeout'.
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7199]: Error splitting the argument list: Option not found
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7204]: Unrecognized option 'stimeout'.
Dec 03 15:09:11 birdnetpi birdnet_recording.sh[7204]: Error splitting the argument list: Option not found
Dec 03 15:09:12 birdnetpi birdnet_recording.sh[7209]: Unrecognized option 'stimeout'.
Dec 03 15:09:12 birdnetpi birdnet_recording.sh[7209]: Error splitting the argument list: Option not foundI'm going to revert to the base configuration by running the update via the web UI. I will wait to hear if you want me to try anything else. I actually have a good bit of time this week and can devoite some attention to this. Sorry about the delay in getting back to you. |
|
The only thing I DID NOT do is run an update via the web interface before I ran the branch. Please let me know if you'd like me to try it again. |
|
Looks like could you run |
I ran JUST that line and it appears to be working. My birdnet log dropped the connected and when I reloaded the page it appears to be working. Do you want me to do anything else other than to check it from time to time? |
|
Just checking if it keeps running without the cron restart job is probably enough. |
I disabled the cron job when I loaded the branch. It’s still running. |
|
@justlen1 Still running fine? |
Chugging away, Sir. I don’t know if there’s a way to tell if the recorder has been restarted, but every time I check it’s running. Thank you. |
|
You can run |
Looks like PID 130911 ● birdnet_recording.service - BirdNET Recording
Loaded: loaded (/etc/systemd/system/birdnet_recording.service; enabled; preset: enabled)
Active: active (running) since Wed 2024-12-04 14:52:09 CST; 3 days ago
Main PID: 130911 (bash)
Tasks: 12 (limit: 3909)
CPU: 5h 30min 1.864s
CGroup: /system.slice/birdnet_recording.service
├─130911 bash /usr/local/bin/birdnet_recording.sh
└─526957 ffmpeg -hide_banner -loglevel error -nostdin -vn -thread_queue_size 512 -timeout 10000000 -i rtsp>
Dec 07 19:29:45 birdnetpi birdnet_recording.sh[519600]: [h264 @ 0x55b65fba20] cabac decode of qscale diff failed at 22 >
Dec 07 19:29:45 birdnetpi birdnet_recording.sh[519600]: [h264 @ 0x55b65fba20] error while decoding MB 22 88, bytestream>
Dec 07 20:46:20 birdnetpi birdnet_recording.sh[526133]: [h264 @ 0x5584398a20] cabac decode of qscale diff failed at 17 >
Dec 07 20:46:20 birdnetpi birdnet_recording.sh[526133]: [h264 @ 0x5584398a20] error while decoding MB 17 87, bytestream>
Dec 07 20:54:21 birdnetpi birdnet_recording.sh[526795]: [h264 @ 0x55a7867a20] left block unavailable for requested intr>
Dec 07 20:54:21 birdnetpi birdnet_recording.sh[526795]: [h264 @ 0x55a7867a20] error while decoding MB 0 13, bytestream >
Notice: journal has been rotated since unit was started, output may be incomplete.And if I am reading this correctly it has been running for 1:27, which means it has been successfully restarted. Or it's been running since 12/4, and my issue automagically fixed itself. At one point I had three recording devices running and I recently cut it back to two, which may have improved the reliability of the process. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
birder 4761 0.1 0.2 19704 10240 ? Ss Dec03 10:18 /lib/systemd/systemd --user
birder 4762 0.0 0.1 170712 5476 ? S Dec03 0:00 (sd-pam)
birder 4783 0.0 0.3 126348 12416 ? Ssl Dec03 0:00 /usr/bin/pipewire
birder 4785 0.0 0.3 1237216 12056 ? Ssl Dec03 4:10 /usr/bin/rpi-connectd -socket /run/user/1000/rpi-conn
birder 4786 0.0 0.9 699760 38276 ? Ssl Dec03 0:01 /usr/bin/wireplumber
birder 4787 0.0 0.2 102108 7808 ? Ssl Dec03 0:00 /usr/bin/pipewire-pulse
birder 4793 0.0 0.1 19728 6560 ? S Dec03 0:00 sshd: birder@pts/1
birder 4800 0.0 0.0 8236 3712 ? Ss Dec03 0:00 /usr/bin/dbus-daemon --session --address=systemd: --n
birder 4809 0.0 0.1 8624 5120 pts/1 Ss Dec03 0:00 -bash
birder 104097 0.0 0.2 14608 8064 pts/1 S+ Dec04 0:00 ssh [email protected]
birder 104169 0.0 0.1 19728 6560 ? S Dec04 0:00 sshd: birder@pts/2
birder 104170 0.0 0.1 8624 5248 pts/2 Ss+ Dec04 0:00 -bash
birder 130822 0.0 0.0 6800 3072 ? Ss Dec04 0:00 bash /usr/local/bin/livestream.sh
birder 130824 0.0 1.2 276120 50128 ? SLl Dec04 2:12 ffmpeg -nostdin -loglevel error -ac 2 -i rtsp://justl
birder 130873 1.6 6.4 778804 251732 ? Ssl Dec04 78:35 /home/birder/BirdNET-Pi/birdnet/bin/python3 /usr/loca
birder 130878 0.0 0.0 6800 2944 ? Ss Dec04 0:00 bash /usr/local/bin/spectrogram.sh
birder 130881 0.0 0.0 2504 1280 ? S Dec04 0:02 inotifywait -m -e close_write /home/birder/BirdSongs/
birder 130882 0.0 0.0 7696 2344 ? S Dec04 1:02 bash /usr/local/bin/spectrogram.sh
birder 130911 0.0 0.0 6928 3072 ? Ss Dec04 1:27 bash /usr/local/bin/birdnet_recording.sh
birder 130937 0.0 0.0 6800 3072 ? Ss Dec04 0:00 bash /usr/local/bin/gotty --address localhost -p 8080
birder 130943 0.0 0.4 1525568 16128 ? Sl Dec04 0:27 gotty-aarch64 --address localhost -p 8080 --path log
birder 130954 0.0 3.8 664448 148892 ? Ssl Dec04 0:16 /home/birder/BirdNET-Pi/birdnet/bin/python3 /home/bir
birder 386844 36.7 11.1 1151444 434320 ? Ssl Dec06 610:19 /home/birder/BirdNET-Pi/birdnet/bin/python3 /usr/loca
birder 526934 0.0 0.1 19728 6560 ? S 20:55 0:00 sshd: birder@pts/0
birder 526935 0.0 0.1 8624 5248 pts/0 Ss 20:55 0:00 -bash
birder 527096 7.2 1.7 475680 68744 ? SLl 20:57 0:01 ffmpeg -hide_banner -loglevel error -nostdin -vn -thr
birder 527114 400 0.1 11148 4224 pts/0 R+ 20:57 0:00 ps ux |
|
@justlen1 still doing fine? I almost forgot about this one.. I'll probably just merge the change into main |
It's working perfectly, Sir. Thanks so much. |
|
@justlen1 the changes were merged into main, so you can switch back to main: Tools->System Controls->Update |
Outstanding, thank you. |
I am running BirdNet on a Raspberry Pi 4. After several weeks of stable operation I recently ran the software updates. Since then it appears that the recording/analysis of audio fails on a random basis. Sometimes after a couple of hours, quite often within an hour. The log shows:
17:56:04---[server][INFO] LOADING TF LITE MODEL...
17:56:04---INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
17:56:04---[server][INFO] LOADING DONE!
17:56:04---[birdnet_analysis][INFO] backlog is 0
17:56:04---[birdnet_analysis][INFO] backlog done
17:56:36---[birdnet_analysis][ERROR] no more notifications: restarting...
17:56:36---[birdnet_analysis][INFO] handle_reporting_queue done
17:56:40---[server][INFO] LOADING TF LITE MODEL...
17:56:40---INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
17:56:40---[server][INFO] LOADING DONE!
17:56:40---[birdnet_analysis][INFO] backlog is 0
17:56:40---[birdnet_analysis][INFO] backlog done
Restarting the Recording Service seems to resolve the issue temporarily:
18:01:33---[birdnet_analysis][INFO] Analyzing BirdSongs/StreamData/2024-10-01-birdnet-RTSP_1-18:01:15.wav
18:01:33---[server][INFO] READING AUDIO DATA...
18:01:33---[server][INFO] READING DONE! READ 10 CHUNKS.
18:01:33---[server][INFO] ANALYZING AUDIO...
18:01:36---[server][INFO] DONE! Time 2.84 SECONDS
18:01:36---[server][INFO] 0.0;3.0-('Branta canadensis_Canada Goose', 0.12079563)
18:01:36---[server][INFO] 1.5;4.5-('Branta canadensis_Canada Goose', 0.3732961)
18:01:36---[server][INFO] 3.0;6.0-('Branta canadensis_Canada Goose', 0.34368142)
18:01:36---[server][INFO] 4.5;7.5-('Gavia immer_Common Loon', 0.15438817)
18:01:36---[server][INFO] 6.0;9.0-('Branta canadensis_Canada Goose', 0.112380795)
18:01:36---[server][INFO] 7.5;10.5-('Branta canadensis_Canada Goose', 0.05837626)
18:01:36---[server][INFO] 9.0;12.0-('Branta canadensis_Canada Goose', 0.032151874)
18:01:36---[server][INFO] 10.5;13.5-('Cygnus columbianus_Tundra Swan', 0.0765289)
18:01:36---[server][INFO] 12.0;15.0-('Glaucidium gnoma_Northern Pygmy-Owl', 0.24990916)
18:01:36---[server][INFO] 13.5;16.5-('Glaucidium gnoma_Northern Pygmy-Owl', 0.16708429)
The text was updated successfully, but these errors were encountered: