Limit number of recording for a given species ?

[Fish31520]

Hi folks,

I have a very big amount of recordings for a given species (Rougequeue Noir / Phoenicurus ochruros : almost 8000 recordings in 2 months 😅).

Is there a way to detect and keep all charts & stats (daily and total) for this bird but to keep only, let say, the 50 best recordings everyday ?

Thanks for your help !

Jean

[Fish31520]

@Nachtzuster

Hi !

By any chance, do you have an idea for this ?

[Hiemstra87]

What i do is FTP into birdnet with WinSCP, and search with the masked search option for "{birdname}-[1-8]*.mp3" (or any other extension you use). This lets me easily delete the recordings of 1 bird species with certainty 89% or less in one go. Also this keeps the database intact, and i keep the spectograms.
Might not be the most accessible way for everyone though.

[alexbelgium]

Mmh we could rather easily implement that in one of the cron scripts... To have a criteria of max xx files, and have priority criterias based on file age and detection score to delete them... If we embed that in the existing disk check cron it could offer this feature without modifying the app logic. It then depends if we want to make that configurable as an option or not.

[Fish31520]

@Hiemstra87 @alexbelgium

Hi !

A moment ago, a user from the "original/vanilla" Birdnet-Pi project kindly wrote me a script to delete all recordings for a specific date & species.

I'm a real noob when it comes to coding 😅 but I guess it could be modified to meet my requirements... 🤷‍♂️

Let me know if you think I could modify it without completely messing my records database (and how ?). 😄

Thanks for your help !

import os
import re
import sys
import argparse
from datetime import datetime

# Set up argument parser
parser = argparse.ArgumentParser(description='Find all files in the BirdNET-Pi exracted directory that are earlier than a given date and time and less than a given confidence level.')
parser.add_argument('date', type=str, help='the date to match in YYYY-MM-DD format')
parser.add_argument('time', type=str, help='the time to match in HH:MM:SS format')
parser.add_argument('confidence', type=int, help='the maximum confidence value to include')
parser.add_argument('--delete', action='store_true', help='delete matched files')
args = parser.parse_args()

# Read the EXTRACTED directory from the config file
with open('/etc/birdnet/birdnet.conf') as f:
    for line in f:
        if line.startswith('EXTRACTED='):
            EXTRACTED = line.split('=')[1].strip()
            break
    else:
        sys.exit('EXTRACTED directory not found in config file')

# Append "/By_Date" to the EXTRACTED directory
EXTRACTED = os.path.join(EXTRACTED, 'By_Date')

# Assign the regex pattern, date, and time as variables
pattern = r'(.+?)-(\d\d)-(\d\d\d\d-\d\d-\d\d)-.+?-(\d\d:\d\d:\d\d)\.(.*)'
date = args.date
time = args.time
max_confidence = args.confidence

# Convert the input date and time to Unix timestamps
input_timestamp = datetime.strptime(f'{date} {time}', '%Y-%m-%d %H:%M:%S').timestamp()

# Create an empty array to hold the matched filenames
matched_files = []

# Use the find command to locate all files in the directory (including its subdirectories)
# that match the regex pattern, and loop through them
for root, dirs, files in os.walk(EXTRACTED):
    for file in files:
        # Get the filename from the full file path
        filename = os.path.join(root, file)

        # Use regex capture groups to find matches
        match = re.match(pattern, os.path.basename(filename))
        if match:
            # Extract the confidence, date and time from the filename
            file_confidence = int(match.group(2))
            file_date = match.group(3)
            file_time = match.group(4)
        
            # Convert the file date and time to Unix timestamps
            file_timestamp = datetime.strptime(f'{file_date} {file_time}', '%Y-%m-%d %H:%M:%S').timestamp()

            # Check if the file timestamp is earlier than the input timestamp and confidence level
            if file_timestamp < input_timestamp and file_confidence <= max_confidence:
                # Add the matched filename to the array
                matched_files.append(filename)

# Output and optionally delete the matched filenames
for filename in matched_files:
    print(filename)
    if args.delete:
        os.remove(filename)```

[alexbelgium]

Quite interesting! I was starting a script in bash. We should test what is the most efficient as this will have to deal with many many files. This script (WIP) took 30s for around 149k files stored on an external drive. Once we have a working script we'll add it to the cron.

#!/bin/bash

# SPECIES CLEANER 2024-06-24
########################
# 0. Set variable, get files_per_species and use 100 if not defined. This will be the additional number of audio files remaining per species, not considering the last 7 days and the protected files
# 1. Get all individual Com_Names from the sqlite
# 2. For a single specie at a time list all files in the BirdSongs folder
## Containing the right format (*.*) , not *.png
## Not from the last 7 days (to avoid removing recent files)
## Not protected through disk_check_exclude (to respect purge protection)
## Change hyphen in species name
## Sort by confidence (so that less confident files will be removed first), then by year, then month, then day
## Protect the number of files specified in the variable files_per_species
## Restore hyphen in species name
## Remove all other audio files and their linked png


source /etc/birdnet/birdnet.conf
base_dir="$HOME/BirdSongs/Extracted/By_Date"
max_files_species="${MAX_FILES_SPECIES:-1000}"
cd "$base_dir" || true

# If max_files_species is not higher than 1, exit
if [[ "$max_files_species" -lt 1 ]]; then
    exit 0
fi

# Get unique species
bird_names=$(
    sqlite3 -readonly "$HOME"/BirdNET-Pi/scripts/birds.db <<EOF
.mode column
.headers off
SELECT DISTINCT Com_Name FROM detections;
.quit
EOF
)

# Sanitize the bird names (remove single quotes and replace spaces with underscores)
sanitized_names="$(echo "$bird_names" | tr ' ' '_' | tr -d "'" | grep '[[:alnum:]]')"
# Remove trailing underscores
sanitized_names=$(echo "$sanitized_names" | sed 's/_*$//')
# Define how date works
dateformat=""
if test "$(date -d "-7 days" '+%Y-%m-%d' 2>/dev/null)"; then
    dateformat=" days"
fi

# Read each line from the variable and echo the species
while read -r species; do
    echo -n "$species : "
    species_san="${species/-/=}"
    # Dummy file to execute the rm using xargs even if no files are there. Best solution found for code speed
    touch temp
    find */"$species" -type f -name "*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*.*" \
        -not -name "*.png" \
        -not -name "*$(date -d "-7$dateformat" '+%Y-%m-%d')*" \
        -not -name "*$(date -d "-6$dateformat" '+%Y-%m-%d')*" \
        -not -name "*$(date -d "-5$dateformat" '+%Y-%m-%d')*" \
        -not -name "*$(date -d "-4$dateformat" '+%Y-%m-%d')*" \
        -not -name "*$(date -d "-3$dateformat" '+%Y-%m-%d')*" \
        -not -name "*$(date -d "-2$dateformat" '+%Y-%m-%d')*" \
        -not -name "*$(date -d "-1$dateformat" '+%Y-%m-%d')*" \
        -not -name "*$(date '+%Y-%m-%d')*" |
        grep -vFf "$HOME/BirdNET-Pi/scripts/disk_check_exclude.txt" |
        sed "s|$species|$species_san|g" |
        sort -t'-' -k4,4nr -k5,5n -k1,1nr -k2,2nr -k3,3nr |
        tail -n +"$((max_files_species + 1))" |
        sed "s|$species_san|$species|g" |
        sed 'p; s/\(\.[^.]*\)$/\1.png/' |
        awk 'BEGIN{print "temp"} {print}' |
        xargs sudo rm && echo "success ($(find */"$species" -type f -name "*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*.*" \
        -not -name "*.png" | wc -l)) remaining" || echo "failed ($?)"
# rm to be changed to touch or echo if you want to test without deletion
done <<<"$sanitized_names"

[alexbelgium]

The script above should work - although it tested it with "touch" and not "rm" for the moment. It seems to be the most efficient I could find with around 30s to touch 149k files on an external drive. Any opinions on it before I try it? :-p

@lloydbayley : any advice?

[alexbelgium]

v2 : Corrected the code to allow for species with hypens in the name

Is there any benefit to keep the png files? as anyway it won't show in historical recordings if there is no mp3 file. Alternative would be to keep a mp3 file at 0bits to just show the spectrograms...

[lloydbayley]

Tried it here on my dev machine with touch and seems to have the desired results!!

[alexbelgium]

Thanks! Then perhaps I'll try on my system with rm ;) still will do a backup before first run ;)

[wenestvedt]

Just chiming in to add my support: I have had 2198 sparrow detections out of 2378 total detections in my first few days. This is ridiculous, albeit kind of funny!

[Fish31520]

@alexbelgium

Wow, thanks a lot for this very complete report !

I live in France (but I miss Namur, where I lived, so much ! 😄) so most of the settings you use should be perfect for me, I'll try yours for few weeks.

Regarding the microphone, mine seems to do the work so far but I may ask a new one for Christmas, I'll check that later.

Thanks once again for your help !

Jean

[alexbelgium]

Well in the end honestly it's only incremental gains by changing a microphone - my final system costs around 150€ but my first one (using a free rtsp feed from an existing camera) was already giving excellent results ;-) so in the end the best mic is the one that you have, and you can tweak things potentially using your current ones by (for example) playing with the gain with alsamixer (that you can connect to using the web terminal)

Namur is really nice indeed, and lots of birds I was there not a long while ago ! But well France has so many beautiful places too - and great biodiversity !

Have fun with Birdnet !

[Fish31520]

@alexbelgium

Hi,

I'm really happy to see how projects evolve and progress in Github but I have one more noob question ! 😅

What does this concretely means ?

Thanks and bonne journée/dag ! 👋

[alexbelgium]

Ahah, it means it was incorporated in the main code. Therefore by updating birdnet from the settings, or by installing my new version of the add-on, the option appears and the function works. We can now close this discussion ;)

[Fish31520]

Allright ! 😄

Thanks a lot !