CommonNighthawk_Recognizer_Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Common Nighthawk Call Recognizer\n",
    "This notebook has been developed to allow users to use the Common Nighthawk call recognizer developed by Chris Scott and Elly Knight to identify Common Nighthawk calls on their own recordings.  It does not produce a binary score (\"yes\" or \"no\") for simple identification of Common Nighthawk presence, but rather a sigmoidal score between 0 and 1.  The user must determine a threshold at which they consider a score to be a positive identification of an Common Nighthawk.  See [Knight et al. 2017](https://doi.org/10.5751/ACE-01114-120214) for information on how to evaluate a model and how to select a threshold for your statistical purposes.\n",
    "\n",
    "This notebook also provides a distance estimation function that will return an estimated distance (with 95% CI) of the Common Nighthawk from the ARU based on the relative sound level (volume) of the call.  Please read the description of this model and the conditions it was developed under to know the possible limitations of it before using the distance estimation function on your own recordings.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", UNDER THE MIT LICENSE, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/ABbiodiversity/Recognizer_CommonNighthawk/blob/main/CommonNighthawk_Recognizer_Processing.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Packages\n",
    "\n",
    "Import packages to run the model.  Updates to packages since the model was originally created may result in warnings when running the model.  As of June 2024 the most up to date versions of the packages below allowed the model to be run successfully on audio data despite any warnings that python may generate during the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import timedelta\n",
    "import librosa\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "from scipy.special import expit\n",
    "import tensorflow as tf\n",
    "import time\n",
    "from tqdm.notebook import tqdm_notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google Colab Steps\n",
    "\n",
    "If running the notebook in Colab then you'll need to mount Google Drive and clone the repository there in order to call the models and your recording files when processing.  Skip the next two steps if running on a local machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Mount Google Drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clone the Github Repository to Google Drive\n",
    "\n",
    "If running the notebook through Colab then clone the Github repository to your Google Drive in order to access the model files stored in the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/ABbiodiversity/Recognizer_Ovenbird.git\n",
    "\n",
    "#Adjust this link"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Change your working directory\n",
    "Change your working directory to the folder containing the notebook and the model files.\n",
    "\n",
    "## If running locally\n",
    "\n",
    "To run this notebok on your local machine you can either clone this git repository to your local machine, or you can download all the necessary files from github to a local drive folder without cloning the repository. The working directory in either case will be the folder that contains the notebook, model files and distance prediction file.\n",
    "```python\n",
    "%cd \"C:/Users/User/Folder/path\"\n",
    "```\n",
    "\n",
    "## If running in Colab\n",
    "\n",
    "The working directory location should be the location that the github repository was cloned into in the step above.  The output of the above step should look something like this:\n",
    "```\n",
    "Cloning into 'Recognizer_CommonNighthawk'...\n",
    "remote: Enumerating objects: 44, done.\n",
    "remote: Counting objects: 100% (44/44), done.\n",
    "remote: Compressing objects: 100% (34/34), done.\n",
    "remote: Total 44 (delta 19), reused 27 (delta 9), pack-reused 0\n",
    "Receiving objects: 100% (44/44), 2.03 MiB | 8.96 MiB/s, done.\n",
    "Resolving deltas: 100% (19/19), done.\n",
    "```\n",
    "\n",
    "In this instance the github repository was cloned into the location 'Recognizer_CommonNighthawk'.  Therefore you will change the working directory to match that location:\n",
    "\n",
    "```python\n",
    "%cd Recognizer_CommonNighthawk\n",
    "```\n",
    "Note, no additional folder path information (i.e. 'content/drive/MyDrive/') which may be necessary for other Google Drive location mapping is necessary for this step when using a repository cloned to Google Drive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the working directory below to the appropriate directory for your files\n",
    "%cd \"C:/Users/User/Documents/Repos/Recognizer_CommonNighthawk\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Designate Recording Folder\n",
    "\n",
    "Enter the full filepath of the folder where the recording files to be processed are stored. The filepath will need to be changed to wherever you have stored the files on your machine or on Google Drive, depending on your processing choice.\n",
    "\n",
    "Example filepath for local files: \n",
    "```python\n",
    "'C:/Users/User/Documents/Field_recordings/'\n",
    "```\n",
    "If you're unfamiliar with the syntax for Google Drive paths in Colab, \"/content/drive/\" is required, followed by the rest of the filepath in your Google Drive folder structure, excluding any spaces in the folder names. Example filepath for Google Drive folder: \n",
    "```python\n",
    "'/content/drive/MyDrive/Data/Field_recordings/'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change path below to match your recordings folder filepath\n",
    "RECORDINGS = 'C:/Users/User/Documents/Repos/Recognizer_CommonNighthawk/Recordings/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the recording files\n",
    "If you want to check that you have correctly designated the folderpath of your target recordings and that the folder contains the recording files then you can run the following cell to list all the contents of your RECORDINGS folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.listdir(RECORDINGS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Designate the results folder\n",
    "This will create a folder within the RECORDINGS folder that will contain the results of the recognizer.  This folder wil include the date of the model run to help distinguish different runs of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make sure there is somewhere to save results\n",
    "from datetime import date\n",
    "today = date.today()\n",
    "results_recognizer_location = os.path.join(RECORDINGS, 'results_recognizer_'+str(today))\n",
    "if not os.path.exists(results_recognizer_location):\n",
    "    os.makedirs(results_recognizer_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the Common Nighthawk recognizer on your recordings\n",
    "\n",
    "The following script will run the audio files in your \"RECORDINGS\" folder through the Common Nighthawk recognizer and export the results for each recording file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def spectrogram_from_audio(y, sr):\n",
    "    sr = 16000\n",
    "    n_fft = 64\n",
    "    hop_length = 32\n",
    "    D = librosa.stft(y, n_fft=n_fft, hop_length=hop_length, window=np.blackman)[1:,:] # remove DC\n",
    "    D = librosa.power_to_db(np.abs(D)**2, ref=1.0)\n",
    "    D = np.swapaxes(D,0,1)[np.newaxis,np.newaxis,:,:] # [batch,height,width,channels]\n",
    "    return D\n",
    "\n",
    "def process_directory(recordings):\n",
    "    # find tensorflow model files\n",
    "    meta_graph_path = os.path.join(os.path.dirname(os.path.abspath('__file__')), 'coni-model.meta')\n",
    "    checkpoint_path = os.path.join(os.path.dirname(os.path.abspath('__file__')), 'coni-model')\n",
    "    \n",
    "    # list all audio files in directory\n",
    "    audio_files = librosa.util.find_files(recordings, ext=['flac', 'wav'], recurse=True, case_sensitive=False, limit=None, offset=0)\n",
    "    audio_files = list(set(audio_files)) # remove duplicates - not sure why find_files produces them!\n",
    "    \n",
    "    tf.compat.v1.reset_default_graph()\n",
    "    with tf.compat.v1.Session() as sess:\n",
    "        new_saver = tf.compat.v1.train.import_meta_graph(meta_graph_path)\n",
    "        new_saver.restore(sess, checkpoint_path)\n",
    "        features = tf.compat.v1.get_collection('input')[0]\n",
    "        is_training = tf.compat.v1.get_collection('is_training')[0]\n",
    "        level = tf.compat.v1.get_collection('level')[0]\n",
    "        detection_fn = tf.compat.v1.get_collection('detection_fn')[0]\n",
    "        \n",
    "        for filename in tqdm_notebook(audio_files):\n",
    "            y, sr = librosa.load(filename, sr=16000, mono=True)\n",
    "            S = spectrogram_from_audio(y, sr)\n",
    "            level_dB, detection_function = sess.run([level, detection_fn], feed_dict={features: S, is_training: False})\n",
    "            level_dB = level_dB[0,0,:,0]\n",
    "            detection_function = detection_function[0,0,:,0]\n",
    "            duration_secs = y.size / float(sr) \n",
    "            times = np.interp(np.arange(detection_function.size), [0,detection_function.size-1], [0,duration_secs])\n",
    "            df = pd.DataFrame({'detection_window_start_time': times, 'detector_score': detection_function, 'relative_sound_level': level_dB})\n",
    "        \n",
    "            base = os.path.basename(filename)\n",
    "            filename = os.path.splitext(base)[0]\n",
    "            savename = os.path.join(results_recognizer_location, filename + '_recognizer_function.csv')\n",
    "            df.to_csv(savename, header=True, index=False)\n",
    "\n",
    "process_directory(RECORDINGS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common Nighthawk Recognizer Results\n",
    "\n",
    "Results files will be exported in csv format to the folder called \"results_recognizer_(date)\" within the RECORDINGS folder to keep the results and the recordings together.  The files will have the same name as the audio recordings files with \"_detection_function\" as a suffix.  Each row in the files will consist of the detection window start time, the recognizer score (between 0 and 1) for that detection window, and the relative sound level during that detection window."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distance Estimation\n",
    "\n",
    "In addition to predictions about the presence of Common Nighthawks, predictions about the distance (in meters) of the bird from the ARU can be made based on the relative sound level of the signal in the recording.\n",
    "\n",
    "**NOTE**: The distance estimation model provided here was developed using calibration recordings from SM4 ARUs (WildLife Acoustics, Concord, MA, USA) in an open habitat. Distance estimates from this model for recordings collected under different conditions may be inaccurate because relative sound level can be affected by density of vegetation around the ARU and ARU recording/microphone settings. Additional details on the field methods and statistical modelling methods can be found in [Yip et al. 2019](https://zslpublications.onlinelibrary.wiley.com/doi/full/10.1002/rse2.118).\n",
    "\n",
    "This process that follows will produce distance estimates for Common Nighthawks in recordings that were run through the recognizer above.  It includes 95% confidence intervals, bounded by the values in the \"distance_CI975\" and \"distance_CI025\" fields."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the data for the Distance Estimation model\n",
    "If you want to estimate the distance of Common Nighthawks in the dataset that you have run through the recognizer above then you can use the following steps to compare the results of the detector to the distance estimation model developed in Yip et al. 2019.\n",
    "\n",
    "First you need to designate where the results are that the distance estimator will use to estimate distance.  If you are running the distance estimator right after you run the recognizer above then you don't have to define the recognizer results folder because it was defined in the steps of running the recognizer.  But if you are running the distance estimator during a different session then you will need to define the location of the recognizer results folder with the script below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This doesn't need to be run if the results_recognizer_location folder was defined in the recognizer script above.  Change this folder path to that of your results folder.\n",
    "results_recognizer_location = \"C:/Users/User/Documents/Repos/Recognizer_CommonNighthawk/Recordings/results_recognizer_20240101\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a list of the csv files in the results_recognizer_location folder for the distance estimator to be run on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_recognizer_files = [os.path.join(results_recognizer_location, f) for f in os.listdir(results_recognizer_location) if f.endswith('.csv')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to check that your recognizer results folder contains the csv's you expect then you can run the code below to see a list of all csv files in that folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_recognizer_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in the csv files to a dataframe\n",
    "And make some adjustments for running through the distance estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read each csv into a separate dataframe\n",
    "individual_dataframes = []\n",
    "for file in results_recognizer_files:\n",
    "    df = pd.read_csv(file)\n",
    "    df['filename'] = os.path.basename(file)  # Add a new column with the recognizer result file name\n",
    "    df = df.map(lambda x: x.replace('_detection_function.csv', '') if isinstance(x, str) else x) # Remove filename additions added in recognizer step\n",
    "    df['species'] = 'CONI' #Add species to dataframe\n",
    "    df['level'] = df['relative_sound_level'].round(1) # Round the relative sound level to a single decimal place\n",
    "    individual_dataframes.append(df)\n",
    "\n",
    "# Combine individual dataframes into one large dataframe to run through the distance estimator\n",
    "distance_input = pd.concat(individual_dataframes, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the Distance Predictions file from the generalized linear model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_model = pd.read_csv(\"DistancePredictions.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Distance Predictions\n",
    "This step takes the relative sound level from the classifier results and uses the generalized linear mixed model developed in [Yip et al. 2019](https://zslpublications.onlinelibrary.wiley.com/doi/full/10.1002/rse2.118) to get an estimate of the distance of the Common Nighthawk calls in the dataset from the ARU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Match the sound level in the results dataframe to the predictions file to generate the distance estimates for each detection window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distance_output = distance_input.merge(predictions_model, on=['level', 'species'], how='left')\n",
    "distance_output = distance_output[['filename', 'detection_window_start_time', 'species', 'detector_score', 'relative_sound_level', 'distance', 'distance_CI975', 'distance_CI025']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export distance estimate results\n",
    "\n",
    "You can export the results of the distance estimation as one large csv of all results that were fed in, or split out as individual csv files; one for each file that was fed through the distance estimation process.\n",
    "\n",
    "Since you may be generating distance estimation results on multiple recording files at the same time you can either export the results as one large csv file with the name of the classifier results files (if you used the recognizer steps above these should be the same as the original recording filename) as a field in the csv, or you can export individual csv files that are named to match the name of the classifier results files that were imported to the function (again, if you used the classifier steps above these should retain the name of the original recording file)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the folder to store the distance estimation results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import date                   \n",
    "today = date.today()                       \n",
    "results_distance_location = os.path.join(RECORDINGS, 'results_distance_'+str(today))\n",
    "os.makedirs(results_distance_location, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Single file output\n",
    "This step produces a single, combined results csv named \"combined_distance_function.csv\" with the name of the individual classifer results files as a field for each row within the csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "distance_output.to_csv(os.path.join(results_distance_location, 'combined_distance_function.csv'), index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Multiple file output\n",
    "\n",
    "This step produces individual results files named after the individual results csv that the distance estimates relate to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "split_field = 'filename'\n",
    "unique_values = distance_output[split_field].unique()\n",
    "\n",
    "for value in unique_values:\n",
    "    split_df = distance_output[distance_output[split_field] == value].drop(columns=[split_field])\n",
    "    output_file = os.path.join(results_distance_location, f'{value}'+ '_distance_function.csv')\n",
    "    split_df.to_csv(output_file, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Note on Distance Estimation Interpretation\n",
    "\n",
    "The results of the distance estimation process are dependent on the relative sound levels calculated by the recognizer but they do not consider whether the detection window in question actually contains a Common Nighthawk.  The distance estimator will estimate a distance for all detection windows regardless of the recognizer score, so users of the distance estimation values should make sure that they only use distance estimates from detection windows that have classifier scores above their calculated threshold for positive identification.\n",
    "\n",
    "In addition, several consecutive detection windows may have high recognizer scores, likely indicating that the Common Nighthawk call overlapped with several detection windows.  The distance estimation results will vary between these detection windows if the relative sound level in these detections windows varied (very likely), so in the case of consecutive detection windows with apparent Common Nighthawk detections you can take the lowest (closest) distance estimation value from these consecutive detection windows to determine how close the Common Nighthawk was."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References:\n",
    "Knight, E. C., K. C. Hannah, G. Foley, C. Scott, R. Mark Brigham, and E. M. Bayne. 2017. Recommendations for acoustic recognizer performance\n",
    "assessment with application to five common automated signal recognition programs. Avian Conservation and Ecology 12(2):14.\n",
    "\n",
    "Yip, D. A., E. C. Knight, E. Haave‐Audet, S. J. Wilson, C. Charchuk, C. D. Scott, P. Solymos & E. M. Bayne. (2020). Sound level measurements from audio recordings provide objective distance estimates for distance sampling wildlife populations. Remote Sensing in Ecology and Conservation, 6(3), 301-315."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "widgets": {
   "state": {
    "10281d73194b41579d349ea7bfc39698": {
     "views": [
      {
       "cell_index": 1
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}