notebooks/Application_of_a_pretrained_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dominican-haven",
   "metadata": {},
   "source": [
    "# Application of a Pretrained Classifier\n",
    "\n",
    "The project aims to use maschine learning methods to emulate a cloud classification scheme. The classifer can be trained using large amounts of data and later be used to predict cloud types from satelite data. Those two steps can be run separately. \n",
    "\n",
    "This notebook contains a short explanation how to use a pretrained classifier in order to predict labels from new input data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "former-stage",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "At first we need to point python to the project folder. The path can be assigned as a relative path as shown below, or as an absolute system path.\n",
    "Than the module can be imported via the `import cloud_classifier` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "biological-pregnancy",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import ctypytool\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "future-florist",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "Our first step is to create a classifier object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "pursuant-network",
   "metadata": {},
   "outputs": [],
   "source": [
    "cc = ctypytool.cloud_classifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "clear-argument",
   "metadata": {},
   "source": [
    "Than we need to point our classifier object to an already existing classifier. The `load_project()` method will load an existing classifier into our classifier object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "second-vault",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"../classifiers/TreeClassifier\"\n",
    "cc.load_project(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stretch-checkout",
   "metadata": {},
   "source": [
    "## Applying the Classifier: Prediction of Cloud Type Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nominated-intention",
   "metadata": {},
   "source": [
    "### Using a User-Defined File List"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "precise-savannah",
   "metadata": {},
   "source": [
    "In order to predict labels with the now loaded classifier, we need to specify input files of satelite data. This can be done manually via in `input_files` option in the `set_project_parameters` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "normal-purple",
   "metadata": {},
   "outputs": [],
   "source": [
    "file_1 = \"../data/example_data/msevi-medi-20190317_1800.nc\"\n",
    "file_2 = \"../data/example_data/msevi-medi-20190318_1100.nc\"\n",
    "\n",
    "cc.set_project_parameters(input_files = [file_1, file_2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "grand-comfort",
   "metadata": {},
   "source": [
    "We now run the prediction pipeline (with the `run_prediction_pipeline()` method) which \n",
    "* applies the classifier to our input data and\n",
    "* stores the predicted labels.\n",
    "\n",
    "The option `create_filelist` is set to `False` to take the user-defined input file list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "corrected-upset",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifier loaded!\n",
      "Masked indices set!\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190317_1800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190318_1100_predicted.nc\n"
     ]
    }
   ],
   "source": [
    "cc.run_prediction_pipeline(create_filelist = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "closed-reserve",
   "metadata": {},
   "source": [
    "### Using an Automatically Generated Input File List"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "senior-spanish",
   "metadata": {},
   "source": [
    "Alternatively to the manual definition, the input file list can be generated automatically.\n",
    "\n",
    "\n",
    "The easiest way to do so is to put all input files into an input data folder (here it is set to `../data_example_data`) and just tell the classifier where to look via the `input_source_folder` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "curious-certificate",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lrwxrwxrwx 1 b380352 bb1174 49 Apr  9 10:42 ../data/example_data -> /work/bb1174/data/meteosat-seviri-subregions/medi\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ls -l ../data/example_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "democratic-individual",
   "metadata": {},
   "outputs": [],
   "source": [
    "cc.set_project_parameters(input_source_folder = \"../data/example_data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "presidential-budget",
   "metadata": {},
   "source": [
    "In a next step, we can let the classifier predict labels from the input files we have specified.\n",
    "This is again done with the `run_prediction_pipeline()` method.\n",
    "\n",
    "If we want the classifier to automatically generate a list of input files and therefore set the option `create_filelist` to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "compound-instrument",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input filelist created!\n",
      "Classifier loaded!\n",
      "Masked indices set!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20201105_0500_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200106_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191118_1700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200526_1700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200921_1300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20201104_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190207_0000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190818_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191212_1500_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200807_1200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190823_2300_predicted.nc\n",
      "Removed 201441 vectors for containig 'Nan' values\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200708_0200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190301_0200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20201116_0900_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200211_1000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200213_1800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190826_1400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200302_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190820_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191025_0100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191111_0600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191118_0300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190619_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190502_1700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200521_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200518_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191205_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200721_1800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200407_1700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200321_0000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190318_1100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190903_0000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190310_1300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190606_0100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190224_1400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200516_1700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190314_2000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190601_1400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190305_2200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190823_2000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190907_0600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190523_0200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200405_0500_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190204_1300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200724_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200129_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200518_1000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190420_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191016_1600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190106_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200205_0600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190402_0100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190103_0300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200411_1800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190826_0300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191128_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190113_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191215_1400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190512_0000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200308_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190619_1300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20201107_0000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190413_1800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20201122_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190828_0200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200802_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190217_0100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190616_0900_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190107_1200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190815_0900_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190611_0100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190225_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200131_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200422_0600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191102_1800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200914_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200111_1100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191119_0500_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200816_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190403_0600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190615_1000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190906_0900_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191021_1400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190318_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190410_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200110_2200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190326_1500_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200206_0300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200809_1700_predicted.nc\n",
      "Removed 201441 vectors for containig 'Nan' values\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200426_1100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190726_1600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190306_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190818_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190101_1600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200222_2100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200103_2000_predicted.nc\n",
      "Removed 201441 vectors for containig 'Nan' values\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191129_1200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191024_1700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191030_1000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191016_1200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20201124_1500_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191104_1000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200110_2000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191212_1600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200823_0300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190301_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200809_1400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190101_0800_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200209_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190612_0700_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200406_0300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200101_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191011_0600_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190905_1000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191109_0000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20191112_2200_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200814_1300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200830_0800_predicted.nc\n",
      "Removed 18781 vectors for containig 'Nan' values\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200717_2000_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200803_0500_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20190113_0400_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200730_1100_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200107_2300_predicted.nc\n",
      "Input vectors created!\n",
      "Labels saved as nwcsaf_msevi-medi-20200317_0800_predicted.nc\n"
     ]
    }
   ],
   "source": [
    "cc.run_prediction_pipeline(create_filelist = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mature-fundamentals",
   "metadata": {},
   "source": [
    "## Accessing predicted labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "confused-digest",
   "metadata": {},
   "source": [
    "The predicted labels are stored in the folder of the classifier we are using. They are located in the subfolder `labels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "joined-seeking",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nwcsaf_msevi-medi-20190317_1800_predicted.nc\n",
      "nwcsaf_msevi-medi-20190318_1100_predicted.nc\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ls ../classifiers/TreeClassifier/labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "great-johnson",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "f90nml-python3.10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}